Open Point Clouds Format | Interview with DR. Daniel Wujanz

An open petition for unrestricted point cloud exchange.
hobu
I have made <0 posts
I have made <0 posts
Posts: 4
Joined: Fri Sep 03, 2021 1:10 pm
2
Full Name: Howard Butler
Company Details: Hobu Inc
Company Position Title: President
Country: USA
Linkedin Profile: No
Been thanked: 5 times

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by hobu »

While PDAL has a decent array of readers and writers they largely deal with existing open source formats sitting on existing open source SDKs for those formats.
PDAL definitely has explicit support for allowing developers to provide access to their firm's data formats through binary-only SDKs. We would gladly welcome more of these readers and writers as well. It isn't just about open source stuff, although that's what's been most convenient to date.
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by smacl »

hobu wrote: Thu Oct 07, 2021 7:13 pm
While PDAL has a decent array of readers and writers they largely deal with existing open source formats sitting on existing open source SDKs for those formats.
PDAL definitely has explicit support for allowing developers to provide access to their firm's data formats through binary-only SDKs. We would gladly welcome more of these readers and writers as well. It isn't just about open source stuff, although that's what's been most convenient to date.
Hopefully this initiative will encourage development of such SDKs. The substantive issue being addressed here isn't the framework, it is the lack of efficient lossless SDKs to the most commonly used native formats within this domain.
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1237
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 220 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by jedfrechette »

smacl wrote: Thu Oct 07, 2021 6:57 pmIf Leica were to provide an SDK to allow reading of LGS and that was implemented within SCC, the problem is solved.
It is solved for the one specific case of LGS -> SCC.

If Scantra wants to import LGS files they need to implement much of the same code that you did for SCC.
If PointCap wants to import LGS files they need to implement much of the same code that you did for SCC.
If PointSharePlus wants to import LGS files they need to implement much of the same code that you did for SCC.
...

Then when Leica changes the API of their SDK all of you also need to update your individual implementations.

How is that scalable to more than a handful of vendors?

On the other hand, if all three of you contribute equally to the development of a hypothetical LGS reader for PDAL the maintenance cost for each of you is immediately reduced by 2/3.

Don't get me wrong I completely agree that vendors should provide programmatic access to their data stores and as I wrote before my company actively chooses software based on exactly that criteria. However, simply getting everybody to provide an SDK is only one small part of what is necessary to actually achieve the aspirational goals that this petition lays out. Many vendors already provide SDKs and yet compatibility is still an issue.

I guess what bothers me about the petition is that it doesn't offer any concrete solutions to the very real limitations that do exist, while at the same time a lot of the messaging around it has implied that it is in fact offering a better alternative to existing Open standards and formats. As far as I can tell it was also developed with little if any input from the people who have actually been putting in the hard work on these problems for years; by serving on standards committees, working on specifications and buy in from large organizations, and publishing code that others in the community can build on. It's very easy to write about how the current status quo isn't ideal, but it's often a lot harder to actually come up with and implement better solutions.

I sincerely hope those of you spearheading this prove me wrong, but to me it just feels like one of the thousands of throwaway petitions on change.org. Low impact "activism" that is easy to agree with and sign-on to, but unlikely to result in anybody taking up the hard work necessary to change and improve the situation on the ground.
Jed
VXGrid
V.I.P Member
V.I.P Member
Posts: 544
Joined: Fri Feb 24, 2017 10:47 am
7
Full Name: Martin Graner
Company Details: PointCab GmbH
Company Position Title: Research and Development
Country: Germany
Linkedin Profile: No
Has thanked: 160 times
Been thanked: 175 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by VXGrid »

jedfrechette wrote: Thu Oct 07, 2021 9:17 pm
smacl wrote: Thu Oct 07, 2021 6:57 pmIf Leica were to provide an SDK to allow reading of LGS and that was implemented within SCC, the problem is solved.
It is solved for the one specific case of LGS -> SCC.

If Scantra wants to import LGS files they need to implement much of the same code that you did for SCC.
If PointCap wants to import LGS files they need to implement much of the same code that you did for SCC.
If PointSharePlus wants to import LGS files they need to implement much of the same code that you did for SCC.
...

Then when Leica changes the API of their SDK all of you also need to update your individual implementations.

How is that scalable to more than a handful of vendors?

On the other hand, if all three of you contribute equally to the development of a hypothetical LGS reader for PDAL the maintenance cost for each of you is immediately reduced by 2/3.

Don't get me wrong I completely agree that vendors should provide programmatic access to their data stores and as I wrote before my company actively chooses software based on exactly that criteria. However, simply getting everybody to provide an SDK is only one small part of what is necessary to actually achieve the aspirational goals that this petition lays out. Many vendors already provide SDKs and yet compatibility is still an issue.

I guess what bothers me about the petition is that it doesn't offer any concrete solutions to the very real limitations that do exist, while at the same time a lot of the messaging around it has implied that it is in fact offering a better alternative to existing Open standards and formats. As far as I can tell it was also developed with little if any input from the people who have actually been putting in the hard work on these problems for years; by serving on standards committees, working on specifications and buy in from large organizations, and publishing code that others in the community can build on. It's very easy to write about how the current status quo isn't ideal, but it's often a lot harder to actually come up with and implement better solutions.

I sincerely hope those of you spearheading this prove me wrong, but to me it just feels like one of the thousands of throwaway petitions on change.org. Low impact "activism" that is easy to agree with and sign-on to, but unlikely to result in anybody taking up the hard work necessary to change and improve the situation on the ground.
You are right, everyone who wants to use a certain format needs to implement the SDK usage in one way or another, may it be via PDAL (if the readers & writers are accessible), or due to own means.

If I want to read the inclinometer data + GPS + extra taken images from an external camera from hardware XYZ, and this is not implemented in an exchange format (or in PDAL), I need to write my own format Reader.
So if I have a SDK (which provides the stuff I'm interested in), I can access this stuff, and don't need to worry about anything else.

Regarding interface changes:
How often do they occure, and if they occure, are they well documented?
If I think back over the last two years, I don't see many changes we have adapted in our format readers or writers (yes if we take every reader/write into accout it is quite some time - due to the amount of readers/writers we have).

Regarding a standardized comitee:
I was talking to another developer at Intergeo this year about standardization and the open source file formats. He told me that he was part of the cometee, but the changes he needed for his software solution were all rejected. The other comitee members wanted to go down a different road.

Regarding adoption of only the big vendors and not the small ones:
As every other vendor is doing, we are listening to our clients. If they want to read format "qwerty", and there is a Reader SDK available (open source or license based), we are going to (if necessary, license it and) implement it for them.
We (as in PointCab) don't care if we spend one week fixing challenges with one SDK, if our clients then save tons of time more, because they can import/export the format directly.
If we need 40 hours to implement it or fix it, the client saves 4 hours per project, then we only need 10 clients (or one client with 10 projects), to have the time better invested.

I can see that you prefer to use an exchange format, instead of an unstable, not well documented SDK - but we don't want the exchange formats to go away either!
They are already an important way to go from software A into software B, and yes if we had the most superior exchange format ever, where all the data is available, the compression is top notch, the interface is super easy to implement, we would implement it and use it.
But the best format is useless if nobody else reads/writes it.

As I have mentioned a couple of times now: For me, this petition is there to enable the end users to faster, more efficient work flows, without the time constraint of converting data to an intermediate exchange format, reimport the data, only to throw away the intermediate files.
In addition I hope it will be an eye opener to other manufacturers / vendors to understand the needs and wishes of the end customers.
User avatar
Jason Warren
Administrator
Administrator
Posts: 4224
Joined: Thu Aug 16, 2007 9:21 am
16
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: No
Location: Retford, UK
Has thanked: 443 times
Been thanked: 246 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by Jason Warren »

It looks like there is a similar issue with Machine Control products (IE Leica, Topcon & Trimble) too... :?



https://machinecontrolforum.com/viewtopic.php?f=23&t=55
Jason Warren
Co_Founder

Dedicated to 3D Laser Scanning
LaserScanningForum
RJGEOMATICS
V.I.P Member
V.I.P Member
Posts: 133
Joined: Tue Apr 23, 2019 7:20 pm
4
Full Name: Bart Man
Company Details: Laser Scanning
Company Position Title: Passionate about making amazing PCs
Country: Canada
Linkedin Profile: No
Has thanked: 38 times
Been thanked: 68 times

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by RJGEOMATICS »

Jason Warren wrote: Fri Oct 08, 2021 5:48 pm It looks like there is a similar issue with Machine Control products (IE Leica, Topcon & Trimble) too... :?



https://machinecontrolforum.com/viewtopic.php?f=23&t=55
Well first.

As for the work around on the Machine Control Forum. If they have Lat/Long/Height and the Corresponding NEZ coordinates. Then with their office software be it TBC or Infinity etc. they will be able to KeyIn the values and force a virtual calibration file. Actually even with Trimble Survey Controller, or Trimble ACCESS you can do this directly on the data collector. But you do need the Lat/Long/Heights with the corresponding NEZ values. Preferably 5 or more well displaced control point through out the project.

Now Rant time.

I would say the root start of all of this, was in circa 2004 when Autodesk decided to start making proprietary formats for their software, which really reared its ugly head with the demise of LDD3 and release of Map and Civil3D.

It was soon discovered, there is no true continual profit stream in simply selling a one time item, be it a scanner, camera, software etc... The true profits are making things proprietary and in requirement of maintenance or continual subscription fees.

Kudos to Autodesk, they realized how to drive profits higher, and soon everyone else started or is starting to follow course.

So anyone providing software for sale. Simply get the manufactures providing proper agreements and SDK packages so you can take full advantage of their proprietary formats. Of course, you'll need pass those costs onto your end users. That is reality of Capitalism and profit, otherwise we are better all getting government jobs and collecting a wage.

Anyone in business is in it to make money. We don't do it just for S&G's. Nor should a business be a charity. I would say most the demand for open format is driven largely by end users simply too cheap to buy all the necessary software's and SDK's etc. Driving all prices down to the ditch.

Great example of that is the BLK360 users that were hoping to skate by on Autodesk ReCAP and using old outdated licenses of Faro Scene between the BLK360 and ReCAP. And never mind the number of users out there that try to skate by with just CloudCompare and QGIS.

Don't get me wrong. CloudCompare and QGIS are both great. But you are not going to get the same power from free open source products, as you do from Paid products. I even came across one user that was so cheap, he hacked code on a scanner, so that he could just get e57 files of each station direct off the scanner, and then based his entire business on ReCAP and CloudCompare. Of course with hacks like that, you'll never be capable of doing what the full blown proprietary software's do.

There is a reason, we need buy new computer every 1 to 2 years, need pay in excess of $10,000/year just to maintain maintenance agreements etc.. And this is for a small company. There is good reason the large engineering firms charge in excess of $250/hr while the small shows are doing work for $95/hr. The smaller operators, seem to forget that things cost money, and at those low rates, most are bankrupt in a few years.

Pay as you go SDK packages seem to be the solution. I think Leica is on the correct course with LGS. Now they need release a full SDK package for a fee to all Software providers that can benefit from LGS format. We see things like this happen in other industries, such as Snowmobiles where Bombardier invented electronic reverse, and now other manufactures pay a recurring licensing fee to adapt electronic reverse to their off-road vehicles. Great example from my home of CANADA. The true north strong and free.

Unfortunately much of the industry has become a race to the bottom. So many forget to equate computers, software, hardware, insurance, vehicles etc. into their pricing. Honestly some of the rates are absolutely ridiculous. One shady company from the US wanted some scanning done in Canada. The only way I could justify looking at doing the project, was to utilize a BLK360, as what they wanted to pay, was worse than working in a coffee shop up here in Canada. It was spring, and I was bored, plus had family where the job was. So I did it. Worst experience of my life of 30 years in this industry. Not only did I do the job for pittens, I never even got paid. And was expected to meet some antiquated proprietary rigid methods that was absolutely ridiculous as it was based on Open Data formats and a hacked together system of multiple free or low cost solutions. Kick in my arse for doing anything cheap. Every no and again, you need reminders as to why good customers don't mind paying your normal price. When clients aren't willing to pay your price, you are better off not even worrying about getting the job.

I can't count the number of Scanner operators out there, that I meet, and when we sit and chat, they never even considered the cost of Computers, and Registration Licenses, plus the time to register scans... They simply go ok 3 days to scan, plus few hundred a day for gear, there is my price. They don't consider a very expensive and powerful computer for registration, the data management and registration time, often being 2 to 3 times more than field time, the cost of a registration license and keeping it current on maintenance, and all these other factors.

End of RANT. Brutal reality, people. Things aren't free in this world. It costs money. So as the saying goes. If it is too hot in the kitchen, then get out of the kitchen...
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1237
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 220 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by jedfrechette »

RJGEOMATICS wrote: Fri Oct 08, 2021 6:30 pmBut you are not going to get the same power from free open source products, as you do from Paid products.
The modern internet begs to differ. Unlike the path our industry has chosen so far, the Internet and Cloud Computing has been built by people being paid to work on the open source infrastructure that underlies much of it. Open Source products aren't free, they cost just as much to develop as proprietary software. If done right though that cost can be leveraged to make the rest of the tech stack (open and closed source) much more efficient.
VXGrid wrote: Fri Oct 08, 2021 8:43 amFor me, this petition is there to enable the end users to faster, more efficient work flows, without the time constraint of converting data to an intermediate exchange format, reimport the data, only to throw away the intermediate files.
Absolutely! But I can do this today for all of the data storage end points PDAL supports. To give a concrete example. If I want to load data from Reigl's in-house database format and convert it to my own application's in-house format stored in a PostgreSQL table I can do it like so:

Code: Select all

[
    {
        "type": "readers.rdb",
        "filename": "my_riegl_db.rdbx",
    },
    {
        "type":"writers.pgpointcloud",
        "connection":"host='localhost' dbname='lidar' user='pramsey'",
        "table":"my_imported_db",
    }
]
It is a direct, in memory conversion, no need to store any temporary files. If instead I need to to import raw Optech data (scan angles, ranges, IMU+GNSS, and boresight calibration) I can change 2 lines and in 40 seconds, not 40 hours, be importing Optech data. The story is the same if I need to switch to las, e57, pts, Terasolid, ESRI i3s or any of the other data sources PDAL supports. The only reason I can't do the same with PointCab or SCC data is that nobody has written the plugins yet.

BTW, kudos to Reigl for not only providing a SDK, but also contributing a reader plugin to PDAL to make accessing your data so easy. I hope more vendors concerned with interoperability follow your lead.
VXGrid wrote: Fri Oct 08, 2021 8:43 am Regarding adoption of only the big vendors and not the small ones:
As every other vendor is doing, we are listening to our clients. If they want to read format "qwerty", and there is a Reader SDK available (open source or license based), we are going to (if necessary, license it and) implement it for them.
I absolutely applaud this effort, but I think there are practical limits to what any given vendor can support, especially smaller ones. It looks like PointCab can read data from about 14 different file formats. When I open my GIS application it can read data from more than 150. Is that what our industry is going to look like in 20 years and do you really want to be maintaining the code to support all of those formats by yourself?

At the end of the day if a particular vendor can only afford to support 5 formats (or 10, or 50) Leica and Faro will almost certainly make the short list, but PointCab or Scantra might not. That puts your product at a disadvantage relative to similar products from Leica or Faro. I'm not blaming vendors for making completely rational decisions about what to support based on user requests and marketshare, everybody has limited resources, but it does naturally increase the ability of the large established players to control the industry. One way to help mitigate that is for the smaller players to collaborate. Reigl isn't as big as Leica or Faro, but they have chosen to collaborate with PDAL so everything else being equal it costs me less to use data from their platform compared to PointCab's. If they offer a product that is similar to PointCab that gives them an advantage.

I'm largely in agreement with the fundamental goal of what the petition seems to be trying to achieve, I'm just advocating for working towards that goal in the most efficient way possible, and I think what the petition actually proposes is just a small baby step in that direction. I would argue that the last 20 years of software development has pretty clearly demonstrated that, when properly supported, Open Source is really good at efficiently providing the type of comparatively low-level infrastructure we're discussing here. We need compatibility libraries, not just specifications.

I don't want the developers of each application I use to spend 6,000 hours supporting 150 different file formats just so I can get data in to their apps. I want them to import a single library that allows them to support all of those formats, then spend the remaining 5,999 hours developing unique new features that provide other benefits to me.
Jed
User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 1616
Joined: Tue May 01, 2012 5:19 pm
11
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 183 times
Been thanked: 548 times

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by landmeterbeuckx »

I'm happy for independent software providers to exist and they should get all support.

I remember Kubit being taken over by Faro and it went downwards since then.

These independent companies like Pointcab, Luposcan, Scantra, Undet, SCC,... should be given access to these datasets. They implement workflows and solutions the big boys can also incorporate if they wish but please let them be independent.
LSBbvba
Surveying services - 3D Laserscanning
Tel : +32477753126
www.lsbbvba.be
[email protected]
VXGrid
V.I.P Member
V.I.P Member
Posts: 544
Joined: Fri Feb 24, 2017 10:47 am
7
Full Name: Martin Graner
Company Details: PointCab GmbH
Company Position Title: Research and Development
Country: Germany
Linkedin Profile: No
Has thanked: 160 times
Been thanked: 175 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by VXGrid »

RJGEOMATICS wrote: Fri Oct 08, 2021 6:30 pm [...]

Now Rant time.

[...]


End of RANT. Brutal reality, people. Things aren't free in this world. It costs money. So as the saying goes. If it is too hot in the kitchen, then get out of the kitchen...
The petition I signed up for is not about free stuff, it is about open stuff.
Yes developing things costs money and every company likes to earn money, so I don't have an issue if we as a company - or our customers - need to pay a license fee to be able to import/export the data.
Sure thing: If it is for free - even better, if there is no legal agreement needed - great.

But I do understand that a corperation / company / self-employed can't just do things for free, since their employees need to feed their family as well. All I want is the possibility to go for an easy to use, easy to license SDK, may it be with a yearly or based on users fee, but I want to have the ability (and give this power to our clients) to import/export that data.
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: Open Point Clouds Format | Interview with DR. Daniel Wujanz

Post by smacl »

I think it is worth distinguishing the arguments for having an open SDK for read access to proprietary data formats and having read/write access. An efficient, easy to implement SDK to enable read access allows native formats to be used within multi-vendor workflows but importantly also makes native formats valid formats for long term archival of data. These add significant value to that data format and should be the primary reason for providing a read-only SDK with a permissive license. Using a more restrictive license makes it hazardous to use the native format for archival purposes as the ability to invoke that SDK license might not be available in the future for one reason or the other. Still reasonable to charge for the license, but as a single fee for a perpetual license that doesn't require any input from the vendor to use at any point in the future. Also reasonable to charge for functional updates to that license to support format additions and changes on the same basis.

Read/write access has a different value proposition which in my opinion is reasonable to charge for as you would any other piece of software functionality. If you have a very efficient storage mechanism for certain types of data that adds value to anyone creating data in that format. ECW image format is a good example of distinguishing read and write costs in the above manner already in wide use in this industry.
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
Post Reply

Return to “openpointcloudformats.org”