Data cleaning...what is the best method?

Discuss all Cyclone related issues here.
User avatar
James Hall
V.I.P Member
V.I.P Member
Posts: 118
Joined: Tue Feb 02, 2010 5:13 pm
Full Name: James E Hall
Company Details: Precision Measurements Inc
Company Position Title: Survey Technician - Cyclone Modeler
Country: USA
Location: Chantilly, VA

Re: Data cleaning...what is the best method?

Post by James Hall » Fri Dec 07, 2018 4:05 pm

dirtysouthsurveyor wrote:
Thu Oct 04, 2018 7:35 pm
When you say "compact" are you referring to optimizing the database?

I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.

In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
A way and not the most elegant, is to select the the cleaned data and export them to an e57 file. Then re-import them (make sure to calculate normals) and registered the cleaned scans. This is also a way to fix a problem scan world.

James,

Origin.KR
I have made <0 posts
I have made <0 posts
Posts: 3
Joined: Wed Jan 17, 2018 8:15 pm
Full Name: Kyle Rothwell
Company Details: Origin 4n6
Company Position Title: Tech
Country: usa

Re: Data cleaning...what is the best method?

Post by Origin.KR » Fri Dec 07, 2018 8:44 pm

I have found the Cloth Simulation Fliter used in CloudCompare to be very useful to isolate noise generated from passing traffic.

http://www.cloudcompare.org/doc/wiki/in ... F_(plugin)

Jonny5isAlive
I have made 20-30 posts
I have made 20-30 posts
Posts: 28
Joined: Thu Feb 18, 2010 5:01 pm
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of Architectural Services
Country: USA
Linkedin Profile: Yes
Location: Abington, PA
Contact:

Re: Data cleaning...what is the best method?

Post by Jonny5isAlive » Fri Dec 07, 2018 10:57 pm

Thank you all for your responses!

It sounds like there is still yet room for improvement inside of Cyclone (i.e. keep all the data there, no export/process/reimport) to address the extremely slow process of cleaning scans.

Region grow is nice and does a decent job, but I look forward to a time when selections are made instantly, and deletions as well.

The waiting for these two items is mind-numbingly painful.

I hope something comes soon.

Sent from my SM-N950U using Tapatalk


Yuelin
I have made 60-70 posts
I have made 60-70 posts
Posts: 62
Joined: Sat Mar 19, 2011 5:16 am
Full Name: Yuelin
Company Details: Leica Geosystems
Company Position Title: Lead software engineer
Country: USA
Linkedin Profile: No

Re: Data cleaning...what is the best method?

Post by Yuelin » Sat Dec 08, 2018 2:35 am

Hi Jon,

Regarding to data cleaning, please read my recent posting here: https://laserscanningforum.com/forum/vi ... 984#p73836

Jonny5isAlive
I have made 20-30 posts
I have made 20-30 posts
Posts: 28
Joined: Thu Feb 18, 2010 5:01 pm
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of Architectural Services
Country: USA
Linkedin Profile: Yes
Location: Abington, PA
Contact:

Re: Data cleaning...what is the best method?

Post by Jonny5isAlive » Sat Dec 08, 2018 1:45 pm

Yuelin wrote:Hi Jon,

Regarding to data cleaning, please read my recent posting here: https://laserscanningforum.com/forum/vi ... 984#p73836
Yes, but even with this process, cleaning is a slow affair.

Starts out relatively quick, but the more edits made the slower the process becomes.

Compact/optimize helps some when slow downs occur, but overall selections/deletions are far from instantaneous (which would be ideal).



Sent from my SM-N950U using Tapatalk


User avatar
Matt Young
Global Moderator
Global Moderator
Posts: 3735
Joined: Fri Nov 30, 2007 3:03 pm
Full Name: Matt Young
Company Details: Baker Hicks
Company Position Title: CAD-BIM Lead
Country: UK
Linkedin Profile: No

Re: Data cleaning...what is the best method?

Post by Matt Young » Mon Dec 10, 2018 8:24 am

When you are working with very large datasets, instant delete is unlikely to be a thing for a while yet. It's like someone telling you to open a thousand page novel and manually remove every occurrence of a single word. A human could do that but it would take a very long time. Now make the book a point cloud with hundreds of millions of points. The computer has to read all the points, decide which ones are to be hidden (Cyclone does not delete points) and then hide them. When you optimize and compress your database, the points are deleted and the database decreased in size to compensate for the new point count. This is like you re-writing the thousand page book, so that no spaces occur where you removed all the words.

Post Reply

Return to “Leica Cyclone”