Data cleaning...what is the best method?

Discuss all Cyclone related issues here.
Post Reply
Jonny5isAlive
I have made 20-30 posts
I have made 20-30 posts
Posts: 28
Joined: Thu Feb 18, 2010 5:01 pm
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of Architectural Services
Country: USA
Linkedin Profile: Yes
Location: Abington, PA
Contact:

Data cleaning...what is the best method?

Post by Jonny5isAlive » Wed Sep 12, 2018 1:46 pm

Just wondering what people are doing to clean datasets of unwanted noise/vehicles/pedestrians.

When we get into "busy" (urban) areas, cleaning time can slow to a crawl in Cyclone. Cleaning can take hours, waiting on selections/deletions to update.

We usually try to avoid unifying prior to cleaning, as it is often beneficial to be able to isolate clouds during the cleaning process.

We are working locally, data stored on SSD, have 64GB RAM, and Intel Xeon CPU E5-2687W v3 @ 3.10GHz

Any ideas appreciated.

Thanks!

Tookie
I have made 30-40 posts
I have made 30-40 posts
Posts: 36
Joined: Thu Sep 21, 2017 9:43 pm
Full Name: Iain Took
Company Details: WMP
Company Position Title: Dogsbody
Country: United Kingdom
Linkedin Profile: No

Re: Data cleaning...what is the best method?

Post by Tookie » Wed Sep 12, 2018 2:27 pm

We unify then select the noisy area and use the flat surface tool to deselect the road surface.

I've made a training guide with this in if you need more detail.

Edited to add that we don't have big datasets but do have lots of people moving in them despite all my shouting :-D

Jonny5isAlive
I have made 20-30 posts
I have made 20-30 posts
Posts: 28
Joined: Thu Feb 18, 2010 5:01 pm
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of Architectural Services
Country: USA
Linkedin Profile: Yes
Location: Abington, PA
Contact:

Re: Data cleaning...what is the best method?

Post by Jonny5isAlive » Wed Sep 12, 2018 2:30 pm

Sure, I'd love to take a look. Do you have a link?

Are you referring to "region grow smooth surface"?

Tookie
I have made 30-40 posts
I have made 30-40 posts
Posts: 36
Joined: Thu Sep 21, 2017 9:43 pm
Full Name: Iain Took
Company Details: WMP
Company Position Title: Dogsbody
Country: United Kingdom
Linkedin Profile: No

Re: Data cleaning...what is the best method?

Post by Tookie » Wed Sep 12, 2018 2:54 pm

Yes region grow. I can email a copy of the guide if you PM me your email address.

User avatar
Matt Young
Global Moderator
Global Moderator
Posts: 3735
Joined: Fri Nov 30, 2007 3:03 pm
Full Name: Matt Young
Company Details: Baker Hicks
Company Position Title: CAD-BIM Lead
Country: UK
Linkedin Profile: No

Re: Data cleaning...what is the best method?

Post by Matt Young » Wed Sep 12, 2018 3:05 pm

Definitely agree with the above methods.

If you work with larger unified datasets then compression of the database is important from time to time when cleaning and re-unifying the data.

On the subject of noise, did you know that the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise.

Jonny5isAlive
I have made 20-30 posts
I have made 20-30 posts
Posts: 28
Joined: Thu Feb 18, 2010 5:01 pm
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of Architectural Services
Country: USA
Linkedin Profile: Yes
Location: Abington, PA
Contact:

Re: Data cleaning...what is the best method?

Post by Jonny5isAlive » Wed Sep 12, 2018 3:50 pm

the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise
Yes, I am aware. Considering this as well. Actually just ran across the a topic here from 2014 when this may have been first suggested! :D

So suggestions are:
Unify (Fast? Reduce?)
Clean
If slow, compact? Re-unify? (Fast?)

Thanks

User avatar
sreed
V.I.P Member
V.I.P Member
Posts: 166
Joined: Mon Apr 28, 2014 4:31 pm
Full Name: Scott Reed
Company Details: Prologue Systems
Company Position Title: Principal
Country: USA
Skype Name: sabotva
Linkedin Profile: Yes

Re: Data cleaning...what is the best method?

Post by sreed » Wed Sep 12, 2018 10:37 pm

I have an App we developed to do in Scene/Faro what the RTC360 does by double-scanning. PM me if you are interested in a demo trial. The App is a plugin for Scene.

User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 223
Joined: Tue May 01, 2012 5:19 pm
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes

Re: Data cleaning...what is the best method?

Post by landmeterbeuckx » Thu Sep 13, 2018 7:54 am

@sreed : what is it called. Cannot find it immediately
LSBbvba
Surveying services - 3D Laserscanning
Tel : +3211661621
www.landmeter-beuckx.be
info@landmeter-beuckx.be

dirtysouthsurveyor
I have made <0 posts
I have made <0 posts
Posts: 1
Joined: Thu Jun 07, 2018 4:19 pm
Full Name: Brandon B
Company Details: BKF Engineers
Company Position Title: PLS Project Surveyor
Country: USA
Linkedin Profile: Yes

Re: Data cleaning...what is the best method?

Post by dirtysouthsurveyor » Thu Oct 04, 2018 7:35 pm

When you say "compact" are you referring to optimizing the database?

I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.

In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...

User avatar
gsisman
V.I.P Member
V.I.P Member
Posts: 138
Joined: Fri Oct 07, 2016 1:51 pm
Full Name: Steve Long
Company Details: Montgomery County DOT _ MD
Company Position Title: Land Survey Supervisor
Country: United States
Skype Name: gsisman1
Linkedin Profile: Yes

Re: Data cleaning...what is the best method?

Post by gsisman » Thu Oct 04, 2018 11:13 pm

dirtysouthsurveyor wrote:
Thu Oct 04, 2018 7:35 pm
When you say "compact" are you referring to optimizing the database?

I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.

In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
There is a great terrain modeling from Scan Cloud program called QT Modeler (John Hopkins Applied Imagery Lab). They have recently added this feature at our request and other miltary clients. The side profile-defined corridor width-polygonal clip box. Check it out. The next request is an E57 unified import-as it only accepts LAS & PTS format exports from Leica Products
http://appliedimagery.com/

Post Reply

Return to “Leica Cyclone”