Data cleaning...what is the best method?

Discuss all Leica Cyclone, Cyclone REGISTER 360 & Cyclone FIELD 360 software here.
Post Reply
Jonny5isAlive
I have made 50-60 posts
I have made 50-60 posts
Posts: 53
Joined: Thu Feb 18, 2010 5:01 pm
14
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of 3D Reality Capture
Country: USA
Linkedin Profile: Yes
Location: PA

Data cleaning...what is the best method?

Post by Jonny5isAlive »

Just wondering what people are doing to clean datasets of unwanted noise/vehicles/pedestrians.

When we get into "busy" (urban) areas, cleaning time can slow to a crawl in Cyclone. Cleaning can take hours, waiting on selections/deletions to update.

We usually try to avoid unifying prior to cleaning, as it is often beneficial to be able to isolate clouds during the cleaning process.

We are working locally, data stored on SSD, have 64GB RAM, and Intel Xeon CPU E5-2687W v3 @ 3.10GHz

Any ideas appreciated.

Thanks!
Tookie
I have made 100> posts
I have made 100> posts
Posts: 108
Joined: Thu Sep 21, 2017 9:43 pm
6
Full Name: Iain Took
Company Details: WMP
Company Position Title: Dogsbody
Country: United Kingdom
Linkedin Profile: No
Has thanked: 3 times
Been thanked: 6 times

Re: Data cleaning...what is the best method?

Post by Tookie »

We unify then select the noisy area and use the flat surface tool to deselect the road surface.

I've made a training guide with this in if you need more detail.

Edited to add that we don't have big datasets but do have lots of people moving in them despite all my shouting :-D
Jonny5isAlive
I have made 50-60 posts
I have made 50-60 posts
Posts: 53
Joined: Thu Feb 18, 2010 5:01 pm
14
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of 3D Reality Capture
Country: USA
Linkedin Profile: Yes
Location: PA

Re: Data cleaning...what is the best method?

Post by Jonny5isAlive »

Sure, I'd love to take a look. Do you have a link?

Are you referring to "region grow smooth surface"?
Tookie
I have made 100> posts
I have made 100> posts
Posts: 108
Joined: Thu Sep 21, 2017 9:43 pm
6
Full Name: Iain Took
Company Details: WMP
Company Position Title: Dogsbody
Country: United Kingdom
Linkedin Profile: No
Has thanked: 3 times
Been thanked: 6 times

Re: Data cleaning...what is the best method?

Post by Tookie »

Yes region grow. I can email a copy of the guide if you PM me your email address.
User avatar
Matt Young
Honorary Member
Honorary Member
Posts: 3929
Joined: Fri Nov 30, 2007 3:03 pm
16
Full Name: Matt Young
Company Details: Baker Hicks
Company Position Title: CAD-BIM Lead
Country: UK
Linkedin Profile: No
Has thanked: 45 times
Been thanked: 40 times

Re: Data cleaning...what is the best method?

Post by Matt Young »

Definitely agree with the above methods.

If you work with larger unified datasets then compression of the database is important from time to time when cleaning and re-unifying the data.

On the subject of noise, did you know that the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise.
If you don't see that there is nothing, then you are kidding yourself.
Jonny5isAlive
I have made 50-60 posts
I have made 50-60 posts
Posts: 53
Joined: Thu Feb 18, 2010 5:01 pm
14
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of 3D Reality Capture
Country: USA
Linkedin Profile: Yes
Location: PA

Re: Data cleaning...what is the best method?

Post by Jonny5isAlive »

the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise
Yes, I am aware. Considering this as well. Actually just ran across the a topic here from 2014 when this may have been first suggested! :D

So suggestions are:
Unify (Fast? Reduce?)
Clean
If slow, compact? Re-unify? (Fast?)

Thanks
User avatar
sreed
V.I.P Member
V.I.P Member
Posts: 257
Joined: Mon Apr 28, 2014 4:31 pm
9
Full Name: Scott Reed
Company Details: Prologue Systems
Company Position Title: Principal
Country: USA
Skype Name: sabotva
Linkedin Profile: Yes
Location: Richmond, VA USA
Has thanked: 16 times
Been thanked: 15 times

Re: Data cleaning...what is the best method?

Post by sreed »

I have an App we developed to do in Scene/Faro what the RTC360 does by double-scanning. PM me if you are interested in a demo trial. The App is a plugin for Scene.
User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 1615
Joined: Tue May 01, 2012 5:19 pm
11
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 183 times
Been thanked: 548 times

Re: Data cleaning...what is the best method?

Post by landmeterbeuckx »

@sreed : what is it called. Cannot find it immediately
LSBbvba
Surveying services - 3D Laserscanning
Tel : +32477753126
www.lsbbvba.be
[email protected]
dirtysouthsurveyor
I have made <0 posts
I have made <0 posts
Posts: 1
Joined: Thu Jun 07, 2018 4:19 pm
5
Full Name: Brandon B
Company Details: BKF Engineers
Company Position Title: PLS Project Surveyor
Country: USA
Linkedin Profile: Yes

Re: Data cleaning...what is the best method?

Post by dirtysouthsurveyor »

When you say "compact" are you referring to optimizing the database?

I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.

In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
User avatar
gsisman
V.I.P Member
V.I.P Member
Posts: 898
Joined: Fri Oct 07, 2016 1:51 pm
7
Full Name: Steve Long
Company Details: Montgomery County DOT _ MD
Company Position Title: Land Survey Supervisor
Country: United States
Skype Name: gsisman1
Linkedin Profile: Yes
Has thanked: 767 times
Been thanked: 148 times

Re: Data cleaning...what is the best method?

Post by gsisman »

dirtysouthsurveyor wrote: Thu Oct 04, 2018 7:35 pm When you say "compact" are you referring to optimizing the database?

I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.

In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
There is a great terrain modeling from Scan Cloud program called QT Modeler (John Hopkins Applied Imagery Lab). They have recently added this feature at our request and other miltary clients. The side profile-defined corridor width-polygonal clip box. Check it out. The next request is an E57 unified import-as it only accepts LAS & PTS format exports from Leica Products
http://appliedimagery.com/
Post Reply

Return to “Leica Cyclone, Cyclone REGISTER 360 & Cyclone FIELD 360”