Data cleaning...what is the best method?
-
- I have made 50-60 posts
- Posts: 53
- Joined: Thu Feb 18, 2010 5:01 pm
- 14
- Full Name: Jon Adams
- Company Details: Forensic and Architectural Consulting
- Company Position Title: Director of 3D Reality Capture
- Country: USA
- Linkedin Profile: Yes
- Location: PA
Data cleaning...what is the best method?
Just wondering what people are doing to clean datasets of unwanted noise/vehicles/pedestrians.
When we get into "busy" (urban) areas, cleaning time can slow to a crawl in Cyclone. Cleaning can take hours, waiting on selections/deletions to update.
We usually try to avoid unifying prior to cleaning, as it is often beneficial to be able to isolate clouds during the cleaning process.
We are working locally, data stored on SSD, have 64GB RAM, and Intel Xeon CPU E5-2687W v3 @ 3.10GHz
Any ideas appreciated.
Thanks!
When we get into "busy" (urban) areas, cleaning time can slow to a crawl in Cyclone. Cleaning can take hours, waiting on selections/deletions to update.
We usually try to avoid unifying prior to cleaning, as it is often beneficial to be able to isolate clouds during the cleaning process.
We are working locally, data stored on SSD, have 64GB RAM, and Intel Xeon CPU E5-2687W v3 @ 3.10GHz
Any ideas appreciated.
Thanks!
-
- I have made 100> posts
- Posts: 108
- Joined: Thu Sep 21, 2017 9:43 pm
- 6
- Full Name: Iain Took
- Company Details: WMP
- Company Position Title: Dogsbody
- Country: United Kingdom
- Linkedin Profile: No
- Has thanked: 3 times
- Been thanked: 6 times
Re: Data cleaning...what is the best method?
We unify then select the noisy area and use the flat surface tool to deselect the road surface.
I've made a training guide with this in if you need more detail.
Edited to add that we don't have big datasets but do have lots of people moving in them despite all my shouting
I've made a training guide with this in if you need more detail.
Edited to add that we don't have big datasets but do have lots of people moving in them despite all my shouting
-
- I have made 50-60 posts
- Posts: 53
- Joined: Thu Feb 18, 2010 5:01 pm
- 14
- Full Name: Jon Adams
- Company Details: Forensic and Architectural Consulting
- Company Position Title: Director of 3D Reality Capture
- Country: USA
- Linkedin Profile: Yes
- Location: PA
Re: Data cleaning...what is the best method?
Sure, I'd love to take a look. Do you have a link?
Are you referring to "region grow smooth surface"?
Are you referring to "region grow smooth surface"?
-
- I have made 100> posts
- Posts: 108
- Joined: Thu Sep 21, 2017 9:43 pm
- 6
- Full Name: Iain Took
- Company Details: WMP
- Company Position Title: Dogsbody
- Country: United Kingdom
- Linkedin Profile: No
- Has thanked: 3 times
- Been thanked: 6 times
Re: Data cleaning...what is the best method?
Yes region grow. I can email a copy of the guide if you PM me your email address.
- Matt Young
- Honorary Member
- Posts: 3929
- Joined: Fri Nov 30, 2007 3:03 pm
- 16
- Full Name: Matt Young
- Company Details: Baker Hicks
- Company Position Title: CAD-BIM Lead
- Country: UK
- Linkedin Profile: No
- Has thanked: 45 times
- Been thanked: 40 times
Re: Data cleaning...what is the best method?
Definitely agree with the above methods.
If you work with larger unified datasets then compression of the database is important from time to time when cleaning and re-unifying the data.
On the subject of noise, did you know that the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise.
If you work with larger unified datasets then compression of the database is important from time to time when cleaning and re-unifying the data.
On the subject of noise, did you know that the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise.
If you don't see that there is nothing, then you are kidding yourself.
-
- I have made 50-60 posts
- Posts: 53
- Joined: Thu Feb 18, 2010 5:01 pm
- 14
- Full Name: Jon Adams
- Company Details: Forensic and Architectural Consulting
- Company Position Title: Director of 3D Reality Capture
- Country: USA
- Linkedin Profile: Yes
- Location: PA
Re: Data cleaning...what is the best method?
Yes, I am aware. Considering this as well. Actually just ran across the a topic here from 2014 when this may have been first suggested!the new Leica RTC360 has a feature that automatically deletes noise from people or objects passing in front of the scanner. It takes a double scan and compares both to see what changed, then removes the noise
So suggestions are:
Unify (Fast? Reduce?)
Clean
If slow, compact? Re-unify? (Fast?)
Thanks
- sreed
- V.I.P Member
- Posts: 257
- Joined: Mon Apr 28, 2014 4:31 pm
- 9
- Full Name: Scott Reed
- Company Details: Prologue Systems
- Company Position Title: Principal
- Country: USA
- Skype Name: sabotva
- Linkedin Profile: Yes
- Location: Richmond, VA USA
- Has thanked: 16 times
- Been thanked: 15 times
Re: Data cleaning...what is the best method?
I have an App we developed to do in Scene/Faro what the RTC360 does by double-scanning. PM me if you are interested in a demo trial. The App is a plugin for Scene.
- landmeterbeuckx
- V.I.P Member
- Posts: 1616
- Joined: Tue May 01, 2012 5:19 pm
- 11
- Full Name: Lieven Beuckx
- Company Details: Studiebureau Beuckx
- Company Position Title: Owner
- Country: Belgium
- Linkedin Profile: Yes
- Has thanked: 183 times
- Been thanked: 549 times
-
- I have made <0 posts
- Posts: 1
- Joined: Thu Jun 07, 2018 4:19 pm
- 5
- Full Name: Brandon B
- Company Details: BKF Engineers
- Company Position Title: PLS Project Surveyor
- Country: USA
- Linkedin Profile: Yes
Re: Data cleaning...what is the best method?
When you say "compact" are you referring to optimizing the database?
I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.
In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.
In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
- gsisman
- V.I.P Member
- Posts: 898
- Joined: Fri Oct 07, 2016 1:51 pm
- 7
- Full Name: Steve Long
- Company Details: Montgomery County DOT _ MD
- Company Position Title: Land Survey Supervisor
- Country: United States
- Skype Name: gsisman1
- Linkedin Profile: Yes
- Has thanked: 767 times
- Been thanked: 149 times
Re: Data cleaning...what is the best method?
There is a great terrain modeling from Scan Cloud program called QT Modeler (John Hopkins Applied Imagery Lab). They have recently added this feature at our request and other miltary clients. The side profile-defined corridor width-polygonal clip box. Check it out. The next request is an E57 unified import-as it only accepts LAS & PTS format exports from Leica Productsdirtysouthsurveyor wrote: ↑Thu Oct 04, 2018 7:35 pm When you say "compact" are you referring to optimizing the database?
I had a 3 mile stretch of an extremely busy roadway that we scanned at night. It was a nightmare cleaning that thing up with region grow because of how quickly it would break the unification. I ended up sectioning off parts of the road with the limit box, flip to ortho profile few then fenced in everything approximately 0.1' above the surface and deleted inside fence.
In retrospect, i wouldn't have brought in every day of data into the same database. Definitely going to keep future projects in separate databases and then clean them as I go. Just not sure of how to register the clean scanworlds together...
http://appliedimagery.com/