Calibration / Accuracy verification

Discuss all Aerospace related issues here.
Post Reply
User avatar
JosephChumbley
I have made 10-20 posts
I have made 10-20 posts
Posts: 11
Joined: Tue May 13, 2008 9:21 pm
Full Name: Joseph Chumbley
Company Details: Lockheed Martin Space Systems Co
Company Position Title: Mech Eng Senior
Country: USA

Calibration / Accuracy verification

Post by JosephChumbley » Thu Apr 15, 2010 10:09 pm

I'm interested in learning how the scanning community is tackling the issue of verification. We need to certify the accuracy of the data we collect, and spend a lot of time with on-site calibration and accuracy verification...

Running Dim Control surveys is great for verifying the control that's used, but how do you verify the scan data? what certifications are you signing off on? I'm an old RLS who prior to Aerospace, spent 20+yrs in heavy civil / industrial settings. I had to certify the accuracy of the data we collected in-house, and stuff that was given to us by subcontractors.

With conventional survey equipment, calibration and certification processes are pretty much set in stone.

What's the trend regarding laser scanners and their data sets?

User avatar
Garth
I have made 20-30 posts
I have made 20-30 posts
Posts: 20
Joined: Fri Jan 15, 2010 2:58 am
Full Name: Garth Archibald
Company Details: GNS Science
Company Position Title: Engineering Geological Surveyor
Country: New Zealand
Location: Wellington, New Zealand
Contact:

Re: Calibration / Accuracy verification

Post by Garth » Fri Apr 16, 2010 6:09 am

Hi Joseph,

I would guess you are not the only Scanner struggling with this issue.

My approach is that it is not the scan data but the derived surfaces that must have errors assigned as the pointcloud data is not the deliverable but it is the volumes, sections, models etc that are important.

I look forward to others responses on this crucial issue.

Regards Garth

MarcDB
I have made 10-20 posts
I have made 10-20 posts
Posts: 13
Joined: Mon Oct 26, 2009 3:25 pm
Full Name: Marc De Bruyne
Company Details: Grontmij Industry
Company Position Title: Technology Coordinator
Country: Belgium
Linkedin Profile: Yes
Location: Belgium
Contact:

Re: Calibration / Accuracy verification

Post by MarcDB » Fri Apr 16, 2010 8:29 am

Joseph,

Glad that somebody is starting up this discussion.
Somebody told me once: Redundancy is King...
And this statement never got out of my mind.
Quality is something that concerns me for 10 years now...talking about point cloud data.
We are working in the Process&Power business, and I always work with Service providers, even for the post processing.
We always do our engineering with point clouds; modeling is not an option for me.
The reasons why are all obvious for me, but not a part of this topic.
So...Quality control.
For me a laser scanner is an equipment just like a TPS: it delivers point measurements in XYZ.
A bit more points than a TPS, but I don’t know why I shouldn’t expect something less for a Laser scanning point.
I have to know the Quality, tolerance, environment situation (eg temperature, vibrations, etc)

Some guy's give me the answer:
- My scanner is been calibrated every year....
- My scanning software will generate you a deviation report.

But what does this really give me as a end customer?
To be honest, for me ...a "pretty picture" of an environment.. :-)
I can assure you, the early years we had some drama's with wrong data, fuzzy data, what ever.
N this cases the software reported “excellent data” 
And I had no grip on it; once data was delivered: is this data really OK, Y/N?
In fact I was manually rechecking almost every step my service provider had done..digging in to the point cloud to be sure that there were no errors in.

So for me there's no compromising any more on Quality:
- Never scanning without TPS assistance
- For every area Plant benchmarks are set by the Surveyor that is responsible for the plant (typically the in house contractor)
- All targets are measured by TPS (minimum 4 in sight per scan)
- Scans are consolidated based on the TPS coordinates, and leveled out afterwards as an extra action by the scan software
-A xyz report from the targets is extracted out of the final consolidated point cloud.
-This xyz report is merged with the xyz report out of the TPS and there is a third comprising table generated.

So once my service provider delivers a hard disk to me, the only thing I check before I give the scan to my project team is this Q-report…and I can put my feet’s on the desk. (as a matter of speaking )

Because I know 2 things:
-The scanner is reliable
-The scan data is reliable, and will guaranty a perfect new integrated 3D design in the existing plant.

User avatar
Matt Young
Global Moderator
Global Moderator
Posts: 3785
Joined: Fri Nov 30, 2007 3:03 pm
Full Name: Matt Young
Company Details: Baker Hicks
Company Position Title: CAD-BIM Lead
Country: UK
Linkedin Profile: No

Re: Calibration / Accuracy verification

Post by Matt Young » Fri Apr 16, 2010 8:59 am

Joseph,

I could not explain it better.

Everything that we scan within industrial environments goes through the same process. We end up with a fully reportable control file that verifies the registration of the cloud data. When we deliver our point clouds we can say with confidence that the data is accurate and we can state how accurate.

Although it is not always required, we use the same methodology for projects other than the industrial ones.

Making sure that data is accurately measured and recorded is key to the success of laser scanning.

Matt

MarcDB
I have made 10-20 posts
I have made 10-20 posts
Posts: 13
Joined: Mon Oct 26, 2009 3:25 pm
Full Name: Marc De Bruyne
Company Details: Grontmij Industry
Company Position Title: Technology Coordinator
Country: Belgium
Linkedin Profile: Yes
Location: Belgium
Contact:

Re: Calibration / Accuracy verification

Post by MarcDB » Fri Apr 16, 2010 10:08 am

Matt,

Of course I work in industrial environments, where any inconsistency or clash can cost millions.

So, I don't want to claim that I'm totally against cloud to cloud registration, or Target to target registration.
If this is "fit for use" ...why not.
If I would be in Film industry, I don’t think my customer would be so concerned in the exact dimensions of the BAT-mobile? :-) In that business there are other kinds of complex HQ requirements than we need.

My point is that I can't take the risk, too much involved in industrial environments.

So yes, in the early days of the HDS4500 we blasted 60/70 scans per day...now the maximum is 25. :-)
Not always commercial defendable of course....but everybody knows, as long as we all are using the same equipment and software tools, higher quality always has to be paid.

User avatar
Matt Young
Global Moderator
Global Moderator
Posts: 3785
Joined: Fri Nov 30, 2007 3:03 pm
Full Name: Matt Young
Company Details: Baker Hicks
Company Position Title: CAD-BIM Lead
Country: UK
Linkedin Profile: No

Re: Calibration / Accuracy verification

Post by Matt Young » Fri Apr 16, 2010 10:39 am

Its nice to see that more and more people in the industry are on the same page and that laser scanning has reached its maturity. :)

This will be good for us all. :D

User avatar
JosephChumbley
I have made 10-20 posts
I have made 10-20 posts
Posts: 11
Joined: Tue May 13, 2008 9:21 pm
Full Name: Joseph Chumbley
Company Details: Lockheed Martin Space Systems Co
Company Position Title: Mech Eng Senior
Country: USA

Re: Calibration / Accuracy verification

Post by JosephChumbley » Tue May 04, 2010 11:04 pm

I'm glad to see others interested in this topic... Verification is a sticky issue...

I agree that running Dimensional Control Surveys to the targets allows a good 'apples to apples' comparison to the target locations (checking how close the scanner measurements compare with the Control Survey work)... But that's only a few points out of the 10's of millions of points gathered by the scanner at each set-up... How do you certify the accuracy for the remainder of this data?

In example, if it's a critical area, we will gather discrete measurements using conventional equipment (reflectorless, 1 second total station) for 60 - 100 or more points spread randomly throughout the scan area. Then by overlaying measurements on the scan data, we can come up with a calculated spread on the accuracy. This way we have a defensible statistical method for calculating an accuracy statement that does not rely on just the targets, but the actual elements being measured...

Let's face it, different surface types (material, paint, etc.) give different measurement results, distance from the scanner effects accuracy, and then you get to factor in proper warm-up time on the laser, atmospheric conditions, etc... All these need to be properly accounted for, and are major factors to the resultant Statement of Accuracy.

Any input on this type of validation effort would be welcome... We are shooting for final registered scan data that is in the "overall" millimeter accuracy ranges... relative accuracy (areas in close proximity) is easy to get, but when you start comparing a measurment taken 2 feet from the scanner vs one thats 150 feet away, things get tougher.

US Federal standards require validation on conventional survey equipment at the start and end of a project. And if the project lasts longer, accuracy validation is required during the length of the project (typically a minimum of weekly verifications). Is anyone going to this effort with their scanners?

User avatar
Nick M
I have made 50-60 posts
I have made 50-60 posts
Posts: 55
Joined: Wed Jul 22, 2009 2:34 am
Full Name: Nicholas McKelvey
Company Details: Wolter Consulting
Company Position Title: Senior Surveyor- Spatial Specialist
Country: Australia

Re: Calibration / Accuracy verification

Post by Nick M » Fri May 07, 2010 4:09 am

To expand on the validation of scanner accuray I agree with all the posts above inregards to utilising conventional survey methods to provide control and sample data to confirm accuracies. Before commencing critcial projects and after transportation we use a target range (6+ targets setup in our carpark with even distribution through the full field of view of the scanner). This allows us for a quick verification of scanner performance on a repeatabile site.

We have also run our scanner over a baseline range that is setup for conventional survey instruments which is setup and monitored by our lands department to ensure all survey equipment is accurate and certified for use and performing as specified. The scanner passed this test and it has proved useful when speaking to clients that this test has been done as the accuracies are proven in a familair and approved method.

On the flip side, whilst as a sureyor I wish to ensure desired accuracies are acheived at all times this can run into extremes. For example with GPS you don't calculate the algorithims manually to ensure the accuracies, nor do you critique the pole location and level for conventional single point locations (you have to have some faith in the machine and operators).

I believe as long as you have a sound understanding of the equipment and it's inherinet accuracies (and the project requirements) and have enough redundancy in your scanning and target aquisitions to prove accuracy all will be fine.

The hard part is knowing where to draw that fine line!

Post Reply

Return to “Aerospace Industry”