Out of calibration? How do you know? What's the accuracy?

Discuss all FARO related issues here.
Post Reply
kramretsof
I have made <0 posts
I have made <0 posts
Posts: 9
Joined: Wed Sep 05, 2012 3:04 am
Full Name: Mark Foster
Company Details: None
Company Position Title: None
Country: NZ
Linkedin Profile: No
Been thanked: 2 times

Out of calibration? How do you know? What's the accuracy?

Post by kramretsof »

So our S150 took a bit of a tumble (gust of wind tipped over the tripod / scanner hit the ground), resulting in superficial damage.... actually cracked the outer case.

I know I know.... we need to send it to Faro.... but... before we do, we thought it would be interesting to see if we could spot any difference in the scan data... pre / post event.

The exercise has raised a few questions and we thought it would be a good idea to reach out to the LSF community for comment / insights / learning.

First, let me explain how we carried out our comparison...

1) since we have an array of fixed scan positions for our spheres (via ATS washers installed on structural members inside a building), it was pretty easy to set up a comparison using an existing scan data as reference (carried out about 6 weeks prior to incident).

So we chose a particular scan location where:

A - there were at least 2 sphere positions located at no more than 10m away from the scanner (more like 6-7m away), and

B- there was no noticeable vibration, and

C - there were large structural members present (in x,y,z directions) within 10m distance from scanner - to be able to extract planes to run a fit comparison later

Note:

- The scanner settings used were the same as the reference scan (resolution set to 7.1mm @ 10m, quality set to 3, scan in standard colour).

- The scanner was set-up at approximately the same location as the reference scan - probably within +/- 1m in X & Y and +/- 0.25m in Z.

- We used ATS sphere kit, where spheres rated to +/- 1mm


2) After carrying out the scan, we performed a registration (in Scene 2019.1) using only 2 fixed spheres, (both fixed locations at no more than 6-7 m away from the scanner - as previously stated)

Note:

- We thought it would be wise to see what impact using the inclinometer would have (and whether it exhibited any strange behaviour), so we used 2 spheres only - and therefore relied on the inclinometer to lock-in the remaining degrees of freedom.

- There was no noticeable 'stitch line' (after out-of-the-box inclinometer compensation applied) between the two hemispheres in the newly captured scan data (an issue we knew to look out for if the scanner was supposedly out of calibration), and we confirmed that this was consistent for the reference scan as well.

Here's the registration values (from scan manager in Scene):

scan manager.jpg

(image displays the resulting 'mean' and 'max' distance error of 1.09mm for the registration)


3) We then used this registration to carry out a cloud-to-cloud distance comparison (using CloudCompare software - awesome by the way!), where we observed the following:

cloudcompare data incl data beyond 10m.jpg

(Image above shows deviation in the C2C absolute distances up to 5mm)

But this is a bit misleading since the scan resolution becomes a factor here, as we've scanned using "7.1mm @ 10m" resolution.


4) So we then decided to clip the data to 10m radius from the scanner, to be aligned with Faro's accuracy statement for the S150 (+/- 1mm @ 10m).

cloudcompare data.jpg
cloudcompare data histogram.jpg
(The histogram displays the C2C absolute distances, where the mean value is just under 1.5mm)

Up to this point the result certainly doesn't show us any glaring issues with the scanner, well at least not when you consider the error stack-up in our test scenario.

5) But..... what is the expected accuracy for our test scenario ??? .... when:

A - The Faro S150 unit's stated accuracy is +/- 1mm @ 10m (and the spheres used in the registration are no more than 7m away from the scanner)

B - We are using fixed spheres as reference (mounted to ATS washers on structural members), using ATS spheres rated at +/- 1mm accuracy

C - The max distance error in the registration (using our 2 fixed sphere locations) was 1.09mm

Note:

- We got a bit confused about 'sphere fit error' though. Should this be taken into account as well, or does distance error account for this???

- The sphere fit error (calculated in Scene) for our 2 fixed reference sphere positions was as follows:

Sphere A: +/- 0.36mm (Ref scan), and +/- 0.22mm (test scan)
Sphere B: +/- 0.77mm (Ref scan), and +/- 0.26mm (test scan)

D - The scan data is clipped to 10m, as there is no reported scan accuracy stated for the S150 unit beyond this distance (well at least not that we've found in spec sheets online, etc)


6) To go one step further, we also decided to extract reference features (planes) from both datasets in the registration to see how these compared.

Note:

- A reference plane was extracted (from large structural beam faces) in each of the main scan directions (x,y,z).

- Each reference feature is within a 10m radius from the scanner

- Each reference plane had at least 500 scan sample points, and was extracted with precision using the 'polygon select' tool instead of the 'expand plane' tool.

Here is the result of this comparison:
cloudcompare plane data comparison.jpg
(The images above show that the bulk of the data (of the test scan) lies within +/- 0.5mm of the reference scan data)

So what should we make of this? This supports that the data looks really good (for the three features within a 10m radius from the scanner)

7) The next test we would like to carry out would be to get a single-point survey done (as a benchmark) for beam faces scanned beyond the 10m range, say at intervals up to about 80m from the scanner position.

Note, our thinking here is that we can rely on plane extraction tools in scene for this comparison (lots of sample points), as we can't register spheres beyond 20m or so... but the reality is that we wouldn't know what to expect from this test, given we don't have any accuracy info from Faro for the S150 beyond 10m....

thoughts??

This whole exercise has opened a can of worms for us.... in particular around:

A - how do you even confirm when your scanner is out of calibration? (Our test certainly doesn't indicate that it is.... but then again, can we really conclude anything for data observed beyond 10m from the scanner - given the logic in our testing - but please do let us know if our logic is indeed flawed)

B - Is it even legitimate to assume that if your scanner takes a knock it must be immediately sent back for a calibration - it doesn't seem to stack up from our testing - but perhaps we are flawed in our approach - are there other factors we haven't considered?

C - For our test scenario, if we are evaluating the position of an item, or say measuring the distance between two points (within a 10m radius from the scanner) what is the correct statement on accuracy, given:

- The Faro S150 unit's stated accuracy is +/- 1mm @ 10m

- Our registration is using 2 spheres only, and they are each rated at +/- 1mm accuracy (by manufacturer)

- The max distance error (calculated in Scene) for the registration (using our 2 fixed sphere locations) was 1.09mm

- The sphere fit error (calculated in Scene) for our 2 fixed reference sphere positions was as follows:

Sphere A: +/- 0.36mm (Ref scan), and +/- 0.22mm (test scan)
Sphere B: +/- 0.77mm (Ref scan), and +/- 0.26mm (test scan)

D - How would we even begin to state the expected accuracy on a linear measurement between two points from a single scan if both points are beyond 10m from the scanner, say in opposite directions, what could we reasonably expect? Say at 20m, 50m, 100m ??

E - We haven't even considered reflectivity either....

A can of worms we have opened..... hope some of you experts out there can provide us with a little bit of insight and clarity. Sorry to be so long-winded....

cheers

Mark
You do not have the required permissions to view the files attached to this post.

kramretsof
I have made <0 posts
I have made <0 posts
Posts: 9
Joined: Wed Sep 05, 2012 3:04 am
Full Name: Mark Foster
Company Details: None
Company Position Title: None
Country: NZ
Linkedin Profile: No
Been thanked: 2 times

Re: Out of calibration? How do you know? What's the accuracy?

Post by kramretsof »

I thought this thread would have generated some interest, perhaps it was too long winded (sorry).... at the very least was hoping Faro might chime in on a few of my points, i.e:

************

"Is it even legitimate to assume that if your scanner takes a knock it must be immediately sent back for a calibration - it doesn't seem to stack up from our testing - but perhaps we are flawed in our approach - are there other factors we haven't considered?"

***********

For our test scenario, if we are evaluating the position of an item, or say measuring the distance between two points (within a 10m radius from the scanner) what is the correct statement on accuracy, given:


- The Faro S150 unit's stated accuracy is +/- 1mm @ 10m

- Our registration is using 2 spheres only, and they are each rated at +/- 1mm accuracy (by manufacturer)

- The max distance error (calculated in Scene) for the registration (using our 2 fixed sphere locations) was 1.09mm

- The sphere fit error (calculated in Scene) for our 2 fixed reference sphere positions was as follows:

Sphere A: +/- 0.36mm (Ref scan), and +/- 0.22mm (test scan)
Sphere B: +/- 0.77mm (Ref scan), and +/- 0.26mm (test scan)

Given info above, how would you state the accuracy of a linear measurement between two points within a 10m radius of the scanner, extracted from Scene?

****************

"How about the expected accuracy on a linear measurement between two points from a single scan if both points are beyond 10m from the scanner, say in opposite directions, what could we reasonably expect? Say at 20m, 50m, 100m ??"

***************

Thanks,

Mark

User avatar
Daniel Wujanz
I have made 70-80 posts
I have made 70-80 posts
Posts: 71
Joined: Fri Aug 24, 2018 11:26 am
Full Name: Daniel Wujanz
Company Details: technet GmbH
Company Position Title: 3D Laser Scanning Specialist
Country: Germany
Linkedin Profile: Yes
Location: Berlin
Has thanked: 11 times
Been thanked: 24 times
Contact:

Re: Out of calibration? How do you know? What's the accuracy?

Post by Daniel Wujanz »

Dear Mark,

your post definitely deserves a lot more attention! I will add the vital topic of calibration to the "taming errors" series and will discuss it in great detail. I see banana style data on a daily basis and it gives me the...

Most people claim a few millimetres in accuracy but are actually several mms off (without knowing it) due to poorly maintained scanners.

All the best

Daniel Wujanz

User avatar
sreed
V.I.P Member
V.I.P Member
Posts: 221
Joined: Mon Apr 28, 2014 4:31 pm
Full Name: Scott Reed
Company Details: Prologue Systems
Company Position Title: Principal
Country: USA
Skype Name: sabotva
Linkedin Profile: Yes
Location: Richmond, VA USA
Has thanked: 7 times
Been thanked: 9 times

Re: Out of calibration? How do you know? What's the accuracy?

Post by sreed »

Hi Mark,

Sorry you haven't gotten a lot of traction with your thoughtful and thorough post. I am just now seeing it and am very impressed. I like your approach, your invitation for suggestions, inquiry about specs and clear communication. I'm not qualified to respond other than through experience, but I cannot see any flaws in your assessment. Perhaps you are disqualifying data beyond 10m unnecessarily. I would certainly include and be interested in that data. I believe we can find ranging error specs on the S series beyond 10 mm (some factor per m, but i can't recall number - ~0.2mm/10m??)

Somewhat of a different take on the calibration question, you may want to consider it if only for peace of mind for you and your clients. We rarely are asked for calibration certificate, but have been and a few times its after a project is complete and delivered.

You might also want to check for collimation error by scanning on a large flat surface and inspecting the "seem" closely. This used to be a big issue, but less so now. And finally, check for wave pattern at base of tripod. Colorize by elevation with tight range and you should't see any meaningful wave at base, a phenomenon we've experienced when our instrument gets out of calibration.

Have you performed the onsite calibration routine?

kramretsof
I have made <0 posts
I have made <0 posts
Posts: 9
Joined: Wed Sep 05, 2012 3:04 am
Full Name: Mark Foster
Company Details: None
Company Position Title: None
Country: NZ
Linkedin Profile: No
Been thanked: 2 times

Re: Out of calibration? How do you know? What's the accuracy?

Post by kramretsof »

Hi Daniel and Scott, thanks for chiming in - I appreciate your input.

In regards to your points (Scott):

"Perhaps you are disqualifying data beyond 10m unnecessarily. I would certainly include and be interested in that data. I believe we can find ranging error specs on the S series beyond 10 mm (some factor per m, but i can't recall number - ~0.2mm/10m??)"

I've only disqualified the data beyond 10m because I wasn't sure what to expect beyond 10m (in terms of stated accuracy).... would be great to see some 'error specs' beyons 10m as you say.

"You might also want to check for collimation error by scanning on a large flat surface and inspecting the "seem" closely. This used to be a big issue, but less so now. And finally, check for wave pattern at base of tripod. Colorize by elevation with tight range and you should't see any meaningful wave at base, a phenomenon we've experienced when our instrument gets out of calibration"

We did have a look at the factors and did not see any of these behaviours. I've definitely seen this behaviour on our older S120 unit, but it didn't show up on the S150 in our testing.

cheers

Mark

Post Reply

Return to “FARO”