Out of calibration? How do you know? What's the accuracy?

Discuss all FARO related issues here.
Post Reply
kramretsof
I have made <0 posts
I have made <0 posts
Posts: 6
Joined: Wed Sep 05, 2012 3:04 am
Full Name: Mark Foster
Company Details: None
Company Position Title: None
Country: NZ
Linkedin Profile: No

Out of calibration? How do you know? What's the accuracy?

Post by kramretsof » Sun Nov 10, 2019 6:18 am

So our S150 took a bit of a tumble (gust of wind tipped over the tripod / scanner hit the ground), resulting in superficial damage.... actually cracked the outer case.

I know I know.... we need to send it to Faro.... but... before we do, we thought it would be interesting to see if we could spot any difference in the scan data... pre / post event.

The exercise has raised a few questions and we thought it would be a good idea to reach out to the LSF community for comment / insights / learning.

First, let me explain how we carried out our comparison...

1) since we have an array of fixed scan positions for our spheres (via ATS washers installed on structural members inside a building), it was pretty easy to set up a comparison using an existing scan data as reference (carried out about 6 weeks prior to incident).

So we chose a particular scan location where:

A - there were at least 2 sphere positions located at no more than 10m away from the scanner (more like 6-7m away), and

B- there was no noticeable vibration, and

C - there were large structural members present (in x,y,z directions) within 10m distance from scanner - to be able to extract planes to run a fit comparison later

Note:

- The scanner settings used were the same as the reference scan (resolution set to 7.1mm @ 10m, quality set to 3, scan in standard colour).

- The scanner was set-up at approximately the same location as the reference scan - probably within +/- 1m in X & Y and +/- 0.25m in Z.

- We used ATS sphere kit, where spheres rated to +/- 1mm


2) After carrying out the scan, we performed a registration (in Scene 2019.1) using only 2 fixed spheres, (both fixed locations at no more than 6-7 m away from the scanner - as previously stated)

Note:

- We thought it would be wise to see what impact using the inclinometer would have (and whether it exhibited any strange behaviour), so we used 2 spheres only - and therefore relied on the inclinometer to lock-in the remaining degrees of freedom.

- There was no noticeable 'stitch line' (after out-of-the-box inclinometer compensation applied) between the two hemispheres in the newly captured scan data (an issue we knew to look out for if the scanner was supposedly out of calibration), and we confirmed that this was consistent for the reference scan as well.

Here's the registration values (from scan manager in Scene):

scan manager.jpg

(image displays the resulting 'mean' and 'max' distance error of 1.09mm for the registration)


3) We then used this registration to carry out a cloud-to-cloud distance comparison (using CloudCompare software - awesome by the way!), where we observed the following:

cloudcompare data incl data beyond 10m.jpg

(Image above shows deviation in the C2C absolute distances up to 5mm)

But this is a bit misleading since the scan resolution becomes a factor here, as we've scanned using "7.1mm @ 10m" resolution.


4) So we then decided to clip the data to 10m radius from the scanner, to be aligned with Faro's accuracy statement for the S150 (+/- 1mm @ 10m).

cloudcompare data.jpg
cloudcompare data histogram.jpg
(The histogram displays the C2C absolute distances, where the mean value is just under 1.5mm)

Up to this point the result certainly doesn't show us any glaring issues with the scanner, well at least not when you consider the error stack-up in our test scenario.

5) But..... what is the expected accuracy for our test scenario ??? .... when:

A - The Faro S150 unit's stated accuracy is +/- 1mm @ 10m (and the spheres used in the registration are no more than 7m away from the scanner)

B - We are using fixed spheres as reference (mounted to ATS washers on structural members), using ATS spheres rated at +/- 1mm accuracy

C - The max distance error in the registration (using our 2 fixed sphere locations) was 1.09mm

Note:

- We got a bit confused about 'sphere fit error' though. Should this be taken into account as well, or does distance error account for this???

- The sphere fit error (calculated in Scene) for our 2 fixed reference sphere positions was as follows:

Sphere A: +/- 0.36mm (Ref scan), and +/- 0.22mm (test scan)
Sphere B: +/- 0.77mm (Ref scan), and +/- 0.26mm (test scan)

D - The scan data is clipped to 10m, as there is no reported scan accuracy stated for the S150 unit beyond this distance (well at least not that we've found in spec sheets online, etc)


6) To go one step further, we also decided to extract reference features (planes) from both datasets in the registration to see how these compared.

Note:

- A reference plane was extracted (from large structural beam faces) in each of the main scan directions (x,y,z).

- Each reference feature is within a 10m radius from the scanner

- Each reference plane had at least 500 scan sample points, and was extracted with precision using the 'polygon select' tool instead of the 'expand plane' tool.

Here is the result of this comparison:
cloudcompare plane data comparison.jpg
(The images above show that the bulk of the data (of the test scan) lies within +/- 0.5mm of the reference scan data)

So what should we make of this? This supports that the data looks really good (for the three features within a 10m radius from the scanner)

7) The next test we would like to carry out would be to get a single-point survey done (as a benchmark) for beam faces scanned beyond the 10m range, say at intervals up to about 80m from the scanner position.

Note, our thinking here is that we can rely on plane extraction tools in scene for this comparison (lots of sample points), as we can't register spheres beyond 20m or so... but the reality is that we wouldn't know what to expect from this test, given we don't have any accuracy info from Faro for the S150 beyond 10m....

thoughts??

This whole exercise has opened a can of worms for us.... in particular around:

A - how do you even confirm when your scanner is out of calibration? (Our test certainly doesn't indicate that it is.... but then again, can we really conclude anything for data observed beyond 10m from the scanner - given the logic in our testing - but please do let us know if our logic is indeed flawed)

B - Is it even legitimate to assume that if your scanner takes a knock it must be immediately sent back for a calibration - it doesn't seem to stack up from our testing - but perhaps we are flawed in our approach - are there other factors we haven't considered?

C - For our test scenario, if we are evaluating the position of an item, or say measuring the distance between two points (within a 10m radius from the scanner) what is the correct statement on accuracy, given:

- The Faro S150 unit's stated accuracy is +/- 1mm @ 10m

- Our registration is using 2 spheres only, and they are each rated at +/- 1mm accuracy (by manufacturer)

- The max distance error (calculated in Scene) for the registration (using our 2 fixed sphere locations) was 1.09mm

- The sphere fit error (calculated in Scene) for our 2 fixed reference sphere positions was as follows:

Sphere A: +/- 0.36mm (Ref scan), and +/- 0.22mm (test scan)
Sphere B: +/- 0.77mm (Ref scan), and +/- 0.26mm (test scan)

D - How would we even begin to state the expected accuracy on a linear measurement between two points from a single scan if both points are beyond 10m from the scanner, say in opposite directions, what could we reasonably expect? Say at 20m, 50m, 100m ??

E - We haven't even considered reflectivity either....

A can of worms we have opened..... hope some of you experts out there can provide us with a little bit of insight and clarity. Sorry to be so long-winded....

cheers

Mark
You do not have the required permissions to view the files attached to this post.

Post Reply

Return to “FARO”