e57 to rgb conversion

Discuss all FARO related issues here.
Post Reply
mawall
I have made <0 posts
I have made <0 posts
Posts: 3
Joined: Mon Apr 27, 2020 10:08 pm
Full Name: Marcus
Company Details: Imperial College London
Company Position Title: Research Assistant
Country: UK
Has thanked: 1 time
Been thanked: 1 time

e57 to rgb conversion

Post by mawall »

I want to convert unorganised point clouds in .e57 format into panoramic RGB images for further processing. The scans have been made using a FARO Focus 3D X 330. I'm using numpy and pye57 in the below Python snippet:

Code: Select all

pos = e57.scan_position(0).squeeze()
ox, oy, oz = pos[0], pos[1], pos[2]
data = e57.read_scan(0, colors=True)
xd = np.asarray(data["cartesianX"]) - ox
yd = np.asarray(data["cartesianY"]) - oy
zd = np.asarray(data["cartesianZ"]) - oz
coords = np.asarray(list(zip(xd, yd, zd)))

r = np.sqrt(np.sum(np.square(coords), axis=1))
yaw = np.degrees(np.arccos(xd/np.sqrt(np.square(xd) + np.square(zd))) * np.asarray([-1 if b else 1 for b in zd < 0]))
pitch = np.degrees(np.arccos(zd/r))
Afterwards, I combine the converted polar points with their respective RGB values and manually reshape to restore the image aspect ratio. This is the result:
Image

I'm trying to understand where the distortions arise and how I can get get rid of them.

[Update: The below plots show unexpected patterns due to an error in my computation of yaw and pitch. This turned out to be unrelated to the actual problem of getting rid of the above distortions. The correct way to compute yaw and pitch is of course: yaw = np.arctan2(yd, xd); pitch = np.arcsin(zd/r)]

Plotting the yaw and pitch values yields an unexpected result:
Image

The pitch values themselves behave as one might expect and run through to 160 degrees before dropping to 0:
Image Image

The yaw values however exhibit another strange pattern:
Image Image
The second plot shows points 460,000 to 480,000. Given the pitch range of 160 degrees, I would expect monotonously increasing yaw values.

I am wondering wheter the above patterns arise due to the working mechanism of the scanner or perhaps because some datapoints are lost on export to an unorganised format like e57. Unfortunately I have never operated a scanner myself and don't know if the scanner does a 180 degree yaw rotation in total, while recording full pitch rotations (e.g. 2*160), or if it rotates a full 360 degrees yaw while recording a window of e.g. 160 degrees pitch. [I hope this explanation is somewhat understandable.]

The complete e57 header:

Code: Select all

<StringNode 'guid'>: 5523356c-8b4c-4254-a5a2-ba227fa54321
<StringNode 'name'>: 19179_Visit2_000
<StringNode 'description'>: <AttrContainer>
<Attr type="String">
<Name value="'Name'" />
<Value value="'E57ExportScanInfos'" />
</Attr>
<Attr type="SVec2i">
<Name value="'OverlapHalf1'" />
<Value value="16 2515 " />
</Attr>
<Attr type="SVec2i">
<Name value="'OverlapHalf2'" />
<Value value="0 0 " />
</Attr>
</AttrContainer>

<StringNode 'sensorVendor'>: FARO Scanner Production GmbH
<StringNode 'sensorModel'>: FARO Focus 3D X 330
<StringNode 'sensorSerialNumber'>: LLS071406053
<StringNode 'sensorSoftwareVersion'>: 5.2.4.50259
<StringNode 'sensorFirmwareVersion'>: "iQLib" 5.5.8.50259
<StructureNode 'indexBounds'>
    <IntegerNode 'rowMinimum'>: 0
    <IntegerNode 'rowMaximum'>: 1084
    <IntegerNode 'columnMinimum'>: 0
    <IntegerNode 'columnMaximum'>: 2530
    <IntegerNode 'returnMinimum'>: 0
    <IntegerNode 'returnMaximum'>: 0
<StructureNode 'intensityLimits'>
    <ScaledIntegerNode 'intensityMaximum'>
    <ScaledIntegerNode 'intensityMinimum'>
<StructureNode 'colorLimits'>
    <IntegerNode 'colorRedMaximum'>: 255
    <IntegerNode 'colorRedMinimum'>: 0
    <IntegerNode 'colorGreenMaximum'>: 255
    <IntegerNode 'colorGreenMinimum'>: 0
    <IntegerNode 'colorBlueMaximum'>: 255
    <IntegerNode 'colorBlueMinimum'>: 0
<StructureNode 'cartesianBounds'>
    <FloatNode 'xMinimum'>: -8.334406852722168
    <FloatNode 'xMaximum'>: 1.3688486814498901
    <FloatNode 'yMinimum'>: -8.993896484375
    <FloatNode 'yMaximum'>: 24.559860229492188
    <FloatNode 'zMinimum'>: -4.53898811340332
    <FloatNode 'zMaximum'>: 8.431367874145508
<StructureNode 'sphericalBounds'>
    <FloatNode 'rangeMinimum'>: 0.38969722390174866
    <FloatNode 'rangeMaximum'>: 27.12574577331543
    <FloatNode 'elevationMinimum'>: -1.0913551474529104
    <FloatNode 'elevationMaximum'>: 1.565733700672494
    <FloatNode 'azimuthStart'>: 6.2682989444847985
    <FloatNode 'azimuthEnd'>: -0.0023119976631008877
<StructureNode 'pose'>
    <StructureNode 'rotation'>
        <FloatNode 'w'>: 0.9689291496689083
        <FloatNode 'x'>: 0.0005766788550809741
        <FloatNode 'y'>: 0.0024771845580909358
        <FloatNode 'z'>: -0.24732536044661868
    <StructureNode 'translation'>
        <FloatNode 'x'>: 569.7579212241267
        <FloatNode 'y'>: 619.4259505796962
        <FloatNode 'z'>: 83.7357981130912
<StructureNode 'pointGroupingSchemes'>
    <StructureNode 'groupingByLine'>
        <StringNode 'idElementName'>: columnIndex
        <CompressedVectorNode 'groups'>
<CompressedVectorNode 'points'>
Last edited by mawall on Tue May 19, 2020 3:40 pm, edited 1 time in total.
User avatar
infrared_sight
I have made 10-20 posts
I have made 10-20 posts
Posts: 13
Joined: Thu Aug 17, 2017 12:18 pm
Full Name: Charles Thomson
Company Details: Correvate - Developers of Vercator
Company Position Title: Product Manager
Country: UK
Linkedin Profile: Yes
Location: London
Has thanked: 2 times
Contact:

Re: e57 to rgb conversion

Post by infrared_sight »

Hi Marcus,

Correct me if I'm wrong, but you say that you are converting unorganised point clouds in .e57 format but the example looks structured or organised in the 'grid' it was captured in. E57 can hold either type but your header information example implies that the structure exists. In that case you should be able to extract the equi-rectangular image as the data is already stored in the grid you wish to generate.

Code: Select all

<StructureNode 'indexBounds'>
    <IntegerNode 'rowMinimum'>: 0
    <IntegerNode 'rowMaximum'>: 1084
    <IntegerNode 'columnMinimum'>: 0
    <IntegerNode 'columnMaximum'>: 2530
The following line states that the point data is stored in blocks of columnIndex so iterating through these should give you each column in the grid/image

Code: Select all

<StructureNode 'groupingByLine'>
        <StringNode 'idElementName'>: columnIndex
Charles Thomson, Product Manager
Vercator software - fast, automatic point cloud processing
Cloud • Mobile • Free trial and more at www.vercator.com
User avatar
Daniel Wujanz
I have made 100> posts
I have made 100> posts
Posts: 108
Joined: Fri Aug 24, 2018 11:26 am
Full Name: Daniel Wujanz
Company Details: technet GmbH
Company Position Title: 3D Laser Scanning Specialist
Country: Germany
Linkedin Profile: Yes
Location: Berlin
Has thanked: 44 times
Been thanked: 41 times
Contact:

Re: e57 to rgb conversion

Post by Daniel Wujanz »

Dear Marcus,

panorama scans are captured in two faces, hence a 2 x 180° scan. The pattern you're seeing is most likely not something that arises from the scanner but from how the data was added to the file. Most manufacturers write funny and sometimes contradictory information to the header of E57 files.

If Charly suggestion will not help (due to the file's potentially funny content), this may do the trick:
The easiest solution to solve this issue is to convert the Cartesian points into polar values. Then you have to create a matrix that will contain the points. The dimensions in the polar world are 2 x pi in horizontal direction and pi in the other. Look into the header of the scan to retrieve the angular increments of the scan. Divide the horizontal and vertical "polar" range by the resolution and you've got the according size of the resulting image matrix. Fill the points into the matrix and receive the glory you deserve ; )

All the best

Daniel
jamesworrell
V.I.P Member
V.I.P Member
Posts: 389
Joined: Mon Jun 16, 2014 1:45 pm
Full Name: James Worrell
Company Details: Bennett and Francis
Company Position Title: Director
Country: Australia
Linkedin Profile: Yes
Location: Brisbane, Queensland, Australia
Been thanked: 31 times
Contact:

Re: e57 to rgb conversion

Post by jamesworrell »

Daniel Wujanz wrote: Tue Apr 28, 2020 2:12 pm Fill the points into the matrix and receive the glory you deserve ; )
aka do your image classifications and start feeding it to the ML algorithm ;-p

Also code for "further processing" in the initial post ...

It was a feature request to Leica a while back to export sphericals based on RGB and/or intensity.

Just a guess.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 16 times
Been thanked: 63 times
Contact:

Re: e57 to rgb conversion

Post by jedfrechette »

mawall wrote: Tue Apr 28, 2020 1:18 amperhaps because some datapoints are lost on export
I think this is your problem. All scans are going to have some invalid data, and it doesn't look like you're accounting for that when building your array. I haven't used pye57, but from the readme it looks like there is an alternate method read_scan_raw that you could use instead of read_scan that will return all points including the invalid ones. I suspect, however, this will only work if you have organized data. As Charles noted though it looks like that's what you have. If you are working with organized data then your job is much simpler, you should be able to construct the array ahead of time then just fill in the valid point values based on row and column indexes.

If you do need to ingest unstructured data things get tricker, but I would always assume you that you have missing points.
Daniel Wujanz wrote: Tue Apr 28, 2020 2:12 pm panorama scans are captured in two faces, hence a 2 x 180° scan.
This is worth highlighting since it is a somewhat subtle point that you may need to consider after you get the correct image array built. Each scan face is actually slightly larger that 180°, so what you get instead of a single continuous 0-360 panorama are two roughly 0-181 panoramas next to each other with a small amount of overlap. See the center of a similarly constructed image array below.
overlap.jpg
You do not have the required permissions to view the files attached to this post.
Jed
mawall
I have made <0 posts
I have made <0 posts
Posts: 3
Joined: Mon Apr 27, 2020 10:08 pm
Full Name: Marcus
Company Details: Imperial College London
Company Position Title: Research Assistant
Country: UK
Has thanked: 1 time
Been thanked: 1 time

Re: e57 to rgb conversion

Post by mawall »

Thank you Charles, Daniel and Jed! Your input was immensely helpful.

I am relatively new to the subject of laser scanning and did indeed make two important wrong assumptions: That my data would be unorganised and that no datapoints would be missing.

Turns out I should have had another look at the file format specifications and pye57. We can get row and column ids by specifying 'row_column=True' when reading the scan:

Code: Select all

data = e57.read_scan(idx, colors=True, row_column=True)
Creating a zero-filled array and filling in based on row and column indices yields the desired result:
Image

There are quite a few missing/invalid data points in this particular scan, due to reflections off the window it seems, causing the twisting effects above:
Image

It seems there is little or no overlap of the two scan faces in this example, but it's good to be aware of this issue!

And yes, you got me James ; )
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 16 times
Been thanked: 63 times
Contact:

Re: e57 to rgb conversion

Post by jedfrechette »

mawall wrote: Tue Apr 28, 2020 6:45 pm It seems there is little or no overlap of the two scan faces in this example
Are you sure about that? ;) The (angular) amount of overlap is going to be very small, but if there isn't any your scans are likely broken in some other fun and interesting way.
pano_fixed.png
Glad we could help.
You do not have the required permissions to view the files attached to this post.
Jed
mawall
I have made <0 posts
I have made <0 posts
Posts: 3
Joined: Mon Apr 27, 2020 10:08 pm
Full Name: Marcus
Company Details: Imperial College London
Company Position Title: Research Assistant
Country: UK
Has thanked: 1 time
Been thanked: 1 time

Re: e57 to rgb conversion

Post by mawall »

Oh, right. It's even more visible at the bottom end of the board. Thanks for pointing this out! Good to know that this is to be expected for every scan then.
User avatar
Daniel Wujanz
I have made 100> posts
I have made 100> posts
Posts: 108
Joined: Fri Aug 24, 2018 11:26 am
Full Name: Daniel Wujanz
Company Details: technet GmbH
Company Position Title: 3D Laser Scanning Specialist
Country: Germany
Linkedin Profile: Yes
Location: Berlin
Has thanked: 44 times
Been thanked: 41 times
Contact:

Re: e57 to rgb conversion

Post by Daniel Wujanz »

Another advantage of using polar coordinates for the image creation is a) you don't see these "overlapping offsets" because of the 181-ish° rotation and b) you do not have issues because of "lost" points, where the successive pixels are consequently shifted. B.t.w. The angular matrix has always empty cells which yield in white or black pixels in the resulting images - some more, some less and of course in dependence to the applied scanner.
jedfrechette
V.I.P Member
V.I.P Member
Posts: 988
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 16 times
Been thanked: 63 times
Contact:

Re: e57 to rgb conversion

Post by jedfrechette »

Daniel Wujanz wrote: Wed Apr 29, 2020 7:38 am Another advantage of using polar coordinates for the image creation is a) you don't see these "overlapping offsets" because of the 181-ish° rotation
I guess it depends on what you're trying to achieve. For visualization that certainly makes sense, but for our most typical use cases where I'm more interested in the pixels as data, it makes more sense to convert the gridded scan topology directly to a pixel array rather than interpolating values on to a new pixel grid based on polar coordinates. I'd expect to see edge effects at the seam even with interpolation so I it seems simpler to just treat the left and right halfs as separate images.
Jed
Post Reply

Return to “FARO”