10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post any marterial related the future technology of laser scanning.
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

A new update on the 14TB helium filled HDD

05-04-2018 07-52-00 AM.jpg

Some interesting data from Backblaze on their helium vs air filled HDD

05-03-2018 02-02-50 PM.jpg

Backblaze 2018 Q1 Lifetime reliability for their HDD

05-01-2018 11-05-46 AM.jpg
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

Since I posted this recent information on May 4, 2018, there have been approximately 200+ views of this thread. Of this total number of viewers, only 17 have seen the valuable information on the 3 images that I extracted from the original sources since they were registered members of the LSF. Unfortunately the other 90% of the viewers are guests that are missing a significant amount of information that is being posted.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

While organizing some information on storage, some interesting comparisons between enterprise and desktop HDD from INTEL

05-19-2018 08-18-43 AM.jpg

https://www.intel.com/content/www/us/en ... ducts.html

https://www.intel.com/content/dam/suppo ... Ds_2.0.pdf
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by smacl »

Dennis, don't know if the cost would justify the benefit for your workflow, but have you considered getting a big RAM disk for storage purposes on the currently active job. By rights, most 64 bit software shouldn't need it and should use available RAM over disk, but this isn't always the case. RAM disk is about 4x - 6x speed of SSD but expensive. Not point cloud, but I use a large RAM disk on my dev environment and it removes a big bottleneck. Next spend for me will be putting together a decent thread ripper build, which should give me a serious performance boost at a reasonable price.
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

Shane

Thank you for the comment. We are using the Seagate SAS HDD (10TB, 12TB, currently at 220TB), in raid 6 to move files to workstations using our dual 10GbE office system. The AVAGO SAS raid 6 controllers give us large storage at about the same transfer speed as SSD at a significantly cheaper cost. We also have AVAGO SATA raid5 SSD sysytems at 7TB.

The main system that we use for processing, visualization, and development is a INTEL server with dual Xeon 3.5GHz for a total of 16 threads and 256GB of RAM of the total 1.5TB of space. We also have NVMe on several systems, but may take your suggestion and try the RAM disk. The price of DDR4 has dropped in the last year or two: 64GB LRDIMM @ $700 and 32GB LRDIMM @ $370, so I might increase one of the systems and try the RAM disk.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by smacl »

Sounds like a real beast of a set up Dennis. If your processing workflow is anyway CPU bound, the AMD threadripper could be worth considering in the future, as it offers 32 threads at a very reasonable price. At high levels of performance you really need to profile the system over time to see where the bottlenecks lie and even then they're prone to change. e.g. our own software uses as many CPU threads as it can get its hands on without using the GPU, but we're currently porting some of the more intensive processing to GPU. We're currently doing a lot of automated extraction on Pegasus 2 mobile LIDAR, with each data set seeming to be bigger than the last, so we're always trying to squeeze all we can out of the hardware. The crypto miners have been hoovering up all the decent GPUs of late but I'd guess that will change soon enough.
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

There is also space for 4 GPU boards on the system described above. We had NVIDIA GTX980Ti, Quadro 5000, and GTX1080Ti installed on it. Tracking most of the processing, it did not matter except to have one, so the GTX1080Ti is the only one left. Using Register360 only uses 6 threads, and if my memory is correct, Recap used approximately 25% GPU maximum.

Most of our previous testing was texturing large meshes, so we increased the RAM to the 256GB as a practical limit. It seems more, larger point clouds are being created. We used our Reigl VZ400i at pano20 (99Mp scan) to scan the outside of two Waikiki hotel towers and our two ZF5010X at high (40-50Mp) to scan the inside of two pent house floors in each tower. Total storage was 280 GB for raw and processed data including photos for all scans (120 scans in 8 hours). We now have a database of 20 scans in central Waikiki in full color since the Riegl VZ400i has a range of approximately 250 meters, with each scan position taking 180 seconds and the 7 36Mpixel photos 60 seconds.
Last edited by dhirota on Wed May 08, 2019 2:59 am, edited 1 time in total.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

This was from several years ago
dhirota wrote: Thu Feb 16, 2017 1:38 am Everyone has been speaking about larger SSDs in the last year. I do not know if this subject is FUTURE TECHNOLOGY or not, but many folks think rotating HDD will be gone soon. We have looked at creating our own internal cloud for many reasons and last year decided to extend what we had into faster, larger internal storage. We decided on a pair of INTEL servers, following the prototype workstation built last year using an INTEL server motherboard (dual Xeon, .25TB/1.5TB RAM, dual 10GbE, 64TB SAS RAID6 8TB Seagate HDD, 2-GTX1080 GPU).
.....
The Seagate 10TB ST10000NM0206 HDD are SAS vs. SATA which gives us 12Gbs on the Broadcom/AVAGO 9361-16i controllers that we are using. This gives us 100TB in Raid6 with 12 HDD on each server with dual 10GbE on each server. In the tests today, Robocopy is copying a 15TB block of projects at a speed of approximately 275MB/second over our internal 10GbE ethernet network at about 2-3Gbs total from 2 ports to another server with two ports.

We decided not to fill both servers with disks and wait until prices on the 10TB reduced due to the introduction of the 12 or 14TB HDD.
I always check on the Backblaze evaluation of their system storage, which is interesting for the detail that they include. They are indicating testing of some 20TB HDD.

05-07-2019 12-57-01 PM.jpg

I just purchased 30 each of 12TB Seagate SAS to replace our dependable 4TB and 6TB Seagate SAS drives like it shows in the table. The 14TB are still difficult to get and at significantly higher cost.

What is interesting from Jason's discussion in this thread is the use of 10GbE ports. We have decided to stay with our 10GbE ports, and will try to dual aggregate them to all switches. I do not think it will work with W10 or W7 for the workstations. A quick test between W10 and Ubuntu workstations using iperf3-6 showed transferring 10.9GB of information in 10 seconds at a rate of 9.4Gbs over a single port.

05-07-2019 01-32-02 PM.jpg

We did another experiment opening up and viewing a 17.6GB Civil3D file folder with about 3K items (2,651 files, 322 folders):

Operational Server, 8X10TB Raid6 Seagate SAS, dual 10GbE ports, 32GB RAM workstation with 10GbE port = 55 seconds
Experimental Server, 8X1TB Raid5 SanDisk, dual 10GbE ports, 32GB RAM workstation with 10GbE port = 47 seconds

We have looked at an all SSD server. Has anyone purchased or built their own? A Seagate 12TB HDD is $350, whereas a 2TB 860EVO SSD is $300.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
Jason Warren
Administrator
Administrator
Posts: 4224
Joined: Thu Aug 16, 2007 9:21 am
16
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: No
Location: Retford, UK
Has thanked: 443 times
Been thanked: 246 times
Contact:

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren »

Thank you for the update Dennis. I think this topic has become more relevant for timely back office data logistics, especially with the take up of the RTC360 :roll:
Jason Warren
Co_Founder

Dedicated to 3D Laser Scanning
LaserScanningForum
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by smacl »

dhirota wrote: Wed May 08, 2019 2:26 amWe have looked at an all SSD server. Has anyone purchased or built their own? A Seagate 12TB HDD is $350, whereas a 2TB 860EVO SSD is $300.
Just thinking out loud here, so bear with me. Will replacing slower drives with faster drives on the server have the net effect of shifting the bottleneck to the network as multiple users read and write very large files on an ongoing basis? Depends on the workflow, but for me something like a fast local scratch SSD or DAS SSD being shadowed back to a server would be more efficient. Thinking here is that any one user is typically working on just one large dataset, e.g. <1TB at any given time so it only really needs to be shared on a server if two or more people are working on the same dataset concurrently. If the network copy, through a bandwidth throttled shadowing system and slower HDDs, is a number of minutes behind the live copy, is that an issue?
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
Post Reply

Return to “Future Technology”