10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
-
- V.I.P Member
- Posts: 958
- Joined: Sun Nov 01, 2009 11:18 pm
- 14
- Full Name: Dennis Hirota
- Company Details: Sam O Hirota Inc
- Company Position Title: President
- Country: USA
- Linkedin Profile: Yes
- Location: Hawaii, USA
- Has thanked: 87 times
- Been thanked: 379 times
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
A new update on the 14TB helium filled HDD
Some interesting data from Backblaze on their helium vs air filled HDD
Backblaze 2018 Q1 Lifetime reliability for their HDD
Some interesting data from Backblaze on their helium vs air filled HDD
Backblaze 2018 Q1 Lifetime reliability for their HDD
You do not have the required permissions to view the files attached to this post.
-
- V.I.P Member
- Posts: 958
- Joined: Sun Nov 01, 2009 11:18 pm
- 14
- Full Name: Dennis Hirota
- Company Details: Sam O Hirota Inc
- Company Position Title: President
- Country: USA
- Linkedin Profile: Yes
- Location: Hawaii, USA
- Has thanked: 87 times
- Been thanked: 379 times
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
Since I posted this recent information on May 4, 2018, there have been approximately 200+ views of this thread. Of this total number of viewers, only 17 have seen the valuable information on the 3 images that I extracted from the original sources since they were registered members of the LSF. Unfortunately the other 90% of the viewers are guests that are missing a significant amount of information that is being posted.
-
- V.I.P Member
- Posts: 958
- Joined: Sun Nov 01, 2009 11:18 pm
- 14
- Full Name: Dennis Hirota
- Company Details: Sam O Hirota Inc
- Company Position Title: President
- Country: USA
- Linkedin Profile: Yes
- Location: Hawaii, USA
- Has thanked: 87 times
- Been thanked: 379 times
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
While organizing some information on storage, some interesting comparisons between enterprise and desktop HDD from INTEL
https://www.intel.com/content/www/us/en ... ducts.html
https://www.intel.com/content/dam/suppo ... Ds_2.0.pdf
https://www.intel.com/content/www/us/en ... ducts.html
https://www.intel.com/content/dam/suppo ... Ds_2.0.pdf
You do not have the required permissions to view the files attached to this post.
- smacl
- Global Moderator
- Posts: 1409
- Joined: Tue Jan 25, 2011 5:12 pm
- 13
- Full Name: Shane MacLaughlin
- Company Details: Atlas Computers Ltd
- Company Position Title: Managing Director
- Country: Ireland
- Linkedin Profile: Yes
- Location: Ireland
- Has thanked: 627 times
- Been thanked: 657 times
- Contact:
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
Dennis, don't know if the cost would justify the benefit for your workflow, but have you considered getting a big RAM disk for storage purposes on the currently active job. By rights, most 64 bit software shouldn't need it and should use available RAM over disk, but this isn't always the case. RAM disk is about 4x - 6x speed of SSD but expensive. Not point cloud, but I use a large RAM disk on my dev environment and it removes a big bottleneck. Next spend for me will be putting together a decent thread ripper build, which should give me a serious performance boost at a reasonable price.
-
- V.I.P Member
- Posts: 958
- Joined: Sun Nov 01, 2009 11:18 pm
- 14
- Full Name: Dennis Hirota
- Company Details: Sam O Hirota Inc
- Company Position Title: President
- Country: USA
- Linkedin Profile: Yes
- Location: Hawaii, USA
- Has thanked: 87 times
- Been thanked: 379 times
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
Shane
Thank you for the comment. We are using the Seagate SAS HDD (10TB, 12TB, currently at 220TB), in raid 6 to move files to workstations using our dual 10GbE office system. The AVAGO SAS raid 6 controllers give us large storage at about the same transfer speed as SSD at a significantly cheaper cost. We also have AVAGO SATA raid5 SSD sysytems at 7TB.
The main system that we use for processing, visualization, and development is a INTEL server with dual Xeon 3.5GHz for a total of 16 threads and 256GB of RAM of the total 1.5TB of space. We also have NVMe on several systems, but may take your suggestion and try the RAM disk. The price of DDR4 has dropped in the last year or two: 64GB LRDIMM @ $700 and 32GB LRDIMM @ $370, so I might increase one of the systems and try the RAM disk.
Thank you for the comment. We are using the Seagate SAS HDD (10TB, 12TB, currently at 220TB), in raid 6 to move files to workstations using our dual 10GbE office system. The AVAGO SAS raid 6 controllers give us large storage at about the same transfer speed as SSD at a significantly cheaper cost. We also have AVAGO SATA raid5 SSD sysytems at 7TB.
The main system that we use for processing, visualization, and development is a INTEL server with dual Xeon 3.5GHz for a total of 16 threads and 256GB of RAM of the total 1.5TB of space. We also have NVMe on several systems, but may take your suggestion and try the RAM disk. The price of DDR4 has dropped in the last year or two: 64GB LRDIMM @ $700 and 32GB LRDIMM @ $370, so I might increase one of the systems and try the RAM disk.
- smacl
- Global Moderator
- Posts: 1409
- Joined: Tue Jan 25, 2011 5:12 pm
- 13
- Full Name: Shane MacLaughlin
- Company Details: Atlas Computers Ltd
- Company Position Title: Managing Director
- Country: Ireland
- Linkedin Profile: Yes
- Location: Ireland
- Has thanked: 627 times
- Been thanked: 657 times
- Contact:
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
Sounds like a real beast of a set up Dennis. If your processing workflow is anyway CPU bound, the AMD threadripper could be worth considering in the future, as it offers 32 threads at a very reasonable price. At high levels of performance you really need to profile the system over time to see where the bottlenecks lie and even then they're prone to change. e.g. our own software uses as many CPU threads as it can get its hands on without using the GPU, but we're currently porting some of the more intensive processing to GPU. We're currently doing a lot of automated extraction on Pegasus 2 mobile LIDAR, with each data set seeming to be bigger than the last, so we're always trying to squeeze all we can out of the hardware. The crypto miners have been hoovering up all the decent GPUs of late but I'd guess that will change soon enough.
-
- V.I.P Member
- Posts: 958
- Joined: Sun Nov 01, 2009 11:18 pm
- 14
- Full Name: Dennis Hirota
- Company Details: Sam O Hirota Inc
- Company Position Title: President
- Country: USA
- Linkedin Profile: Yes
- Location: Hawaii, USA
- Has thanked: 87 times
- Been thanked: 379 times
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
There is also space for 4 GPU boards on the system described above. We had NVIDIA GTX980Ti, Quadro 5000, and GTX1080Ti installed on it. Tracking most of the processing, it did not matter except to have one, so the GTX1080Ti is the only one left. Using Register360 only uses 6 threads, and if my memory is correct, Recap used approximately 25% GPU maximum.
Most of our previous testing was texturing large meshes, so we increased the RAM to the 256GB as a practical limit. It seems more, larger point clouds are being created. We used our Reigl VZ400i at pano20 (99Mp scan) to scan the outside of two Waikiki hotel towers and our two ZF5010X at high (40-50Mp) to scan the inside of two pent house floors in each tower. Total storage was 280 GB for raw and processed data including photos for all scans (120 scans in 8 hours). We now have a database of 20 scans in central Waikiki in full color since the Riegl VZ400i has a range of approximately 250 meters, with each scan position taking 180 seconds and the 7 36Mpixel photos 60 seconds.
Most of our previous testing was texturing large meshes, so we increased the RAM to the 256GB as a practical limit. It seems more, larger point clouds are being created. We used our Reigl VZ400i at pano20 (99Mp scan) to scan the outside of two Waikiki hotel towers and our two ZF5010X at high (40-50Mp) to scan the inside of two pent house floors in each tower. Total storage was 280 GB for raw and processed data including photos for all scans (120 scans in 8 hours). We now have a database of 20 scans in central Waikiki in full color since the Riegl VZ400i has a range of approximately 250 meters, with each scan position taking 180 seconds and the 7 36Mpixel photos 60 seconds.
Last edited by dhirota on Wed May 08, 2019 2:59 am, edited 1 time in total.
-
- V.I.P Member
- Posts: 958
- Joined: Sun Nov 01, 2009 11:18 pm
- 14
- Full Name: Dennis Hirota
- Company Details: Sam O Hirota Inc
- Company Position Title: President
- Country: USA
- Linkedin Profile: Yes
- Location: Hawaii, USA
- Has thanked: 87 times
- Been thanked: 379 times
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
This was from several years ago
I just purchased 30 each of 12TB Seagate SAS to replace our dependable 4TB and 6TB Seagate SAS drives like it shows in the table. The 14TB are still difficult to get and at significantly higher cost.
What is interesting from Jason's discussion in this thread is the use of 10GbE ports. We have decided to stay with our 10GbE ports, and will try to dual aggregate them to all switches. I do not think it will work with W10 or W7 for the workstations. A quick test between W10 and Ubuntu workstations using iperf3-6 showed transferring 10.9GB of information in 10 seconds at a rate of 9.4Gbs over a single port.
We did another experiment opening up and viewing a 17.6GB Civil3D file folder with about 3K items (2,651 files, 322 folders):
Operational Server, 8X10TB Raid6 Seagate SAS, dual 10GbE ports, 32GB RAM workstation with 10GbE port = 55 seconds
Experimental Server, 8X1TB Raid5 SanDisk, dual 10GbE ports, 32GB RAM workstation with 10GbE port = 47 seconds
We have looked at an all SSD server. Has anyone purchased or built their own? A Seagate 12TB HDD is $350, whereas a 2TB 860EVO SSD is $300.
I always check on the Backblaze evaluation of their system storage, which is interesting for the detail that they include. They are indicating testing of some 20TB HDD.dhirota wrote: ↑Thu Feb 16, 2017 1:38 am Everyone has been speaking about larger SSDs in the last year. I do not know if this subject is FUTURE TECHNOLOGY or not, but many folks think rotating HDD will be gone soon. We have looked at creating our own internal cloud for many reasons and last year decided to extend what we had into faster, larger internal storage. We decided on a pair of INTEL servers, following the prototype workstation built last year using an INTEL server motherboard (dual Xeon, .25TB/1.5TB RAM, dual 10GbE, 64TB SAS RAID6 8TB Seagate HDD, 2-GTX1080 GPU).
.....
The Seagate 10TB ST10000NM0206 HDD are SAS vs. SATA which gives us 12Gbs on the Broadcom/AVAGO 9361-16i controllers that we are using. This gives us 100TB in Raid6 with 12 HDD on each server with dual 10GbE on each server. In the tests today, Robocopy is copying a 15TB block of projects at a speed of approximately 275MB/second over our internal 10GbE ethernet network at about 2-3Gbs total from 2 ports to another server with two ports.
We decided not to fill both servers with disks and wait until prices on the 10TB reduced due to the introduction of the 12 or 14TB HDD.
I just purchased 30 each of 12TB Seagate SAS to replace our dependable 4TB and 6TB Seagate SAS drives like it shows in the table. The 14TB are still difficult to get and at significantly higher cost.
What is interesting from Jason's discussion in this thread is the use of 10GbE ports. We have decided to stay with our 10GbE ports, and will try to dual aggregate them to all switches. I do not think it will work with W10 or W7 for the workstations. A quick test between W10 and Ubuntu workstations using iperf3-6 showed transferring 10.9GB of information in 10 seconds at a rate of 9.4Gbs over a single port.
We did another experiment opening up and viewing a 17.6GB Civil3D file folder with about 3K items (2,651 files, 322 folders):
Operational Server, 8X10TB Raid6 Seagate SAS, dual 10GbE ports, 32GB RAM workstation with 10GbE port = 55 seconds
Experimental Server, 8X1TB Raid5 SanDisk, dual 10GbE ports, 32GB RAM workstation with 10GbE port = 47 seconds
We have looked at an all SSD server. Has anyone purchased or built their own? A Seagate 12TB HDD is $350, whereas a 2TB 860EVO SSD is $300.
You do not have the required permissions to view the files attached to this post.
- Jason Warren
- Administrator
- Posts: 4224
- Joined: Thu Aug 16, 2007 9:21 am
- 16
- Full Name: Jason Warren
- Company Details: Laser Scanning Forum Ltd
- Company Position Title: Co-Founder
- Country: UK
- Skype Name: jason_warren
- Linkedin Profile: No
- Location: Retford, UK
- Has thanked: 443 times
- Been thanked: 246 times
- Contact:
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
Thank you for the update Dennis. I think this topic has become more relevant for timely back office data logistics, especially with the take up of the RTC360
Jason Warren
Co_Founder
Dedicated to 3D Laser Scanning
LaserScanningForum
Co_Founder
Dedicated to 3D Laser Scanning
LaserScanningForum
- smacl
- Global Moderator
- Posts: 1409
- Joined: Tue Jan 25, 2011 5:12 pm
- 13
- Full Name: Shane MacLaughlin
- Company Details: Atlas Computers Ltd
- Company Position Title: Managing Director
- Country: Ireland
- Linkedin Profile: Yes
- Location: Ireland
- Has thanked: 627 times
- Been thanked: 657 times
- Contact:
Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
Just thinking out loud here, so bear with me. Will replacing slower drives with faster drives on the server have the net effect of shifting the bottleneck to the network as multiple users read and write very large files on an ongoing basis? Depends on the workflow, but for me something like a fast local scratch SSD or DAS SSD being shadowed back to a server would be more efficient. Thinking here is that any one user is typically working on just one large dataset, e.g. <1TB at any given time so it only really needs to be shared on a server if two or more people are working on the same dataset concurrently. If the network copy, through a bandwidth throttled shadowing system and slower HDDs, is a number of minutes behind the live copy, is that an issue?