10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post any marterial related the future technology of laser scanning.
Post Reply
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

Everyone has been speaking about larger SSDs in the last year. I do not know if this subject is FUTURE TECHNOLOGY or not, but many folks think rotating HDD will be gone soon. We have looked at creating our own internal cloud for many reasons and last year decided to extend what we had into faster, larger internal storage. We decided on a pair of INTEL servers, following the prototype workstation built last year using an INTEL server motherboard (dual Xeon, .25TB/1.5TB RAM, dual 10GbE, 64TB SAS RAID6 8TB Seagate HDD, 2-GTX1080 GPU).

Western Digital has announced 10TB helium HDD two years ago, and Seagate announced the helium filled 10TB HDD last year. We placed our order in October, 2016 with a major IT distributor for the Seagate 10TB ST10000NM0206. We finally received our order of a box of 20 HDD after 5 delays, yesterday. In December, 2016 while we were waiting for our 10TB to arrive, WDigital and Seagate announced 12TB HDD for this year and 14TB for 2018.

The Seagate 10TB ST10000NM0206 HDD are SAS vs. SATA which gives us 12Gbs on the Broadcom/AVAGO 9361-16i controllers that we are using. This gives us 100TB in Raid6 with 12 HDD on each server with dual 10GbE on each server. In the tests today, Robocopy is copying a 15TB block of projects at a speed of approximately 275MB/second over our internal 10GbE ethernet network at about 2-3Gbs total from 2 ports to another server with two ports.

We decided not to fill both servers with disks and wait until prices on the 10TB reduced due to the introduction of the 12 or 14TB HDD.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
Jason Warren
Administrator
Administrator
Posts: 4217
Joined: Thu Aug 16, 2007 9:21 am
16
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: No
Location: Retford, UK
Has thanked: 439 times
Been thanked: 241 times
Contact:

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren »

Hi Dennis,
Impressive, but would like to know why you opted for 10Gbe and not Infiniband?
Infiniband.PNG
You do not have the required permissions to view the files attached to this post.
Jason Warren
Co_Founder

Dedicated to 3D Laser Scanning
LaserScanningForum
KDersch
I have made 60-70 posts
I have made 60-70 posts
Posts: 60
Joined: Wed Jun 01, 2016 11:54 am
7
Full Name: Kyle Dersch
Company Details: Matix Technologies
Company Position Title: Engineer
Country: USA
Skype Name: [email protected]
Linkedin Profile: Yes

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by KDersch »

Subscribed...
SAttaya
V.I.P Member
V.I.P Member
Posts: 401
Joined: Mon Mar 23, 2009 4:04 pm
14
Full Name: SAttaya
Company Details: Sev1Tech
Company Position Title: Sr Software Analyst-RemoteSensingEngr
Country: USA
Linkedin Profile: No
Has thanked: 127 times
Been thanked: 17 times

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by SAttaya »

I thought I'd seen something about 60TB drives -
http://www.seagate.com/about-seagate/ne ... oducts-pr/
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

Jason

The very simple short story is that we were looking for relatively, fast, cheaper storage than the cloud, which we are not allowed to use by some clients. The resulting survey of options (we might have missed many), was to use the INTEL servers because of reuse in the future (able to change 12X3.5 inch HDD to 24X2.5 inch SSD, 1.5TB RAM, 4 GTX GPU, 2each 1100W Power Supply) and direct support/warranty to 2021. The net result for 10GbE is that you get two 10GbE ports on the INTEL boxes for FREE. So we have a few INTEL RJ45 10GbE ports and a while ago decided not to spend on money on reorganizing our internal network (which has mostly CAT6, CAT6A but some CAT5e for printers).

An interesting article
https://www.nextplatform.com/2015/04/01 ... o-kill-it/

Maybe 2017 is the year for us to evaluate Infiniband and our internal network.
2-16-2017 2-20-18 PM.jpg
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
Jason Warren
Administrator
Administrator
Posts: 4217
Joined: Thu Aug 16, 2007 9:21 am
16
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: No
Location: Retford, UK
Has thanked: 439 times
Been thanked: 241 times
Contact:

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren »

dhirota wrote: An interesting article
https://www.nextplatform.com/2015/04/01 ... o-kill-it/

Maybe 2017 is the year for us to evaluate Infiniband and our internal network.
Dennis,
I know you like to keep at the cutting edge ;)
A two card Mellanox solution for a dedicated connection between a server and `Registration` workstation would be in the order of £1135.19. Its not cheap, but its not 10GbE.
- MCX353A-FCCT, Single VPI FDR/40/56GbE InfiniBand HCA - £538 ea exVAT
- MC2207130-002, 2m 56Gb/s Passive Copper Cables - £59.16 ea exVAT

If you want a cheaper Infiniband solution, ie not 56GbE but say 20GbE or 40GbE, then check out all the stuff thats coming out of data-centres at the moment and landing on ebay. This kit is cheap, but as always buyer beware!

Lastly, if you get a spare 5 minutes, take a look at what ICEOTOPE are doing. Their machines have a unique cooling system. The entire motherboard etc is submerged in liquid plastic for unique cooling and silent running... ;)
http://www.iceotope.com/
Jason Warren
Co_Founder

Dedicated to 3D Laser Scanning
LaserScanningForum
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

Jason

Thank you for the information on infiniband. Our current "registration/processing" workstation is like a server with 60TB of SAS HDD on it, so it does not have to move to another server at 56Gbs speed for storage. The equivalent of 2X10GbE is fast enough at a cost of FREE. The US$1500 would be interesting to move processed data between servers which is currently 2X10GbE, so the 56Gbs from infiniband would be a significant improvement.

We have not researched to see if the 56Gbs speed would be limited by the average 275MB/second write speed of our server HDD, since when we robocopy files, the task manager is indicating that the HDD is writing at 100% using the 2X10GbE network.

The Iceotope system sounds interesting, but in the middle of 5,000 miles of ocean, and 2,500 miles from any significant land mass, we need to fix it and have it working without sending it back 12,000 miles by FEDEX/DHL.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
Jason Warren
Administrator
Administrator
Posts: 4217
Joined: Thu Aug 16, 2007 9:21 am
16
Full Name: Jason Warren
Company Details: Laser Scanning Forum Ltd
Company Position Title: Co-Founder
Country: UK
Skype Name: jason_warren
Linkedin Profile: No
Location: Retford, UK
Has thanked: 439 times
Been thanked: 241 times
Contact:

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by Jason Warren »

dhirota wrote: We have not researched to see if the 56Gbs speed would be limited by the average 275MB/second write speed of our server HDD, since when we robocopy files, the task manager is indicating that the HDD is writing at 100% using the 2X10GbE network.
True, if your I/O is maxed out at 275MB/s then there`s little point in utilising a 56GbE connection.
dhirota wrote: The Iceotope system sounds interesting, but in the middle of 5,000 miles of ocean, and 2,500 miles from any significant land mass, we need to fix it and have it working without sending it back 12,000 miles by FEDEX/DHL.
Thats the point, the main reason for server component failure is overheating. With these bad boys they shouldn't need returning.... ;)
Jason Warren
Co_Founder

Dedicated to 3D Laser Scanning
LaserScanningForum
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

Jason

I do not know if we have completed the discussion of SAS 10TB to14TB Helium HDD and Infiniband yet, but when I started this thread I mentioned about the announcement of large SAS SSD drives like the Seagate 60TB 3.5 inch SSD that was mentioned earlier by SAttaya or the 100TB SSD by Toshiba.

We decided to delay the implementation of infiniband until later when the pricing of SAS SSD systems arrives at a price point where SSDs like the Samsung PM1633a 15.36TB in volume drops significantly below US$10K. So we decided to fill up our INTEL servers to 100 TB each with the Seagate 10TB SAS HDD rather than wait for the Seagate 10TB HDD to drop. From our source we got them at US$433 each, which is only a few $ above the 8TB. I think we did OK since the box of 20 + 6 extra HDD for 100TB each in raid 6 and spare HDDs was less than US$6,000 each.

We have the space on the INTEL servers to add 4X 2.5 inch SSDs to our BROADCOM/AVAGO raid controllers as cache or raid SSD storage.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: 10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND

Post by dhirota »

It has been about a year since my last post on this thread.

I have moved the Seagate 10TB helium filled HDD into two servers, replacing the 4TB HDD there to increase capacity. The empty Intel server is now filled with 12 X 12TB helium filled HDD, giving us 120TB in raid6.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
Post Reply

Return to “Future Technology”