AMD THREADRIPPER PRO 3995WX

To chat about anything else.
Post Reply
User avatar
Carbix
V.I.P Member
V.I.P Member
Posts: 236
Joined: Sat Mar 16, 2019 1:36 am
5
Full Name: Daniel Loney
Company Details: Excelsior Measuring
Company Position Title: Owner
Country: Canada
Linkedin Profile: Yes
Location: Canada
Has thanked: 33 times
Been thanked: 50 times
Contact:

Re: AMD THREADRIPPER PRO 3995WX

Post by Carbix »

RAM disk could work but tends to come with its own number of issues.

I would run a raid 5 with 4 NVMe Gen 4's and if you feel like living on the whiled side run it in raid 0.

Intel has their optane drives. I would see if they have a new Gen 4 full pci version. Optane is all about dropping IO

they ain't cheap
Daniel Loney - Owner
Excelsior Measuring inc.
Vancouver - Okanagan - Calgary
www.ExcelsiorLevel.com
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: AMD THREADRIPPER PRO 3995WX

Post by dhirota »

smacl wrote: Mon May 10, 2021 7:57 am Not sure if I've asked this before Dennis, but have you considered RAM disks. If you have an I/O bound process and available memory, these are likely to be an order of magnitude faster than SSD. The trick is figuring out how to divide the physical RAM out to leave enough space for the program and storage, may not be workable on larger jobs where you don't have the available excess.
Shane

We has discussed this over the years, and one of the references is listed below:

viewtopic.php?p=69651#p69651

Based on your 4X-6X SSD @ 500MB/second IO, I might be better off with RAID0 as the processing storage?

In feature evaluation of workflow using a processing workstation, I use the many years of observation from other historic systems that I have observed. One in particular, was the PLATO system from the University of Illinois. Most people on this LSF are probably not familiar with it since the majority were not on this planet at its inception in the 1960s. If my memory is still good, it used a multi-tiered system of storage.

https://en.wikipedia.org/wiki/PLATO_(computer_system)

PLATO @50 PLATO Computer Learning System 50th Anniversary
https://www.youtube.com/watch?v=THoxsBw-UmM&t=31s

To concentrate resources into the fastest area, RAM, but also the most expensive. The RAM DISK alternative has been considered as the cost of RAM decreases and the motherboards allow greater number of DIMM slots. In this example with the TR PRO 3995WX, it was important to move large data files from the AVAGO RAID6 PC88E(72TB) to the next tier, a RAID0 PC88D(4TB) or Samsung 980Pro (2TB) for processing storage because of the read/write speed. Shown below is the CrystalDiskMark (CDM) for PC88E and PC88D for the 64threads. As mentioned above, I was able to transfer an E57 20GB file from PC88E to PC88D in 25.31 seconds.

PC88E-64T.jpg
PC88D-64T.jpg

Shane: At this point, I could "sacrifice" 150GB from the 512GB of total RAM available, but I do not have the time to check out potential problems in W10 and Ubuntu to see if it will work with all our raw data set sizes and application programs. I am still evaluating many visualization applications that are running on W10 and Ubuntu, but most are running on W10, so the priorities may change and you will be the first to know.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: AMD THREADRIPPER PRO 3995WX

Post by dhirota »

dhirota wrote: Tue May 11, 2021 8:08 pm
smacl wrote: Mon May 10, 2021 7:57 am Not sure if I've asked this before Dennis, but have you considered RAM disks. If you have an I/O bound process and available memory, these are likely to be an order of magnitude faster than SSD. The trick is figuring out how to divide the physical RAM out to leave enough space for the program and storage, may not be workable on larger jobs where you don't have the available excess.
Shane

We has discussed this over the years, and one of the references is listed below:

viewtopic.php?p=69651#p69651

Based on your 4X-6X SSD @ 500MB/second IO, I might be better off with RAID0 as the processing storage?
........

Shane: At this point, I could "sacrifice" 150GB from the 512GB of total RAM available, but I do not have the time to check out potential problems in W10 and Ubuntu to see if it will work with all our raw data set sizes and application programs. I am still evaluating many visualization applications that are running on W10 and Ubuntu, but most are running on W10, so the priorities may change and you will be the first to know.
Shane:

Since the term RAMDISK was not a familiar in our office, I decided to research the best as rated by others. Depending on requirements such as cost, size, and installation there were about 20+ on the internet. Many were dated from 5+ years ago, when SSD and NVMe storage were difficult to obtain, HDDs were slower and smaller in size. Here is a YouTube example from February 5, 2014

https://www.youtube.com/watch?v=Sv8Ypj3iUWI
HDD-SSD-RAMDISK140205.jpg
There was only one system that met my criteria of free, expandable to 1TB of RAM, and works (I tried another that worked at 4GB but it crashed at 64GB). Below are the CrystalDiskMark benchmarks running at 64threads showing from left to right:

TOTAL.jpg

1. PC88C- Samsung 980Pro-(1TB) attached to ASUS Hyper M.2X16 PCIe Gen 4
2. PC88D- 2X Samsung 980Pro-(2TB) total 4TB attached to ASUS Hyper M.2X16 PCIe Gen 4, using ASUS MB BIOS RAID0
3. PC88E- AVAGO RAID6- (8X12TB Seagate) total 72TB
4. PC88Z- RAMDISK 64GB using 4X2933MHz 128GB ECC DDR4 RAM total 512GB in 4 of 8 channels of AMD 3995WX memory.

As I said earlier, I do not have the time to check on the potential problems with applications and the RAMDISK. The other problem is the relative smaller size of the RAMDISK device created, which may limit the software application and the data set size. To acquire the I/O speeds of 2X Samsung 980Pro for US$800 in RAID0 and 4TB of storage versus the estimated US$20,000 cost of 2TB of ECC RAM at an unknown application speed.

That is about the end.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: AMD THREADRIPPER PRO 3995WX

Post by dhirota »

I started this thread almost 3 months ago as I started to build another AMD, this time a AMD ThreadRipper Pro. It has changed several times since the start and will probably change again many times, especially since I have still not received the NVIDIA RTX A6000 ordered over a month ago. The ability to run applications in both W10 and Ubuntu with 7 PCIe Gen4 X16 slots almost without worrying what will fit where is worth the extra cost of building such a research/production system. Over the many years of doing this it has enabled us not to worry about catching up with our competitors and providing clients with as much as we can afford.

I also track the number of visits to the threads by both LSF Registered Members (RM) and Guests. I am still not sure how all this works on the LSF, but the RM's apparently see all the tables and images that I produce with the significant information and Guest just see the text. Since the beginning of this thread there have been 7,283 views and at least 1,690 views by RM according to LSF. This means that 5,000+ Guests did not get the full value of the information that I posted. I am not sure why someone would spend the time to visit a forum and not access everything the forum had to offer.

I am also interested in the 1,000+ views by RM, since only a few comments or questions resulted from the 20+ RM responses. I have been waiting for someone to ask a question about the last posting on using RAMDISK. Most of the previous examples on the internet in the last 5+ years were 4-20GB for RAMDISK, which seemed too small for many applications. For my posted example, I decided to use 64GB, since I decided that some people could have 256GB of RAM available. There have been no responses or questions in the last week, so I have determined that either no one has any further interest or the presentation is irrelevant. I had a question in my own mind to document a larger size RAMDISK of 256GB since I had the resources, although I will probably not use RAMDISK in the future.

TOTAL.jpg

And finally, the system could have been built at a significantly lower cost, but whether this workstation would be limited by many of the larger applications we are beta testing in W10 is not worth the total cost of multiple research systems at this time. It seems from the threads on the LSF, most people are not willing to post research/testing information.

The shortage of electronic parts will continue into the future, so for those without parts to build, they will have to buy at a higher cost, or they will have to figure out a different workflow for many applications that they do not know about today.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
TommyMaddox
V.I.P Member
V.I.P Member
Posts: 514
Joined: Mon Apr 25, 2016 7:15 pm
7
Full Name: Tommy R Maddox III
Company Details: ONSITE3D
Company Position Title: Director of Technology
Country: USA
Linkedin Profile: Yes
Location: Calgary, Alberta
Has thanked: 67 times
Been thanked: 130 times

Re: AMD THREADRIPPER PRO 3995WX

Post by TommyMaddox »

Dennis, I appreciate all of the testing and documentation you did for the community, beyond what words can convey.

I typed up a full 3 paragraph reply, but because LSF signs me out every few minutes, it was lost.
User avatar
Carbix
V.I.P Member
V.I.P Member
Posts: 236
Joined: Sat Mar 16, 2019 1:36 am
5
Full Name: Daniel Loney
Company Details: Excelsior Measuring
Company Position Title: Owner
Country: Canada
Linkedin Profile: Yes
Location: Canada
Has thanked: 33 times
Been thanked: 50 times
Contact:

Re: AMD THREADRIPPER PRO 3995WX

Post by Carbix »

I second the MAD love for the work you do.
Daniel Loney - Owner
Excelsior Measuring inc.
Vancouver - Okanagan - Calgary
www.ExcelsiorLevel.com
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: AMD THREADRIPPER PRO 3995WX

Post by dhirota »

Tommy and Daniel

The real trick to posting responses on any forum is to (1) compose it using a WORD PROCESSOR to create it; (2) login to the forum; (3) cut and paste the content to the forum; (4) edit on the forum; and finally (5) submit it.

Tommy

I thanked you in advance, but it would be interesting to see your comments since I appreciate them very much.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: AMD THREADRIPPER PRO 3995WX

Post by smacl »

dhirota wrote: Sun May 23, 2021 10:48 pmI am also interested in the 1,000+ views by RM, since only a few comments or questions resulted from the 20+ RM responses. I have been waiting for someone to ask a question about the last posting on using RAMDISK. Most of the previous examples on the internet in the last 5+ years were 4-20GB for RAMDISK, which seemed too small for many applications. For my posted example, I decided to use 64GB, since I decided that some people could have 256GB of RAM available. There have been no responses or questions in the last week, so I have determined that either no one has any further interest or the presentation is irrelevant. I had a question in my own mind to document a larger size RAMDISK of 256GB since I had the resources, although I will probably not use RAMDISK in the future.
Hi Dennis,

As others have said, many thanks for your contribution here, it is greatly appreciated. Admittedly it has been some years since I've used RAM disks regularly and it looks very much like the software out there for them hasn't seen updates to take advantages of new architectures in some time. Out of interest, was the ramdisk solution that you tried out open source?

While I spend a whole lot of time on performance optimization, it is nearly all from the development side, focussed on algorithmics rather than hardware. The posts yourself and Daniel have made are very useful as they challenge (and often disprove) a number of my assumptions as to what the best hardware is to run the type of software that I'm writing. Many of my users are looking for a price / performance sweet spot in the context of being able to work with increasingly large point clouds in a cost effective manner. e.g. what is the minimum spec to work efficiently with a 1 billion, 2 billion, 4 billion or 8 billion points? Much of this comes down to analysis of specific workflows and identifying and removing bottlenecks only to have them move somewhere else in the workflow. If the software is hitting the disk when there is ample RAM available to the detriment of performance, it is really calling out for an optimization. FWIW, for most of the typical SCC workflows, something like an i9 10900k and 64gb of RAM is a decent budget option and I've been recommending the Thinkstation 340 as providing a lot of bang for your buck. Hard to believe the new i9 11900k is actually slower than it's predecessor on many common tasks.
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: AMD THREADRIPPER PRO 3995WX

Post by dhirota »

smacl wrote: Mon May 24, 2021 10:26 am
Hi Dennis,

As others have said, many thanks for your contribution here, it is greatly appreciated. Admittedly it has been some years since I've used RAM disks regularly and it looks very much like the software out there for them hasn't seen updates to take advantages of new architectures in some time. Out of interest, was the ramdisk solution that you tried out open source?
The RAMDISK is open-source as were all of the approximately 20+ applications that I reviewed. I do not know if there are other RAMDISK applications that might be faster, but this version claimed unlimited size (the 256GB example could have been larger) and the price was right. What I did not check was transferring my 20GB E57 file between the RAID0 NVMe and the RAMDISK. It will not matter since I am not planning to use the RAMDISK unless there are applications that will significantly improve our workflows. Having the ability to test and modify workflows without creating another research/production workstation is what this AMD3995WX and the 7 slot PCIe Gen4 ASUS MB are all about.

IMDISK.jpg
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 958
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: AMD THREADRIPPER PRO 3995WX

Post by dhirota »

Since my last update, I have been installing more software on my TR PRO running W10 Pro installed on a 1 TB Samsung 980 Pro NVMe stick. Although it has a 5 year warranty, there is also an endurance limit that comes with the warranty. The majority of most people will probably be under the endurance limit in their use of the SSD/NVMe stick. Shown below are some of the parameters to consider.

SAMSUNG-980-PRO.jpg

I used the Samsung Data Migration software to copy all the information on the 1TB 980 PRO to the 2TB stick. If you have ever copied a boot disk with 200+GB on it, the time with a SSD might take 30 to 60 minutes. In this instance, it took a total of 6 minutes as seen on the screenshot below.

DATA-MIGRATION.jpg

One of the significant reasons for the increased boot disk size besides speed is the accumulation of programs and data as well as endurance as shown in the table above. With the ability to track the health of our boot SSD/NVMe disks and raid individual HDDs it is important to check the remaining 5 year warranty to receive a better idea of system stability.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
Post Reply

Return to “General Chat”