Threadripper

Discuss all Cyclone Leica Cyclone, Field 360 & Register 360 related issues here.
Post Reply
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

dhirota wrote: Mon Feb 10, 2020 2:41 am
smacl wrote: Sun Feb 09, 2020 10:45 am ..... Not sure if you'll get more than 256GB of RAM into it at this stage as AMD want to differentiate between the HEDT and server market with Epyc.
After a little searching found registered ECC RDIMM
DDR4-2666-32GB sticks for $101.00 each (8X32GB)=$ 808 new
DDR4-2933-32GB sticks for $167.13 each (8X32GB)=$1,336 new
DDR4-3200-32GB sticks for $175.00 each (8X32GB)=$1,400 new
DDR4-2933-64GB sticks for $329.56 each (4X64GB)=$1,318 new

....

Tomorrow, I need to check if ASUS tech support says that it is OK to use the 64GB sticks to 512GB total RAM size. I need to also check my source for refurbished RAM sticks for 32GB and 64GB ECC RDIMM size.
Shane

Looks like you are right about AMD differentiating between HEDT and Server total RAM. Since receiving my MB and registering the serial number with ASUS, I was able to get personal answers about using 64GB RAM sticks to achieve 512GB in the 8 RAM slots. NO, use at own risk. As we all know it is best to use the largest RAM sticks available for future expansion. I was planning to use 4X64GB to achieve the 256GB maximum as indicated by Puget Systems and BOXX for their systems. I will now use the 8X32GB, since I have $5,000 invested in the CPU+MB and will not risk it.
Dennis Hirota, PhD, PE, LPLS
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

With new technology there are many issues that need to be resolved even when they say it will work. Depending on timing, purchasing may take one or two days, but DIY still requires integration and testing to see if it will boot with the OS that you are planning to use. From my last week of problem solving, a significant suggestion for future DIYself people is to check on YouTube search for people that have tested your proposed configuration of CPU, MB, RAM, case, GPU, operating system, and application software.

Two months ago there were several forums and YouTube videos that discussed that AMD CPUs 3960X, 3970X would have correctable problems with linux and TRX40 MB. Unfortunately, I did not see that until several days ago. It seems that linux problems are not solved as quickly because of the limited number of people using linux with the new hardware. Because it is President's Day in the USA and I used another workflow to build the OS and application software, it seems to be working except for the overclocking not in place yet.

RAM is still a problem since the 256GB (8X32GB) ECC did not work, so I am trying to get what is listed on the ASUS QVL list instead of a cheaper cost. I have 128GB (8X16GB) installed from my spare RAM box. It is not enough RAM to run NavVIS SiteMaker since it is looking for more RAM, and is using SWAP space with the 128 threads.

Below are a few screen shots of 128 threads running in Ubuntu 18.04.3

Screenshot from 2020-02-17 12-33-23_SMALL.jpg

Screenshot from 2020-02-17 12-42-42_SMALL.jpg

While doing this processing, the 3990X was running at 43 to 67 degrees C, using water cooling.

More news later, when I build the Windows 10 Workstation version of this system.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

BIG LESSON LEARNED

More threads may not result in faster processing.

I processed our NavVIS M6 scan/imaging of the Ala Moana Hotel in HNL using my recently built Ryzen 3990X\ASUS Zentih II Extreme MB with 128 threads running at 2.9 GHz base platform speed without OC. I updated our Ubuntu 18.04 LTS platforms running NavVIS SiteMaker to the most recent version, 2.5.0. The Intel based processors are overclocked to 4.2 GHz. Listed below are the post-process time in hours.

Intel i7-4820K, 64GB RAM, 4-core = 12h:45m
Intel i9-9980XE, 128GB RAM, 18-core = 5h:59m

AMD 3990X, 128GB RAM, 64-core = 7h:35m

One of the lessons learned, it depends on the software. Some portions of SiteMaker are running single core, some are running 4-core, several are running 64-core, but 2.9 GHz is definitely slower than 4.2 GHz.

Second lesson learned, check out the overclocking website for information (even before purchasing a DIY system components)

https://www.overclock.net/forum/11-amd- ... hread.html

Better to learn about most things before, than after purchase. It might be better to have two Ryzen 3970X (32-core @ $1,950 vs 64-core @ $3,980), depending on the major application software.

Next project is OC my Ryzen 3990X
Dennis Hirota, PhD, PE, LPLS
[email protected]
User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 702
Joined: Tue May 01, 2012 5:19 pm
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 46 times
Been thanked: 124 times

Re: Threadripper

Post by landmeterbeuckx »

dhirota wrote: Wed Feb 19, 2020 6:35 pm BIG LESSON LEARNED

More threads may not result in faster processing.

I processed our NavVIS M6 scan/imaging of the Ala Moana Hotel in HNL using my recently built Ryzen 3990X\ASUS Zentih II Extreme MB with 128 threads running at 2.9 GHz base platform speed without OC. I updated our Ubuntu 18.04 LTS platforms running NavVIS SiteMaker to the most recent version, 2.5.0. The Intel based processors are overclocked to 4.2 GHz. Listed below are the post-process time in hours.

Intel i7-4820K, 64GB RAM, 4-core = 12h:45m
Intel i9-9980XE, 128GB RAM, 18-core = 5h:59m

AMD 3990X, 128GB RAM, 64-core = 7h:35m

One of the lessons learned, it depends on the software. Some portions of SiteMaker are running single core, some are running 4-core, several are running 64-core, but 2.9 GHz is definitely slower than 4.2 GHz.

Second lesson learned, check out the overclocking website for information (even before purchasing a DIY system components)

https://www.overclock.net/forum/11-amd- ... hread.html

Better to learn about most things before, than after purchase. It might be better to have two Ryzen 3970X (32-core @ $1,950 vs 64-core @ $3,980), depending on the major application software.

Next project is OC my Ryzen 3990X
How you find the time to test this all out Dennis is a mystery to me
LSBbvba
Surveying services - 3D Laserscanning
Tel : +32477753126
www.lsbbvba.be
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

Lieven

The lessons for others, is that I pay cash for everything. I have been doing testing since I was in graduate school at the University of Michigan and USAF Weapons Lab over 50 years ago. If it does not work, I trash it since it is only $.

It is more productive than watching TV.
Dennis Hirota, PhD, PE, LPLS
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

dhirota wrote: Wed Feb 19, 2020 6:35 pm BIG LESSON LEARNED
More threads may not result in faster processing.

I processed our NavVIS M6 scan/imaging of the Ala Moana Hotel in HNL using my recently built Ryzen 3990X\ASUS Zentih II Extreme MB with 128 threads running at 2.9 GHz base platform speed without OC. I updated our Ubuntu 18.04 LTS platforms running NavVIS SiteMaker to the most recent version, 2.5.0. The Intel based processors are overclocked to 4.2 GHz. Listed below are the post-process time in hours.

overclocked at 4.2 GHz
Intel i7-4820K, 64GB RAM, 4-core = 12h:45m
Intel i9-9980XE, 128GB RAM, 18-core = 5h:59m

original at 2.9 GHz
AMD 3990X, 128GB RAM, 64-core = 7h:35m

One of the lessons learned, it depends on the software. Some portions of SiteMaker are running single core, some are running 4-core, several are running 64-core, but 2.9 GHz is definitely slower than 4.2 GHz.

....
ANOTHER LESSON LEARNED
Overclocking new complex systems, not as easy as they say it is.

Lieven:
I am running out of time to track 7 hour activity logs in real time, so decided to use KAZAM to video the screen. Unfortunately, if the overclock crashes at 4.2, 4.1, 3.9 GHz everything disappears unless you save it on the network with Ubuntu. It finally ran at

overclocked at 3.7 GHz
AMD 3990X, 128GB RAM, 64-core = 6h:50m

Overclocking improved the computer time to a 50 minute difference. I believe I need a better cooling system than the AIO installed, since I saw some CPU temperatures above 80 deg C. Still looking for the 256GB of ECC RAM. The Ryzen 3970X may have been a better trial/purchase, but the testing is not over yet.


I included some better screen shots.

Screenshot from 2020-02-23 08-46-42-RT.jpg

Screenshot from 2020-02-23 08-46-42-LEFT.jpg

Screenshot from 2020-02-23 08-50-53-RT.jpg

Screenshot from 2020-02-23 08-50-53-LEFT.jpg

According to the panorama rendering log, the system was using only 43 threads because of the limited RAM of 128GB (~2.5GB per thread). Hopefully more RAM, more cooling, and more overclocking will reduce the post-processing time.
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

smacl wrote: Sun Feb 09, 2020 10:45 am .... Not sure if you'll get more than 256GB of RAM into it at this stage as AMD want to differentiate between the HEDT and server market with Epyc.
Shane

Some additional documentation for your comment on 3990X RAM

https://www.youtube.com/watch?v=1LaKH5etJoE
check at 14m:20s

Currently running 256GB RAM in my 3990X which will not be enough to satisfy more than 100 threads at 2.5GB per thread for our NavVIS SiteMaker software. I decided to use UDIMMs (8X32GB) instead to ECC RDIMMs in the event of reusing the 8X32GB DDR4 if 64GB sticks become available at a cheaper cost.

The major problem now for overclocking is cooling. The DEEPCOOL Castle 360EX is better than the Corsair H100i Platinum which works OK at clock speed of 3.7 GHz without fancy overclocking. I am thinking about a custom EKWB system, but it will probably wait.

The other remaining project is to build our 3990X with Windows 10 Pro or Windows 10 for Workstation running software that might have interest on the LSF and future applications.

LESSONS LEARNED
You can build an i9-10980XE for $1K less than in 2019 (~$4K today) compared to a Ryzen 3970X for ~$5K or a 3990X for ~$7K in 2020 and have the i9 complete the application in less time, depending on the software and OS.

Too many marketeers are selling too many items with too many changes happening in shorter amounts of time. It seems too many people are consuming too much "technology" because too many others are doing the same thing and not knowing why they are doing it.
Dennis Hirota, PhD, PE, LPLS
[email protected]
User avatar
smacl
Global Moderator
Global Moderator
Posts: 460
Joined: Tue Jan 25, 2011 5:12 pm
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 130 times
Been thanked: 144 times
Contact:

Re: Threadripper

Post by smacl »

dhirota wrote: Thu Feb 27, 2020 7:37 pmLESSONS LEARNED
You can build an i9-10980XE for $1K less than in 2019 (~$4K today) compared to a Ryzen 3970X for ~$5K or a 3990X for ~$7K in 2020 and have the i9 complete the application in less time, depending on the software and OS.
This is the huge caveat for all of these types tests in my opinion, they're very software specific with a lot of code in the field today still being single threaded and Intel optimized. Just a guess, but I'd imagine many of today's applications would perform as well or better on the 3950x as the i9-10980XE system in terms of end to end processing time given similar spec RAM and SSDs. The trend however is that more code is becoming more heavily multi-threaded so the balance will likely shift as new releases come out. Price/performance drops sharply with the latest and greatest components and for me the sweet spot is multiple headless desktops on last generation Threadripper and 3800 where you still get good throughput and aren't being held up too much waiting on things to finish if you've multiple jobs on the go.
Too many marketeers are selling too many items with too many changes happening in shorter amounts of time. It seems too many people are consuming too much "technology" because too many others are doing the same thing and not knowing why they are doing it.
Very true. My advise to anyone looking at any technology claims are "don't believe it, test it". That said, as a code monkey in laser scanning myself, the demand for change from the client side is never ending. Data volume growth against time is certainly a non-linear function, if not quite exponential, where multi-threading becomes increasingly important. Innovation is very important here, e.g. I'm very impressed with what the NavVis guys have done with point cloud rationalization, solving this problem close to source such that it doesn't get passed down to the client.

Anyway congrats and many thanks for the feedback on the 3990x build and I'm looking forward to see how you get on with the Windows 10 build. It would be really interesting to see how it performs on the test case Daniel Loney has set-up here as it would give a head to head comparison on a workflow familier to many Leica customers.
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

smacl wrote: Fri Feb 28, 2020 9:06 am ...
Anyway congrats and many thanks for the feedback on the 3990x build and I'm looking forward to see how you get on with the Windows 10 build. It would be really interesting to see how it performs on the test case Daniel Loney has set-up here as it would give a head to head comparison on a workflow familiar to many Leica customers.
Shane

Tomorrow the Inland PCIe Gen4 1TB and 2TB NVMe M.2 sticks will show up to see if 5Gbps will make a difference with W10 Pro. Use of the AMD Ryzen Master 2.1 Reference Guide, which runs on W10 will be a learning experience to be transferred to Ubuntu.

LESSON LEARNED
Overclocking RAM and syncing it makes a difference using AMD CPUs. Using CPU 3.7GHz and increasing the RAM speed to 3200MHz, reduced my NavVIS test sample to 6h:28m (now only 28m difference with i9).
Dennis Hirota, PhD, PE, LPLS
[email protected]
dhirota
V.I.P Member
V.I.P Member
Posts: 683
Joined: Sun Nov 01, 2009 11:18 pm
Full Name: Dennis Hirota
Company Details: Sam O. Hirota Inc.
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 20 times
Been thanked: 101 times

Re: Threadripper

Post by dhirota »

dhirota wrote: Sun Mar 01, 2020 9:42 pm
Tomorrow the Inland PCIe Gen4 1TB and 2TB NVMe M.2 sticks will show up to see if 5Gbps will make a difference with W10 Pro. Use of the AMD Ryzen Master 2.1 Reference Guide, which runs on W10 will be a learning experience to be transferred to Ubuntu.
I installed them with W10 Pro and they are very fast (5Gbps-read/4+Gbps write), but did not make any difference on the Ubuntu overclock and sample compute time. Looking for some of our W10 Pro applications with ability to use 64-cores seemed remote and a waste of time to benchmark, unless someone else knows of some example applications that I could use more than 32-cores with W10 Pro.

While checking out things, I ran into my Cray 1-S board mounted on our office wall as a reminder of times long ago.

20200304_162142-SMALL.jpg

This Cray 1-S printed circuit board is a genuine part of the second supercomputer ever built. It arrived at Kirtland AFB, New Mexico in 1980. I was stationed at the USAF Weapons Laboratory as a Capt and an Environmental Research Engineer from 1968 to 1971 during the Viet Nam War. The computer system when I was there occupied the entire ground floor of the main lab and was used for such things as simulation of landing aircraft.

The Cray 1-S for it's time was a very powerful computer with 4 million 64-bit words performing 140 million floating calculations per second. It had 200,000 integrated circuits, 3,400 printed circuit boards, and 60 miles of wiring.

This might be my last entry to this thread, because like Lieven asked " How do I continue to test?" That is unless someone asks a question?
You do not have the required permissions to view the files attached to this post.
Dennis Hirota, PhD, PE, LPLS
[email protected]
Post Reply

Return to “Leica Cyclone, Field 360 & Register 360”