Working storage drives and Large Datasets

3DVEM, Leica Cyclone Register 360, FARO Scene, LFM Suite, Autodesk ReCAP, Riegl RiScan Pro, Trimble RealWorks, Etc.
User avatar
silviaks90
I have made 40-50 posts
I have made 40-50 posts
Posts: 42
Joined: Fri Jan 16, 2015 7:03 pm
9
Full Name: Kyle Pickering
Company Details: ONSITE3D
Company Position Title: Co-Founder
Country: Calgary Canada
Linkedin Profile: Yes

Re: Working storage drives and Large Datasets

Post by silviaks90 »

Rikore wrote:Suggest trying a 4 TB external, they are not that much.
Gave this a try on a separate machine just to test using a WD Passport Ultra, in this specific example I was indexing the .e57's in recap. Probably not a viable solution for us moving forward, unless it was just this specific test but it took 19 hours to index 120GB file running USB 3.0. Pretty good machine as well, i7, 1070 card, 32GB ram, m.2 SSD for operating drive, etc. But I do agree they are cheap and a flexible option. Thanks for the suggestion, you guys clearly have some large files!

imcid953 wrote:I'm not working with Cyclone (using Scene), but our computer has 2 1-TB SSDs set to a raid 0.

Bill
Have you benchmarked this setup against running a 1TB m.2 Samsung 950/960 Pro?
dhirota wrote:I posted this on February 15, 2017

10TB, 12TB, 14TB HELIUM FILLED HDD & INFINIBAND
viewtopic.php?f=123&t=11497

From the last 3 months of use, the two Helium 10TB Seagate SAS Raid 6 systems (100TB each) we built are almost as fast as SATA SSD Raid XX systems, but significantly cheaper, especially if you need to be in the 100TB and larger storage capacity.
Dennis this is awesome and I can't wait to have these kinds of storage issues in the near future, something to start thinking about as a long term setup :ugeek:
Phil Marsh wrote: Cant you just have Cyclone and database installed on your SSD but have your working directory on an additional internal hard drive. This is where the 4x times space things comes in for unifing, etc.
Thanks for the suggestion Phil, I tried doing this exact workflow with a number of configurations as we have a few different drives to choose from. The most promising was like you said running cyclone and database from the 1TB SSD completely empty and setting the working drive to an open 4TB WD Black. However I still got the same space errors or what I believe to be space errors (I never got a firm answer from Leica on that). It was only until I moved the database and all temporary and working folders over to the open 4TB drive that it worked properly. I'm not sure if it was situational on the files, the cyclone install, the computer running it. Lots of factors, I intend to do some further testing to see if I can rectify the issue to use the decent horsepower we've put in place to handle these tasks efficiently. Ideally that would be the best route though.
Kyle Pickering
Co-Founder and Designer, ONSITE3D
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: Working storage drives and Large Datasets

Post by dhirota »

Kyle

I have not bench marked application/computations on any of our storage systems yet since I am currently interested in access speed/timing of large files. We can store 10,000 36Mp images from our Nikon D810 on a SD card. If you have 3 Nikons on a Riegl mobile, it will take a while to move it around your IT storage system.

I have a 500GB NVMe Samsung 950 as a boot/compute drive on my DELL 7710. W10 tells me that moving large files with the Xeon on the laptop will copy at a rate of 1 to 2 Gb/second, which I think is about 2X to 4X faster than the 2TB Samsung on the same laptop depending on what you are moving.
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
JNally
I have made 90-100 posts
I have made 90-100 posts
Posts: 96
Joined: Mon Feb 01, 2016 12:00 pm
8
Full Name: Joe Nally
Company Details: EPS
Company Position Title: Lead Surveyor
Country: Ireland
Linkedin Profile: Yes
Location: IRL
Has thanked: 13 times
Been thanked: 18 times

Re: Working storage drives and Large Datasets

Post by JNally »

Just thought I would add what I am currently working with

Before:
2 x 1 TB Samsung SSD's running in Raid 0 (Read Write Speed of about 1 GB/s)
Had our Cyclone Cache running on slower 3.5 Inch HDD

Bigger projects really started to slow us down. So we decided to move to something else

After:
2 x 1.2 TB Intel 750 Series NVME PCIE SSD's in Raid 0 (Read Speed of 5.2 GB/s and Write Speed of 2.4 GB/s)
Cyclone Cache is now running on our old 1TB SSD's running in Raid 0

The difference is absolute night and day. Watching Cyclone use all of that power in Task Manager is quite enjoyable.
We are doing a project as I write this with around 1600 Scans from a P40 and the New storage setup is handling it with ease.

The new intel drives were not cheap, around 550GBP each, but well worth it in my opinion.

Our Export to .E57 time has improved by at least 60%

Granted we only have 2.18TB of storage on the new drives we move each project on to the NAS drives once we have finished. We have 4 Synology NAS Drives with about 40TB of usable storage all with redundancy. We are now looking down the route of installing NIC cards into our Machines to increase our network transfer speeds, which will help for transferring large projects.
dhirota
V.I.P Member
V.I.P Member
Posts: 957
Joined: Sun Nov 01, 2009 11:18 pm
14
Full Name: Dennis Hirota
Company Details: Sam O Hirota Inc
Company Position Title: President
Country: USA
Linkedin Profile: Yes
Location: Hawaii, USA
Has thanked: 87 times
Been thanked: 379 times

Re: Working storage drives and Large Datasets

Post by dhirota »

James

I am back from my long trip to Singapore to visit the IAAPA Asia Expo.

Have you had time to test your newly installed system with the NVIDIA M60? I just viewed a NVIDIA webinar describing a CISCO bladed system with M60 running REVIT. The presentation indicated that they are saving $2,500 per week (I am assuming per person, unknown number of users) in billable hours.

Designers
8 [email protected]
512GB Storage
48GB RAM
1GB vGPU
8 users per blade

Renderers
12 [email protected]
512GB Storage
48GB RAM
4GB vGPU
2 users per blade
Dennis Hirota, PhD, PE, LPLS
www.samhirota.com
[email protected]
Post Reply

Return to “Registration Software”