fast storage solution

CAD Workstations, Network Attached Storage [NAS], Servers, Monitors, SSD`s, Imaging Systems, Field Kit: Targets & Tripods etc.
User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 1615
Joined: Tue May 01, 2012 5:19 pm
11
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 183 times
Been thanked: 548 times

Re: fast storage solution

Post by landmeterbeuckx »

Revamping this thread.

The old laptop is getting old now, i'm getting a new machine (desktop with enough power) and a 10GBe and 1GBe network card.

Was thinking about buying a 8 port 10GBe switch, cat 7 cable and 2 new nas devices (1 for intern projects) and 1 as a ftp-download station for clients so i wouldn't have to upload these massive rar files.

Can you upgrade a laptop to a 10GBe connection? I want to keep the laptop for creating recaps in the background from the Nas without having to transfer the bloody files every time to a local drive.

Any suggestions?
LSBbvba
Surveying services - 3D Laserscanning
Tel : +32477753126
www.lsbbvba.be
[email protected]
User avatar
smacl
Global Moderator
Global Moderator
Posts: 1409
Joined: Tue Jan 25, 2011 5:12 pm
13
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 627 times
Been thanked: 657 times
Contact:

Re: fast storage solution

Post by smacl »

Don't think you can easily upgrade the network adapter on most laptops. While it is a bit of a hack, you could use an external nvme drive connected to your main workstation or NAS via USB-C most of the time and plug it into the laptop as needed. Dirt cheap and gives you the 10gbps to your shared NVME drive but involves plugging and unplugging rather than proper sharing.
Shane MacLaughlin
Atlas Computers Ltd
www.atlascomputers.ie

SCC Point Cloud module
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1236
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 219 times
Contact:

Re: fast storage solution

Post by jedfrechette »

I'd the laptop is new enough to have USB-C you could also look for a USB-C dock with 10gbe.
Jed
User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 1615
Joined: Tue May 01, 2012 5:19 pm
11
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 183 times
Been thanked: 548 times

Re: fast storage solution

Post by landmeterbeuckx »

jedfrechette wrote: Sun Nov 15, 2020 4:46 pm I'd the laptop is new enough to have USB-C you could also look for a USB-C dock with 10gbe.
Jed,
The laptop is 5 years old. No usb-c. I do have an e-sata connection port on the side. Maybe i could do something with that?
LSBbvba
Surveying services - 3D Laserscanning
Tel : +32477753126
www.lsbbvba.be
[email protected]
jedfrechette
V.I.P Member
V.I.P Member
Posts: 1236
Joined: Mon Jan 04, 2010 7:51 pm
14
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 62 times
Been thanked: 219 times
Contact:

Re: fast storage solution

Post by jedfrechette »

You could get a hard drive doc with esata (or USB 3?) and do something like Shane suggested. I think there might be 5gbe USB 3 adapters out there too?

I'd probably just live with 1gbe. I don't know how Recap fares in this regard but as long their file IO code is written in a sane way performance over 1gbe will be slower but not unmanageable. Not a big deal if you're just batch exporting things and aren't sitting there watching the progress bars.
Jed
User avatar
Carbix
V.I.P Member
V.I.P Member
Posts: 236
Joined: Sat Mar 16, 2019 1:36 am
5
Full Name: Daniel Loney
Company Details: Excelsior Measuring
Company Position Title: Owner
Country: Canada
Linkedin Profile: Yes
Location: Canada
Has thanked: 33 times
Been thanked: 50 times
Contact:

Re: fast storage solution

Post by Carbix »

for work stations you can use Asus boards and get one of their 4x4x4x4 raid cards that let you run 4 NVMe 4.0 sticks in them. This makes for a decent raid 5 local work drive. Do not use as a boot drive. Its also faster to use this as a cache card at the same time. I know it sounds dumb but its SOOOO fast its worth doing. This is in regards to IO response times. Running large files from your NAS is a bad idea. No matter if its 1g or 10g. This is because of the IO response time. We work with files that are like sand. It does not block up well.

For your major NAS data. Run it in 2 steps. All current project that are active stay local. Everything that is done old and not being pulled up for updates goes to a cloud. This does not need to be a super fast upload or download and should be mitigated completely by the NAS local download and upload manger.

This solution has helped me leap frog over the 100TB to 1PT local solution and just stick with a very fast local 50TB solution.

Also, there is a USB-C 10G adapter out there. But you're still going to have IO response time and queuing issues if your trying to run the files from the server. Same reason why you shouldn't process scans from a USB.
Daniel Loney - Owner
Excelsior Measuring inc.
Vancouver - Okanagan - Calgary
www.ExcelsiorLevel.com
User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 1615
Joined: Tue May 01, 2012 5:19 pm
11
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 183 times
Been thanked: 548 times

Re: fast storage solution

Post by landmeterbeuckx »

Carbix wrote: Sun Nov 15, 2020 11:00 pm for work stations you can use Asus boards and get one of their 4x4x4x4 raid cards that let you run 4 NVMe 4.0 sticks in them. This makes for a decent raid 5 local work drive. Do not use as a boot drive. Its also faster to use this as a cache card at the same time. I know it sounds dumb but its SOOOO fast its worth doing. This is in regards to IO response times. Running large files from your NAS is a bad idea. No matter if its 1g or 10g. This is because of the IO response time. We work with files that are like sand. It does not block up well.

For your major NAS data. Run it in 2 steps. All current project that are active stay local. Everything that is done old and not being pulled up for updates goes to a cloud. This does not need to be a super fast upload or download and should be mitigated completely by the NAS local download and upload manger.

This solution has helped me leap frog over the 100TB to 1PT local solution and just stick with a very fast local 50TB solution.

Also, there is a USB-C 10G adapter out there. But you're still going to have IO response time and queuing issues if your trying to run the files from the server. Same reason why you shouldn't process scans from a USB.
I don't want to run directly of the nas for registering etc. Enough space to store running projects on the desktop. I just want to export to laz, copy to the nas and import in recap for exemple. Then zip this recap directly to the ftp server without the need to copy the files all over the place.

I spend way to much time post-scanning.... with these issues.
LSBbvba
Surveying services - 3D Laserscanning
Tel : +32477753126
www.lsbbvba.be
[email protected]
ericguizzetti
V.I.P Member
V.I.P Member
Posts: 205
Joined: Tue Mar 06, 2018 6:56 pm
6
Full Name: Eric Guizzetti
Company Details: Construction and Engineering
Company Position Title: RealityCapture
Country: USA
Linkedin Profile: Yes
Has thanked: 14 times
Been thanked: 11 times
Contact:

Re: fast storage solution

Post by ericguizzetti »

landmeterbeuckx wrote: Wed Apr 25, 2018 7:05 am Hello,
The amount of data on my system is going crazy. I'm searching for a system to work with for all my scanning needs.

Currently i'm working on the system drives C and D of my main pc but it gets full rather fast and i'm constantly moving files along.

Would this https://www.synology.com/nl-nl/products/FS1018#features system be something to work on with current projects and move to backup later on?
Have you seen this yet? https://apple.news/A6J6o4t1RQFu3zQ-KrBubSA
User avatar
Carbix
V.I.P Member
V.I.P Member
Posts: 236
Joined: Sat Mar 16, 2019 1:36 am
5
Full Name: Daniel Loney
Company Details: Excelsior Measuring
Company Position Title: Owner
Country: Canada
Linkedin Profile: Yes
Location: Canada
Has thanked: 33 times
Been thanked: 50 times
Contact:

Re: fast storage solution

Post by Carbix »

landmeterbeuckx wrote: Mon Nov 16, 2020 6:59 am
Carbix wrote: Sun Nov 15, 2020 11:00 pm for work stations you can use Asus boards and get one of their 4x4x4x4 raid cards that let you run 4 NVMe 4.0 sticks in them. This makes for a decent raid 5 local work drive. Do not use as a boot drive. Its also faster to use this as a cache card at the same time. I know it sounds dumb but its SOOOO fast its worth doing. This is in regards to IO response times. Running large files from your NAS is a bad idea. No matter if its 1g or 10g. This is because of the IO response time. We work with files that are like sand. It does not block up well.

For your major NAS data. Run it in 2 steps. All current project that are active stay local. Everything that is done old and not being pulled up for updates goes to a cloud. This does not need to be a super fast upload or download and should be mitigated completely by the NAS local download and upload manger.

This solution has helped me leap frog over the 100TB to 1PT local solution and just stick with a very fast local 50TB solution.

Also, there is a USB-C 10G adapter out there. But you're still going to have IO response time and queuing issues if your trying to run the files from the server. Same reason why you shouldn't process scans from a USB.
I don't want to run directly of the nas for registering etc. Enough space to store running projects on the desktop. I just want to export to laz, copy to the nas and import in recap for exemple. Then zip this recap directly to the ftp server without the need to copy the files all over the place.

I spend way to much time post-scanning.... with these issues.
A good NAS will do that for you. My Synology has a number of zip and FTP functions.
Daniel Loney - Owner
Excelsior Measuring inc.
Vancouver - Okanagan - Calgary
www.ExcelsiorLevel.com
Post Reply

Return to “Other Hardware: Field & Back Office”