fast storage solution
- landmeterbeuckx
- V.I.P Member
- Posts: 1615
- Joined: Tue May 01, 2012 5:19 pm
- 11
- Full Name: Lieven Beuckx
- Company Details: Studiebureau Beuckx
- Company Position Title: Owner
- Country: Belgium
- Linkedin Profile: Yes
- Has thanked: 183 times
- Been thanked: 548 times
fast storage solution
Hello,
The amount of data on my system is going crazy. I'm searching for a system to work with for all my scanning needs.
Currently i'm working on the system drives C and D of my main pc but it gets full rather fast and i'm constantly moving files along.
Would this https://www.synology.com/nl-nl/products/FS1018#features system be something to work on with current projects and move to backup later on?
The amount of data on my system is going crazy. I'm searching for a system to work with for all my scanning needs.
Currently i'm working on the system drives C and D of my main pc but it gets full rather fast and i'm constantly moving files along.
Would this https://www.synology.com/nl-nl/products/FS1018#features system be something to work on with current projects and move to backup later on?
- stevenramsey
- Global Moderator
- Posts: 1937
- Joined: Sun Aug 12, 2007 9:22 pm
- 16
- Full Name: Steven Ramsey
- Company Details: 4DMax
- Company Position Title: Technical Specialist Scanning
- Country: UK
- Skype Name: steven.ramsey
- Linkedin Profile: Yes
- Location: London
- Has thanked: 30 times
- Been thanked: 72 times
- Contact:
Re: fast storage solution
looks great add the 10gb card and one for the pc or that will be a big bottle neck without
- landmeterbeuckx
- V.I.P Member
- Posts: 1615
- Joined: Tue May 01, 2012 5:19 pm
- 11
- Full Name: Lieven Beuckx
- Company Details: Studiebureau Beuckx
- Company Position Title: Owner
- Country: Belgium
- Linkedin Profile: Yes
- Has thanked: 183 times
- Been thanked: 548 times
Re: fast storage solution
the problem is also the price of ssd's. it would cost a fortune to fill it up with just 1TB ones
- blazaj
- V.I.P Member
- Posts: 510
- Joined: Tue Jan 26, 2010 1:07 pm
- 14
- Full Name: Jiri Blazek
- Company Details: BW precision systems sro
- Company Position Title: Managing Director
- Country: Czech Republic
- Linkedin Profile: No
- Has thanked: 45 times
- Been thanked: 18 times
- Contact:
Re: fast storage solution
I don't think it is recommended to put SSD's in a NAS server, but you better consult an IT expert. I have a 512 GB fast SSD for actual projects and 4Gig NAS on gigabyte LAN filled with 3 HDD's and an external one for backup. It is reasonably fast to move projects around. If I was to upgrade I would go for 1GB SSD for actual projects. I had a couple of projects where there was not sufficient space for registration in Faro Scene, a sole project on the 512GB SSD! It is crazy.
- JohnT2
- I have made 40-50 posts
- Posts: 45
- Joined: Mon Feb 17, 2014 6:33 pm
- 10
- Full Name: John Tocci Jr
- Company Details: PCL Construction
- Company Position Title: VDC Manager
- Country: United States
- Linkedin Profile: Yes
- Been thanked: 1 time
Re: fast storage solution
SSDs in a synology are a supported solution, both as a RAID array, and solid state caching.
Depending upon your current IT deployment, look at both the rack mounted solution (RS series) and the desktop solution (DS series).
The way we run our Synologies is with a Fast IOPS active projects array, and much slower archive "cold storage" array.
To really get the most out of pure-SSD storage array, you'll need a 10 GBE network topology (NAS, switch, network adapters in client machines/servers/NAS, and compatible cables)
Today (July 2018), We've got our users working with local M.2 NVME drives with Microsoft OneDrive actively syncing the projects (Office 365 E5 has unlimited onedrive storage for users), and our Synology running our project drive and archive storage. We've got a gigabit internet pipe, courtesy Verizon FIOS and onedrive bursts very nicely.
We're holding off on a 10GBE network environment for at least the next 12-18 months, since our VDC department operates off laptops, and there isn't a good 10GBE solution there (yet).
If you're on desktops & workstations, that might change the cost/benefit study.
Depending upon your current IT deployment, look at both the rack mounted solution (RS series) and the desktop solution (DS series).
The way we run our Synologies is with a Fast IOPS active projects array, and much slower archive "cold storage" array.
To really get the most out of pure-SSD storage array, you'll need a 10 GBE network topology (NAS, switch, network adapters in client machines/servers/NAS, and compatible cables)
Today (July 2018), We've got our users working with local M.2 NVME drives with Microsoft OneDrive actively syncing the projects (Office 365 E5 has unlimited onedrive storage for users), and our Synology running our project drive and archive storage. We've got a gigabit internet pipe, courtesy Verizon FIOS and onedrive bursts very nicely.
We're holding off on a 10GBE network environment for at least the next 12-18 months, since our VDC department operates off laptops, and there isn't a good 10GBE solution there (yet).
If you're on desktops & workstations, that might change the cost/benefit study.
- gsisman
- V.I.P Member
- Posts: 898
- Joined: Fri Oct 07, 2016 1:51 pm
- 7
- Full Name: Steve Long
- Company Details: Montgomery County DOT _ MD
- Company Position Title: Land Survey Supervisor
- Country: United States
- Skype Name: gsisman1
- Linkedin Profile: Yes
- Has thanked: 767 times
- Been thanked: 149 times
Re: fast storage solution
We are getting ready to provision a second Synolgy NAS for Scan cloud operating, editing, and short term storage on a 1GB networkJohnT2 wrote: ↑Thu Jul 26, 2018 4:53 pm SSDs in a synology are a supported solution, both as a RAID array, and solid state caching.
Depending upon your current IT deployment, look at both the rack mounted solution (RS series) and the desktop solution (DS series).
The way we run our Synologies is with a Fast IOPS active projects array, and much slower archive "cold storage" array.
To really get the most out of pure-SSD storage array, you'll need a 10 GBE network topology (NAS, switch, network adapters in client machines/servers/NAS, and compatible cables)
Today (July 2018), We've got our users working with local M.2 NVME drives with Microsoft OneDrive actively syncing the projects (Office 365 E5 has unlimited onedrive storage for users), and our Synology running our project drive and archive storage. We've got a gigabit internet pipe, courtesy Verizon FIOS and onedrive bursts very nicely.
We're holding off on a 10GBE network environment for at least the next 12-18 months, since our VDC department operates off laptops, and there isn't a good 10GBE solution there (yet).
If you're on desktops & workstations, that might change the cost/benefit study.
Our first one was 48TB (divided into 10GB iSCI and 38GB partitions) and is running a RAID5 configuration across 8 6GB HDD
We do a partial dailt backup using Synology Cloud Backup app to Azure.
Our new one will be a bit bigger 8 10TB (7200rpm HDD.)
We would really like to take advantage of the 10GB Network capability of the NAS to at least one of our Workstations, and I'm assuming that we need a 10TB Switch to handle that connection, even if we are only hooking up one station right now, possible 2-4 more in the future
Just wondering whether there is a better solution than RAID 5 for ultimate READ/WRITE Performance balance.
At the present time we are serving up our Jetstream from this NAS, working our project directory on it, and storing our RAW Data on it. The iSCI portion we are using for storing LGS portable files and RAF Register 360 Archive projects.
When we add the second NAS should we divide the purposes and use one NAS for operation/Editing Projects, and one for Jetstream service and RAW Storage?
Should we have different RAID on each system depending on how it will be most used
....Jetstream or LGS storage mostly all READ
....RAW storage and Project editing Read/Write
- Leandre Robitaille
- V.I.P Member
- Posts: 542
- Joined: Sat Aug 03, 2019 1:53 am
- 4
- Full Name: Leandre Robitaille
- Company Details: Cima+
- Company Position Title: Civil Technician - Surveyor
- Country: Canada
- Has thanked: 61 times
- Been thanked: 246 times
Re: fast storage solution
This is the biggest issue with 3d scanning. We have filled our servers with over 100tb worth of data(we keep only 5 years worth, as per stated in all our quotes). I have several Scene projects that do by themselves over 1tb (usually get 1tb when 2000+scans)
The way we work internally is that everyone has a local ssd on their computer, our whole server setup is 7200rpm (40tb available, active project, raid10)
then we have 60tb in archives, those are slower 5200rpm.
Our workstations have a m.2 controller that plugs in 4x1tb m.2 ssd in raid 0. We use those for our Scene projects /recap etc. Then when finished we drop the files on our 7200rpm server and strongly suggest to anyone using the data to copy it onto their local ssd. In short, dont get a ssd server it is not worth the money. With hdd you can always do data recovery if many drive fails. SSD when it dies its over.
I'm sadly not a IT, simply a surveyor, but I know about this cause im the source for all the headaches my IT team has haha.
The way we work internally is that everyone has a local ssd on their computer, our whole server setup is 7200rpm (40tb available, active project, raid10)
then we have 60tb in archives, those are slower 5200rpm.
Our workstations have a m.2 controller that plugs in 4x1tb m.2 ssd in raid 0. We use those for our Scene projects /recap etc. Then when finished we drop the files on our 7200rpm server and strongly suggest to anyone using the data to copy it onto their local ssd. In short, dont get a ssd server it is not worth the money. With hdd you can always do data recovery if many drive fails. SSD when it dies its over.
I'm sadly not a IT, simply a surveyor, but I know about this cause im the source for all the headaches my IT team has haha.
-
- V.I.P Member
- Posts: 199
- Joined: Wed Apr 06, 2016 3:24 pm
- 7
- Full Name: Brian Boehmer
- Company Details: Meridian Eng.
- Company Position Title: LiDAR Technician
- Country: United States
- Linkedin Profile: No
- Has thanked: 14 times
- Been thanked: 5 times
Re: fast storage solution
We now have two machines with 4tb SSD's internal for our current projects. We found that if we had a large project, everything wouldn't fit on our 2TB SSD with PPC, Scene2Go, Raw Scans, Exported e57's, Recap Project and RCS files. This gets backed up to a NAS drive and the projects go onto 8tb backup drives for permanent storage. Drives are cheap compared to a client calling several years later with more extraction work for you and you having to tell them you deleted it because it was taking up too much space.
I occasionally work on projects located on the NAS without much problem. Just don't try to do any new registrations or anything. Its simply extraction from a loaded point cloud.
I occasionally work on projects located on the NAS without much problem. Just don't try to do any new registrations or anything. Its simply extraction from a loaded point cloud.
-
- V.I.P Member
- Posts: 1237
- Joined: Mon Jan 04, 2010 7:51 pm
- 14
- Full Name: Jed Frechette
- Company Details: Lidar Guys
- Company Position Title: CEO and Lidar Supervisor
- Country: USA
- Linkedin Profile: Yes
- Location: Albuquerque, NM
- Has thanked: 62 times
- Been thanked: 220 times
- Contact:
Re: fast storage solution
If it's a just a small number of workstations you could always just cable them directly without a switch. This is what we do between our primary NAS and the backup NAS that it replicates to, so that we can keep the replication traffic off of the main network.
After switching to ZFS based central storage I would never go back to simple RAID. ZFS is the closest thing to magic (in a good way) I've come across in a while.
Jed
- Matt Young
- Honorary Member
- Posts: 3929
- Joined: Fri Nov 30, 2007 3:03 pm
- 16
- Full Name: Matt Young
- Company Details: Baker Hicks
- Company Position Title: CAD-BIM Lead
- Country: UK
- Linkedin Profile: No
- Has thanked: 45 times
- Been thanked: 40 times
Re: fast storage solution
Just got hold of a 'My Cloud WD Pro Series 8Tb NAS'. It transfers data quickly and is a useful single location for things. Very easy to set up as a mapped or network drive.
If you don't see that there is nothing, then you are kidding yourself.