Best solution for large file sharing over internet?

3DUserNet, Pointbox, Potree, TrueView, WebShareCloud, Etc.
Post Reply
Jonny5isAlive
I have made 50-60 posts
I have made 50-60 posts
Posts: 50
Joined: Thu Feb 18, 2010 5:01 pm
Full Name: Jon Adams
Company Details: Forensic and Architectural Consulting
Company Position Title: Director of 3D Reality Capture
Country: USA
Linkedin Profile: Yes
Location: PA

Best solution for large file sharing over internet?

Post by Jonny5isAlive » Tue Feb 19, 2019 9:59 pm

Hello-

I am exploring ideas regarding large dataset sharing over the internet, and wondering what some opinions/options might be.

Would like to determine best option for remote work, where large RAW files can be downloaded/copied from server, worked on (registration/cleaning/finalizing), then returned to server when complete.

Data these days can easily be in the 40-60GB range for us.

Have considered Dropbox, but with 40-60GB zipped files, does not seem ideal.

Any thoughts on alternatives/workflows?

Thanks!

User avatar
gordonired
I have made 60-70 posts
I have made 60-70 posts
Posts: 69
Joined: Thu Mar 09, 2017 4:27 pm
Full Name: Gordon
Company Details: IRED
Company Position Title: Support Engineer
Country: Canada
Linkedin Profile: Yes
Location: Alberta
Contact:

Re: Best solution for large file sharing over internet?

Post by gordonired » Tue Feb 19, 2019 10:20 pm

Have you looked into hosting an FTP server?

jedfrechette
V.I.P Member
V.I.P Member
Posts: 864
Joined: Mon Jan 04, 2010 7:51 pm
Full Name: Jed Frechette
Company Details: Lidar Guys
Company Position Title: CEO and Lidar Supervisor
Country: USA
Linkedin Profile: Yes
Location: Albuquerque, NM
Has thanked: 2 times
Been thanked: 8 times
Contact:

Re: Best solution for large file sharing over internet?

Post by jedfrechette » Wed Feb 20, 2019 1:02 am

+1 for going old school with FTP.

After going through a whole range of solutions over the years from Dropbox to transfer services specifically targeted towards large data like Aspera and Signiant I'm happily relying on good old FTP.

It may not be fancy, but at the end of the day it just works and is usually faster than anything else I've tried. Two suggestions if you do go with FTP though.

1. Make sure transfers are encrypted using either TLS of SFTP.
2. Don't make one giant zip file before transferring. Transferring multiple files in parallel is much faster than one large file at a time and if one file does get corrupted you won't have as much data to retransfer.
Jed

User avatar
steves01x
V.I.P Member
V.I.P Member
Posts: 298
Joined: Tue Jul 21, 2015 3:58 pm
Full Name: Steve Smith
Company Details: Scanning
Company Position Title: Surveyor
Country: Scotland
Linkedin Profile: Yes
Has thanked: 3 times
Been thanked: 5 times

Re: Best solution for large file sharing over internet?

Post by steves01x » Wed Feb 20, 2019 8:48 am

Dropbox is good as the app resumes smoothly almost all the time - not all FTP clients/servers like being paused.

Use winrar or similar to split the file up nicely.

User avatar
smacl
V.I.P Member
V.I.P Member
Posts: 227
Joined: Tue Jan 25, 2011 5:12 pm
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 19 times
Been thanked: 23 times
Contact:

Re: Best solution for large file sharing over internet?

Post by smacl » Wed Feb 20, 2019 11:32 am

Jonny5isAlive wrote:
Tue Feb 19, 2019 9:59 pm
Would like to determine best option for remote work, where large RAW files can be downloaded/copied from server, worked on (registration/cleaning/finalizing), then returned to server when complete.
If you really need the data at the remote location, I'd use FTP and find it quite reliable. A better solution would be to remote into a workstation connected directly to the server and skip all the file transfer entirely. I use TeamViewer for this which is ok, though I've heard there are better and cheaper solutions available at this point. You'll need a decent broadband connection if you're using high resolution displays but if you're contemplating sending many GBs backwards and forwards, I guess this isn't an issue. For transferring very big data, posting an SSD is still way faster than using the internet in this part of the world.

User avatar
LPaulCook
V.I.P Member
V.I.P Member
Posts: 354
Joined: Sat Jul 18, 2015 9:42 pm
Full Name: L Paul Cook
Company Details: Land surveyor doing LiDAR
Company Position Title: President
Country: United States
Linkedin Profile: No
Been thanked: 4 times
Contact:

Re: Best solution for large file sharing over internet?

Post by LPaulCook » Wed Feb 20, 2019 2:43 pm

I regularly use Sendthisfile.com and Firefox to send any amount of data to anyone. IE and Chrome are limited to 4GB but Firefox is not.

Someone has to be on the other end to click on the link to download the file. Or you could remotely do that from anywhere.

Compressing the files first saves a lot of time before uploading. You can also set how many times the files can be downloaded too. i.e. maximum 1 time or 10 times etc.
L. Paul Cook, PLS
www.LPC3D.com

User avatar
smacl
V.I.P Member
V.I.P Member
Posts: 227
Joined: Tue Jan 25, 2011 5:12 pm
Full Name: Shane MacLaughlin
Company Details: Atlas Computers Ltd
Company Position Title: Managing Director
Country: Ireland
Linkedin Profile: Yes
Location: Ireland
Has thanked: 19 times
Been thanked: 23 times
Contact:

Re: Best solution for large file sharing over internet?

Post by smacl » Wed Feb 20, 2019 3:22 pm

Depends to an extent on the country and the type of broadband internet connection available. Here in Ireland, most fibre broadband is only available as ADSL at 1:10 upload:download ratio, e.g. 10mbps upload speed, 100mbps download. Great from streaming movies, not so good for uploading scans. My colleagues in the UK have DSL at 1:1 upload:download and 40mbps each way which is way handier even though slightly lower in total bandwidth.

User avatar
infrared_sight
I have made 10-20 posts
I have made 10-20 posts
Posts: 12
Joined: Thu Aug 17, 2017 12:18 pm
Full Name: Charles Thomson
Company Details: Correvate
Company Position Title: Product Manager
Country: UK
Linkedin Profile: Yes
Location: London
Contact:

Re: Best solution for large file sharing over internet?

Post by infrared_sight » Wed Feb 20, 2019 3:42 pm

This is an interesting topic generally. We as a company have been thinking about this a lot. We see the full workflow on the cloud as the most optimal future path. Speed obviously has a bearing as, ideally, the data is uploaded from the field and can begin being processed immediately from any hardware with a web browser rather than having to keep triaging it on and offline.

We will be launching a closed beta of our cloud computing point cloud registration service soon and anyone who is interested in trying it out is welcome to PM me or David Gray or email [email protected] and we'll keep you informed of when it is available.
Charles Thomson, Product Manager
Vercator software - fast, automatic point cloud registration
Desktop • Cloud • Mobile • www.vercator.com

User avatar
landmeterbeuckx
V.I.P Member
V.I.P Member
Posts: 398
Joined: Tue May 01, 2012 5:19 pm
Full Name: Lieven Beuckx
Company Details: Studiebureau Beuckx
Company Position Title: Owner
Country: Belgium
Linkedin Profile: Yes
Has thanked: 11 times
Been thanked: 20 times

Re: Best solution for large file sharing over internet?

Post by landmeterbeuckx » Wed Feb 20, 2019 4:29 pm

The problem lies also with the data plans. A general Riegl project for 1 day with pics can get upto 70GB. Not possible through 4g imho
LSBbvba
Surveying services - 3D Laserscanning
Tel : +3211661621
www.landmeter-beuckx.be
[email protected]

User avatar
steves01x
V.I.P Member
V.I.P Member
Posts: 298
Joined: Tue Jul 21, 2015 3:58 pm
Full Name: Steve Smith
Company Details: Scanning
Company Position Title: Surveyor
Country: Scotland
Linkedin Profile: Yes
Has thanked: 3 times
Been thanked: 5 times

Re: Best solution for large file sharing over internet?

Post by steves01x » Wed Feb 20, 2019 5:27 pm

For us in the UK (home connections) unless a private fiber company installs something in your area you will never really get a good upload... as all packages are capped at around the 20mb/s upload connection so they do not loose their business customers on their leased line packages - something the Virgin person on the phone revealed to me when trying to upgrade.

I asked BT and Virgin home much a home business connection would be.... BT got back to me, 4 year deal, £14k to install less a government grant if available then £975+VAT a month for 100mb up/down.... :lol:

I know people within a few miles dont even get 5mb/s connection so i fear any cloud / mass data sharing is a long way off in many parts of UK, well Scotland and Wales :)

Post Reply

Return to “Web Based Software”