▲ 1 points

Ask HN: Transfer Large Amounts of Data

by Alifatisk·3y ago·12 comments·view on hn ↗

Hi HN,

My friend has folder on his computer with over 1.100 videos, in total it's about 300 GB.

He wants to hand me over the folder but I have no idea on how he should transfer everything over to my computer in the most efficient way possible.

My idea so far have been,

1. Just grab his harddrive and put it in my computer

2. Bring a portable hard drive over to his house

3. Rsync

4. Airtar / Wormhole

5. Torrent from his computer to mine

Do any of you have any better ideas?

12 comments

If you have access to a portable hard drive that’s pretty doable. Looking online [1] it seems that they can usually transfer at 30 mb/s which means it’ll only take about 3-4 hours.

[1] https://askubuntu.com/questions/41397/external-usb-hard-driv...

That sounds like the best option at the moment.

#1 https://massive.io while pricey per gb ($0.25/GB), you get the first 150GB free, so you'd be paying about $35 for the file transfer. Would be the least amount of effort of any of those scenarios, no real special tools required. Could even drag and drop the folder into the browser.

#2 would be portable hard drive. 300gb of video isn't enough to justify any syncing tools. Just drag and drop.

#3 rclone to Cloudflare R2 maybe, somewhere with no egress fees. Be about $5.

And no wifi, ethernet for data on both sides. You want to saturate the pipe to get it quick. Don't let wifi signal strength be your speed factor.

I found torrenting over even a local network was much slower than directly copying to a portable hard drive.

Most likely you want to settle on a process that verifies the copied files. If your friend uses M$ you could take a simple path and use a program like Terracopy [1] to be sure and just copy them all to a portable hard drive. (I've used it since working on a Win 10 system (whatever version it was) where quicker file copy was achieved by stepping back to iirc win 95 era and not verifying files.)

In linux there are many options - personally I tar folders up to 20 or 30 gigs using --verify (-W) ie tar -cvWf folder1 'folder1'/

... in your case you would need to direct the write of tar to the portable drive.

[1] https://codesector.com/teracopy

We're restricted to Windows sadly. But I'll check out Teracopy.

Take a gigabit network switch over to your friends house and connect both the machines to the same switch. Use windows to discover the shared folder on your friends computer and copy it to yours.

What kind of constraints are you under, or what are you trying to optimize for? In many circumstances that wouldn't be considered a lot of data. Is low bandwidth the issue? If so, physical sharing the hard drive, or even mailing a usb hard drive makes sense. How valuable is the data and how much is your time worth?

I don't think none of us are in any constrains, we both have high-speed network (around 300 Mb/s down and 50 Mb/s up).

What I am trying to optimize for is the transfer rate, I would prefer this transfer takes less than a couple of hours.

The data is not that valuable, it would just be nice to have.

Just try transfering it with sftp and see what kind of speed you get. Maybe use tar first to make a few big files, which will reduce overhead

Interesting, I'll add this to my list.

Portable hard drive is probably the most practical. Another option not on this list would be rclone [1].

[1] https://rclone.org/docs/