Tool for transfer big files (60-100gb) reddit / selfhosted ---------------------------------------------------------------------- doctor-ase · 2023-10-14 21:03 A lot of file transfer doens't support this size and i've tried with synology file satation share option, one friend downloaded succesfully the file, but another friend the download stops when the file is downloading. I'm looking for a simple method, my friends doesn't know how to manage pc things. It can be by means of a program that you have to download, but that you do not have to make too many setup. ---------------------------------------------------------------------- FallenFromTheLadder · 2023-10-14 21:15 Split the file in multiple chunks. Send them separately. Your friend can rebuild the entire file together. You just need double the space since at some point there will be both the chunks and the whole file at the same time. ---------------------------------------------------------------------- 6ftClaud · 2023-10-14 21:20 Creating a torrent would work ---------------------------------------------------------------------- Ox450x6c · 2023-10-14 21:28 rsync ---------------------------------------------------------------------- dazchad · 2023-10-14 21:54 SFTP should work too ---------------------------------------------------------------------- Murky-Sector · 2023-10-14 21:57 Use split. its included with all the major linux distros. ---------------------------------------------------------------------- Hamza9575 · 2023-10-14 22:05 Use torrent. ---------------------------------------------------------------------- Looserette · 2023-10-14 22:10 free, open-source, encrypted, cross-platform and allows resuming of file transfers: [https://github.com/schollz/croc#install](https://github.com/schollz/c roc#install) ---------------------------------------------------------------------- maccmiles · 2023-10-14 22:33 If you're on windows tools like winrar have split functions built in, then your friends only need to have all the parts in a folder and the tool will reassemble the file automatically. I hear they have a free 40 day trial for new installs... ---------------------------------------------------------------------- prime_1996 · 2023-10-14 22:45 Syncthing might work too ---------------------------------------------------------------------- dhalem · 2023-10-14 22:58 Torrent does this by design ---------------------------------------------------------------------- ex800 · 2023-10-14 23:20 syncthing would be the simplest ---------------------------------------------------------------------- 10F1 · 2023-10-14 23:45 rsync, rclone, split. ---------------------------------------------------------------------- Blackops12345678910 · 2023-10-14 23:50 Sftp has resume capabilities so any partial download can be resumed after a network disconnect for example ---------------------------------------------------------------------- chrissie_brown · 2023-10-15 00:13 This ---------------------------------------------------------------------- crypto2mex · 2023-10-15 00:59 Came to say this. ---------------------------------------------------------------------- fixjunk · 2023-10-15 01:23 overnight a flash drive. ---------------------------------------------------------------------- zhzhzhzhbm · 2023-10-15 02:12 [https://transfer.zip/](https://transfer.zip/) if it's okay to send over web browser. ---------------------------------------------------------------------- CeeMX · 2023-10-15 02:16 Would I need a tracker for that? Recently tried it and since private torrents have DHT disabled and I didn’t add a tracker it didn’t work at all ---------------------------------------------------------------------- zarlo5899 · 2023-10-15 02:46 keep DHT on and just encrypt the files ---------------------------------------------------------------------- CeeMX · 2023-10-15 02:47 Selecting Private Torrent disables DHT ---------------------------------------------------------------------- mitchboy999 · 2023-10-15 02:51 Just leave it as 'public' then. It won't really be public unless you share the file anyway. ---------------------------------------------------------------------- zarlo5899 · 2023-10-15 03:23 i was inferring that you keep it public ---------------------------------------------------------------------- lechauve911 · 2023-10-15 03:38 Resilio Sync free, uses p2p. 1. download and install, 2. create a shared folder 3. send the link 4. person can download from browser or can install the app ---------------------------------------------------------------------- Official-Wamy · 2023-10-15 04:34 rsync is literally the best answer ---------------------------------------------------------------------- Looserette · 2023-10-15 04:39 Although this is true for most of us: "my friends doesn't know how to manage pc things" <= good luck getting them to generate a SSH key and all the little details ---------------------------------------------------------------------- abjedhowiz · 2023-10-15 04:39 You would need rsync on both ends though? ---------------------------------------------------------------------- abjedhowiz · 2023-10-15 04:40 Curious, how do you break a single file into chunks? ---------------------------------------------------------------------- Official-Wamy · 2023-10-15 04:41 yup ---------------------------------------------------------------------- carl2187 · 2023-10-15 04:59 Or use 7zip, open source free, does the same things on windows. ---------------------------------------------------------------------- _newtesla · 2023-10-15 05:16 Zip, rar, tar…? ---------------------------------------------------------------------- abjedhowiz · 2023-10-15 05:18 That zips the whole file. How do you break a single file down into multiple parts? ---------------------------------------------------------------------- _newtesla · 2023-10-15 05:23 Multipart zip, also rar and tar. ---------------------------------------------------------------------- maomaocake · 2023-10-15 05:47 Buy a connectx 7 set up rdma the use rcopy /s somewhat ---------------------------------------------------------------------- michaelpaoli · 2023-10-15 07:02 >Tool for transfer big files (60-100gb) Most anything that allows an uncompleted transfer to be resumed. I'm sure there are well over a dozen possibilities ... probably many dozens. Just some possibilities that jump to mind (some may require relevant options, version(s), preparation, and/or server or the like): * curl * wget * rsync * dd * jigdo * zsync * BitTorrent * FTP ---------------------------------------------------------------------- FallenFromTheLadder · 2023-10-15 07:14 Oh, sweet summer child... ---------------------------------------------------------------------- FallenFromTheLadder · 2023-10-15 07:16 Yes, but the other person is not tech savy. Splitting the file means they can do what they already know, which is downloading files from we hosting sites. Then you just click on the first file and they do the "magic" for you. ---------------------------------------------------------------------- Kompost88 · 2023-10-15 07:42 Still a popular method to distribute movies to cinemas (DCP's are usually 100-200GB). ---------------------------------------------------------------------- hotapple002 · 2023-10-15 07:42 It should, but it would be slow as shit. ---------------------------------------------------------------------- SEND_NUKES_PLS · 2023-10-15 08:12 Syncthing. ---------------------------------------------------------------------- aew3 · 2023-10-15 09:05 zip and rar have the ability to create multi file archives. Not sure if tarballs can be split tho. ---------------------------------------------------------------------- Sebcorgan · 2023-10-15 09:20 Why ? ---------------------------------------------------------------------- lilolalu · 2023-10-15 09:59 I think with only one seeder that will be incredibly slow. Torrent is a peer to peer technology and not a point to point. ---------------------------------------------------------------------- lilolalu · 2023-10-15 10:01 I think the most promising - unless someone will make a UDT based upload/download Server - is tus.js with or without uppy. https://github.com/tus/tusd https://uppy.io/ ---------------------------------------------------------------------- Bonsailinse · 2023-10-15 10:42 It is absolutely not if one party involved doesn’t know much about pcs, which is the case here. ---------------------------------------------------------------------- ngnirmal · 2023-10-15 10:47 This is in no way the best way to do it, but by far the easiest. 7 zip is available everywhere these days. There is an option to set the max archive size of the given input file/ dir. Having set this, start zip process. ---------------------------------------------------------------------- iProModzZ · 2023-10-15 10:52 Well, it’s literally the same speed. ---------------------------------------------------------------------- Fr3shlama · 2023-10-15 10:53 Trust me bro ---------------------------------------------------------------------- lilolalu · 2023-10-15 10:54 Sorry, I don't think you understood how the torrent protocol works. If you have a single server/seeder (call it what you want) there are much faster ways to transfer files. ---------------------------------------------------------------------- protocolnebula · 2023-10-15 11:13 If you have thousands of files, it will, otherwise I never have speed issues transferring files with FileZilla for example ---------------------------------------------------------------------- iProModzZ · 2023-10-15 12:10 Well, tell me? I use torrent protocol daily using resilient sync, i get the maximum upload speed of my home server connection to my client, how should anything be faster? ---------------------------------------------------------------------- lilolalu · 2023-10-15 12:14 1. UDT (UDP based) transfer (or quick, tsunami)... Here is a udt wrapper for rsync: https://github.com/allisonheath/UDR 2. Protocols featuring parallel transfers, like tus.js - or BitTorrent but only if you have multiple seeders ---------------------------------------------------------------------- lilolalu · 2023-10-15 12:20 That are some "self hostable" options, i.e. in the broadcast video world, the networks commonly use file transfer services like IBM Aspera (UDP transfers) for large files. ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-15 12:21 Syncthing would be my recommendation Not hard for them to download it and run it, link up and accept a folder Then you just leave it until it's done Not much harder than that ---------------------------------------------------------------------- henry_tennenbaum · 2023-10-15 13:01 Actually super hard. Setting up folders seems to be very confusing for new users. No issue if you're there to help or have remote access, but the folder IDs are intimidating for some people. ---------------------------------------------------------------------- Tafelbomber · 2023-10-15 13:20 Could you elaborate why that is? I don't know the protocol's details either and always thought that a single seeder would upload at the same speed. I always thought that was how tools like resilio sync for example worked without slowing you down ---------------------------------------------------------------------- dazchad · 2023-10-15 13:30 Why is that? ---------------------------------------------------------------------- lilolalu · 2023-10-15 13:32 A single connection can usually not max out your bandwidth, because of the tcp/ip overhead etc. Thats why its faster to do parallel connections, even if its from one host to another host. BitTorrent does support multiple parallel connections, but NOT (as far as i know) if used for a single file with a Point to Point transfer. BitTorrent by default does this for "Peer to Peer" connections, where you are grabbing the same file simultaneously from several hosts. As others have mentioned, you can start breaking down your "huge file" into smaller chunks to have several simultaneous connections, which for me seems to be merely a workaround to the problem, but its not something that BitTorrent does by default, unlike other transfer protocols. Like tus.js. Other transfer protocols like UDT (or Quic, Tsunami, IBM Aspera, GridFTP) send data via UDP Packets as data bursts, so they dont wait for every single TCP packet to arrive safely and then send the next. They just send everything as fast as they can and later an algorithm creates checksums and re-sends the packets that were faulty. You can get 2-10 times faster transfer speeds with udp based file transfers compared to regular tcp/ip based transfers. If you are sending dozens or hundreds of GB's which is what OP was asking for, this can make a big difference. It will not make a significant difference for a couple of MB. ---------------------------------------------------------------------- dazchad · 2023-10-15 13:32 Doesn’t make sense why a single giant file would be slow. I agree the multiple files could be a bit slower than other programs ---------------------------------------------------------------------- lilolalu · 2023-10-15 14:47 Anecdotal test transferring a 1GB File, created with "openssl rand -out 1gb.bin -base64 $(( 2\*\*30 \* 3/4 ))) " to my VPS (1Gig LAN) from my 25Mbps uplink with a theoretical max rate of 3.125 MB/s ... Psitransfer (using tus.js):2.9MB/s SCP: 2.7 MB/s Upload via Browser to the minimal Nextcloud i have running on the VPS: 2.4MB/s ---------------------------------------------------------------------- Tafelbomber · 2023-10-15 14:51 Thank you! That makes sense! ---------------------------------------------------------------------- Compux72 · 2023-10-15 14:54 But i feel its too slow for LAN transfers. Probably due to the release server right? ---------------------------------------------------------------------- DREveritt · 2023-10-15 16:23 Yes, Syncthing all day long Multi-platform too! Share files with my parents phones all the time! (using parents as an example as not very "tech savy" ---------------------------------------------------------------------- jesusrambo · 2023-10-15 17:25 Yeah, using a relay server for a LAN transfer will be a bottleneck ---------------------------------------------------------------------- doctor-ase · 2023-10-15 18:02 My friend need to open ports to recieve my file across the internet? ---------------------------------------------------------------------- doctor-ase · 2023-10-15 18:03 My friend needs to open ports to recieve my file across the internet? ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-15 18:11 No, Syncthing can use a relay or you can allow inbound TCP 22000 from your friends IP to speed it up a bit ---------------------------------------------------------------------- geo38 · 2023-10-15 21:05 This is sort of cool. Thanks for suggesting it. ---------------------------------------------------------------------- hotapple002 · 2023-10-16 05:04 SSH isn’t made for big file transfers and (depending on the client) it will be single treaded. (Speaking from experience. Highest transfer speed I have had was with FileZilla and like 10 threads.) ---------------------------------------------------------------------- hotapple002 · 2023-10-16 05:04 SSH isn’t made for big file transfers and (depending on the client) it will be single treaded. (Speaking from experience. Highest transfer speed I have had was with FileZilla and like 10 threads.) ---------------------------------------------------------------------- hotapple002 · 2023-10-16 05:05 With FileZilla it can be okay as it uses multiple threads/connections. Other sftp clients (for example sshfs and Termius) only use 1 connection. ---------------------------------------------------------------------- Sebcorgan · 2023-10-16 09:23 Oh yeah, tell me about something like rsync which is very bad at transfering big files ---------------------------------------------------------------------- Trip4004 · 2023-10-16 12:54 Look into BITS. ---------------------------------------------------------------------- hotapple002 · 2023-10-16 17:14 Might have over exaggerated my statement a bit indeed. ---------------------------------------------------------------------- kingb0b · 2023-10-20 06:13 Flash drive ---------------------------------------------------------------------- Mundane-Highlight-27 · 2023-11-11 16:18 try [file.kiwi](https://file.kiwi)