Hosting large files reddit / selfhosted ---------------------------------------------------------------------- AnomalyNexus · 2023-12-14 20:45 I need a self-host way to share large files (regularly transferring say 10 gig files) with a third party (read only one way is fine). I'd love to hear what sort of tech you'd pick for this: * Server side (me) has static Ipv4 & reliable 1 gig up on a proxmox setup so server side can be whatever it needs to be software wise * Download side is a fast but potentially shaky connection so need something that can recover from outages * Has to be authenticated * End user is not technical per se, but competent in the generic sense so could manage say sFTP login with keys if necessary * I'll be VM'ing this but still has to be pretty solid security wise since I'm exposing this on a static ip (Will probably region lock ufw it) * Password auth preferred, but can do keys if necessary * End user is on a windows system and can install custom software as long as it is reasonably credible (deluge, winscp etc) Leaning towards something sftp-ish but unsure how well that copes with flakey connections. Torrent would be ideal on fault tolerance but then I'd need to encrypt the files manually which feels like a pain. Something file-run-ish would also work I guess but if I can dodge the whole SSL thing I will. I'd rather explain ssh keys for sftp than deal with ssl and domains. Something that can just hit the static ip would be better. Wireguard style point to point tunnel, tailscale etc is not an option. I don't control end user side enough to make that happen and not confident I can get that past end user technical ability remotely. I don't want a nextcloud type custom software solution - something I can stick on a vanilla debian box please. ---------------------------------------------------------------------- thesarthakjain · 2023-12-14 21:00 remindme! 10days ---------------------------------------------------------------------- RemindMeBot · 2023-12-14 21:01 I will be messaging you in 10 days on [**2023-12-24 21:00:39 UTC**](ht tp://www.wolframalpha.com/input/?i=2023-12- 24%2021:00:39%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/18ii893/hosting_large _files/kddf0dz/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F18ii893%2Fhosting_large_files%2Fkddf0dz%2F%5D% 0A%0ARemindMe%21%202023-12-24%2021%3A00%3A39%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%2018ii893) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- adamshand · 2023-12-14 21:01 croc? magic wormhole? ---------------------------------------------------------------------- Sekhen · 2023-12-14 21:01 I'd webhost it. lighttpd or something. One way, super easy, everyone understand a link, any browser can do it. ---------------------------------------------------------------------- thekrautboy · 2023-12-14 21:04 Try Seafile, maybe SFTPGo if the client can/wants to use a SFTP client etc. ---------------------------------------------------------------------- ScribeOfGoD · 2023-12-14 21:12 resilio sync ---------------------------------------------------------------------- ElevenNotes · 2023-12-14 21:13 SFTP container with read only access to the files you want to share. ---------------------------------------------------------------------- auzzlow · 2023-12-14 21:15 I've synced some pretty huge files with Syncthing. Not sure about fault tolerance though.. all of my stuff usually stays online. I'm actually open to reasons why not to use Syncthing for this, and interested in these alternatives as well. ---------------------------------------------------------------------- NobodyRulesPenguins · 2023-12-14 21:25 A simple mod_dav + mod_davfs from Apache2 behind a basic auth page? ---------------------------------------------------------------------- tough_leek · 2023-12-14 21:26 Syncthing ---------------------------------------------------------------------- esturniolo · 2023-12-14 21:32 And what about FTPS? You can resume the downloads and using SSL the transfer in encrypted. ---------------------------------------------------------------------- eddyizm · 2023-12-14 21:37 Syncthing if they can install an app. As long as it stays running anytime you drop a file and will be delivered effortlessly. Outside of that the sftp seems the next best. ---------------------------------------------------------------------- ProKn1fe · 2023-12-14 21:39 Nextcloud ---------------------------------------------------------------------- jeffrey_smith · 2023-12-14 21:39 https://filebrowser.org/ ---------------------------------------------------------------------- Stitch10925 · 2023-12-14 21:49 What about PsiTransfer? It's like a self-hosted WeTransfer: [https://github.com/psi-4ward/psitransfer](https://github.com/psi- 4ward/psitransfer) ---------------------------------------------------------------------- TearDrainer · 2023-12-14 21:49 \+1 for croc, works like a charm for huge files. Most foolproof way for a shaky connection probably would be a torrent. ---------------------------------------------------------------------- cribbageSTARSHIP · 2023-12-14 21:54 I'm running nextcloud AIO and haven't been able to upload files larger than 5gigs ---------------------------------------------------------------------- thekrautboy · 2023-12-14 21:55 But how would you ensure resuming a broken download when using just a browser at a size of 10gb? The client would need to install some kind of browser extension or download manager. OP mentions the potentially shaky connection so downloading with just a browser with no resume, i dont know. ---------------------------------------------------------------------- ProKn1fe · 2023-12-14 21:57 Its webserver configuration limits, not nextcloud. ---------------------------------------------------------------------- cribbageSTARSHIP · 2023-12-14 21:58 Pardon? ---------------------------------------------------------------------- tbleiker · 2023-12-14 22:00 Maybe https://send.vis.ee/ End-to-end encryption and can bei self hosted. ---------------------------------------------------------------------- ferrybig · 2023-12-14 22:06 > Something file-run-ish would also work I guess but if I can dodge the whole SSL thing I will. I'd rather explain ssh keys for sftp than deal with ssl and domains. You could make an account in your server, then allow rsync into the server via ssh. rsync can be interrupted at any time, and can even resume partially transferred files with the `--append-verify` flag ---------------------------------------------------------------------- ferrybig · 2023-12-14 22:08 Browsers are good at detecting an interrupted file transfer, they see the server advertising an large content-length, but if the body is suddenly cut of, they report an error, requiring the user to retry it. At this point, it makes a byte range request to the server to get the remainder of the bytes. Both modern Chromium and Firefox behave this way if the server sends the content-length and advertises supporting byte ranges ---------------------------------------------------------------------- chkno · 2023-12-14 22:18 [git-annex](https://git-annex.branchable.com/) You get solid, full-featured, already-widely-known encryption and authentication in the normal git way, via ssh. git-annex file transfers resume when interrupted. ---------------------------------------------------------------------- purged363506 · 2023-12-14 22:20 Magic wormhole ---------------------------------------------------------------------- thekrautboy · 2023-12-14 22:21 Thats the theory yes, the reality is quite different in my experience using both ff and chrome. ---------------------------------------------------------------------- Got2Bfree · 2023-12-14 22:27 Nextcloud is horrible for that when you don't have fiber internet. ---------------------------------------------------------------------- myelrond · 2023-12-14 22:29 Take a look at liquidfiles.net. self hosted,.auto updating virtual appliance. ---------------------------------------------------------------------- blusls · 2023-12-14 22:44 Try syncthing. https://syncthing.net/ My friend and I use it to sync our plex servers. Works great! Set and forget. ---------------------------------------------------------------------- JellyfishCultural765 · 2023-12-14 23:07 I'd use copyparty ---------------------------------------------------------------------- Mean_Einstein · 2023-12-14 23:13 IPFS could be an option ---------------------------------------------------------------------- LeJWhy · 2023-12-14 23:16 It also depends on the server configuration, i.e. if it is configured for byte serving. ---------------------------------------------------------------------- spusuf · 2023-12-14 23:33 That's correct, it's either php-ftm, nginx, Apache, etc that dictates max block size and max transfer size. ---------------------------------------------------------------------- spusuf · 2023-12-14 23:35 I've tried Nextcloud and Owncloud and they both kinda suck between bloat, performance or needing heaps of configuration. I've switched to Seafile with good results. Resilio sync and those other sync style services aren't what OP is asking for, no client, no change detection, just host and download. ---------------------------------------------------------------------- thekrautboy · 2023-12-14 23:37 Yep. ---------------------------------------------------------------------- blue2020xx · 2023-12-14 23:39 Do some searching bro. Easily solvable ---------------------------------------------------------------------- nunogrl · 2023-12-14 23:49 This is good, if you have a VPN, you change the SSH cypher to improve speed. ---------------------------------------------------------------------- nunogrl · 2023-12-14 23:50 Would you consider torrent? ---------------------------------------------------------------------- idontbelieveyouguy · 2023-12-14 23:55 What ssh cypher do you change to for increased speed? ---------------------------------------------------------------------- nunogrl · 2023-12-14 23:57 I used arcfour, but it is known for security issues. I was using it on intranet. ---------------------------------------------------------------------- idontbelieveyouguy · 2023-12-15 00:15 Interesting, guess I'll do some research. ---------------------------------------------------------------------- xskydevx · 2023-12-15 01:24 https://min.io/ open source. S3 compatible + http ---------------------------------------------------------------------- jla0 · 2023-12-15 01:28 https://zend.to/ I've used it and it's great. Support files up to 20 gig in size. Simple to use and you can send requests for dropoffs or others can drop files. ---------------------------------------------------------------------- blusls · 2023-12-15 02:07 They also have Windows and Linux clients, as well as iOS and Android. ---------------------------------------------------------------------- sirrush7 · 2023-12-15 02:46 ERMAGERD THIS IS WHAT I'VE BEEN DREAMING OF!!! Thank you!!! ---------------------------------------------------------------------- laffer1 · 2023-12-15 03:02 It depends on the security needed. rsync would work with smaller files but that’s a bit large for it A zfs send receive over ssh could work also. A web server could work if configured properly and they could receive with command line downloader like aria2. I use rsync for transferring packages between servers for my bsd project as well as ISOs ---------------------------------------------------------------------- blusls · 2023-12-15 03:47 No worries! ---------------------------------------------------------------------- ItalyPaleAle · 2023-12-15 04:14 Any static HTTP web server, including Apache2, nginx, etc. Client can use something like JDownloader2 to download the file. If the connection drops, JDownloader2 can resume the transfer. ---------------------------------------------------------------------- xkcd__386 · 2023-12-15 04:16 filebrowser is the easiest I can think of -- no special tech skills needed for your audience. Has userid/password (probably also 2FA but I am not sure). Or you can create time-limited public URLs (they look like https://foo.example.com/api/public/dl/AKkjX0iE/samplefile.zip -- note the random component before the filename) that _anyone_ can use, without a userid/password syncthing is the best "set it up once and forget about it" solution. Copy a file to the sync-ed directory and it automagically appears (eventually) on the other side. ---------------------------------------------------------------------- bucknthompson · 2023-12-15 05:24 https://filebrowser.org/ ---------------------------------------------------------------------- ElevenNotes · 2023-12-15 06:17 Don’t. Use ed25519 for the key exchange and just make sure both server and client use NTRU (which should be the case by default if you update your OpenSSH since 2022). You can enter *ssh -Q kex* and check for sntrup761x25519-sha512@openssh.com or similar ntru algorithm. ---------------------------------------------------------------------- ElevenNotes · 2023-12-15 07:16 aria2 comes to mind ---------------------------------------------------------------------- TorSenex · 2023-12-15 08:15 I use rsync over nebula for the same purpose. Mostly to schedule during off hours and throttle the transfer. ---------------------------------------------------------------------- Blackops12345678910 · 2023-12-15 08:21 Sftp has trouble over wan. Speed can drop rapidly when latency is involved An https static file server with basic auth should be ok. Something like nginx or Dufs (on GitHub) should be suitable. I personally use dufs and it’s solid. Assuming you use basic auth, the download can be retried via the browser built in download manager as they both support the range headers required to resume downloada ---------------------------------------------------------------------- douglagm · 2023-12-15 08:25 Also has a scriptable API. Brilliant software ---------------------------------------------------------------------- dumbasPL · 2023-12-15 09:29 When it comes to flaky connections, Torrent is probably one of the best options out there. If you don't want to encrypt files and you know the IP of the other side you could run your own tracker and limit who can access the tracker and the torrent ports on your firewall. ---------------------------------------------------------------------- michaelpaoli · 2023-12-15 09:38 So, how 'bout rsync over ssh? ---------------------------------------------------------------------- Blackops12345678910 · 2023-12-15 10:00 FTPS has issues when Nat is involved. Depending on router setup, the control channel connection terminates so not ideal imo ---------------------------------------------------------------------- jla0 · 2023-12-15 13:37 Not sure why you got down voted. It's awesome software and does exactly what it does. It was started at the University of Southampton and should be recognized for what it is. ---------------------------------------------------------------------- jassalmithu · 2023-12-15 14:15 File downloaders can add accounts and download from authenticated sites. ---------------------------------------------------------------------- blusls · 2023-12-15 14:15 I get that, and it makes sense. I know we can throttle the transfer speed in Syncthing, but I don't recall if there is a scheduling mechanism. I usually don't worry about throttling it, though. We both have high-speed internet access. ---------------------------------------------------------------------- idontbelieveyouguy · 2023-12-15 15:15 Just fyi: I didn't mean I would look into arcfor, I meant I would look into using different ciphers to increase throughput. Obviously security is the highest priority. ---------------------------------------------------------------------- aadoop6 · 2023-12-15 15:18 Croc is a really good program. I use it as a lazy man's replacement for 'scp' quite frequently. ---------------------------------------------------------------------- Sgt_ZigZag · 2023-12-15 16:17 FYI you can share Plex libraries natively in Plex. No file syncing needed. ---------------------------------------------------------------------- blusls · 2023-12-15 16:22 That is not the same. It just means they can watch what you have on your Plex server. We sync the files for off-site back up. ---------------------------------------------------------------------- PerkySloth · 2023-12-15 16:33 The most reliable thing I've ever used was and ftp/sftp server with the FlashFXP client(not free and not the best guy to support but he's a great dev). I had a shitty DSL connection back in the day and was getting constantly disconnected and that was the only client that never corrupted a file. I had to set up file hash checking with other ftp clients because they were all unreliable in some way. I've had good success with nextcloud these days with a few 8-10GB files, but that was with both people having stable connections. My suggestion would be that whatever you use, just make sure to create some kind of hash file so they can check if the transfer is 100% correct and make par2 files to fix them if it's not. ---------------------------------------------------------------------- thekrautboy · 2023-12-15 19:54 Sure, i would count that as a form of a download manager tho, some extra effort that the client has to put in. Not as basic as just "serve large file with webserver and have client use browser, done" as it was suggested. ---------------------------------------------------------------------- t1nk3rz · 2023-12-15 21:56 I use pingvin for large files,mine are 3-5Gb ---------------------------------------------------------------------- douglagm · 2023-12-16 11:36 If memory serves me correctly, it is php