[HN Gopher] Ask HN: Best modern file transfer/synchronization pr...
___________________________________________________________________
Ask HN: Best modern file transfer/synchronization protocol?
Hi HN, I'm looking to build something to transfer files between
two devices on a network (one client and one server, both of which
are under my control). Obviously I could write something bespoke
with raw TCP sockets, but rather than reinventing the wheel I'm
curious about what existing options people recommend. I assume
there are some better technologies than FTP nowadays? Ideally some
kind of built-in fault tolerance would be great, because my plan is
to use this on a phone/in an environment where the connection could
be interrupted. Edit: just to clarify - this is something I want
to build into an application I am writing, ideally with support
across iOS (client), Windows, and mac (server). One way transfer
is all I need, and I mostly plan on photos/videos (so multiple
files ~3-20MB in size). Thanks!
Author : daveidol
Score : 54 points
Date : 2023-12-22 17:46 UTC (5 hours ago)
| jayknight wrote:
| I would probably start with rsync.
| daveidol wrote:
| Thanks - is there a reference implementation for rsync that
| works on windows and iOS? My impression was it was more of a
| CLI tool for Linux/macOS.
| necovek wrote:
| If you are looking to integrate into another app, maybe check
| https://librsync.github.io/ out.
| daveguy wrote:
| Unfortunately one of the caveats about what librsync is not
| (from the link):
|
| librsync also does not include any network functions for
| talking to SSH or any other server. To access a remote
| filesystem, you need to provide your own code or make use
| of some other virtual filesystem layer.
|
| Having this seems to be one of the primary requirements.
| teddyh wrote:
| For Windows:
| <https://community.chocolatey.org/packages/rsync>
| sneak wrote:
| rsync over ssh for one-shots.
|
| syncthing for continuous use.
| thedaly wrote:
| I've starting using rclone over rsync for this application.
| Rclone can do segmented transfers and handles large numbers of
| files better, at least in my experience.
| sneak wrote:
| What are segmented transfers?
| daveguy wrote:
| Send multiple pieces of the file and reconstruct the file
| on the other end. It's more reliable to send smaller chunks
| and the transfer can be hash validated on individual chunks
| and after reconstruction. Multiple download streams
| typically performs better too (from single or multiple
| servers).
|
| e.g. BitTorrent for the multiple servers case.
| xnx wrote:
| rclone is great! So useful as an alternative/superior client
| for cloud storage too (Google Drive, OneDrive, etc.)
| bhaney wrote:
| Yeah I usually just use rsync for this. In a loop if the network
| is unreliable.
| whalesalad wrote:
| seconding rsync and syncthing.
|
| the server could expose an smb or nfs share, the client could
| mount it, and then sync to that mount.
|
| rsync over ssh also works, if you do not want to run smb/nfs.
|
| this is also a cool tool https://rclone.org/
| ryukoposting wrote:
| I use syncthing for this. It's a little fiddly to set up, and
| transfer speeds aren't great. However, it's very reliable once
| configured, and it barely uses any resources after an initial
| scan of the folder you want to sync.
| BrandoElFollito wrote:
| I ended up with Syncthing after trying everything else (that O
| can self-host).
|
| You can ease the eureka moment by remembering that each node is
| completely independent and decided what comes in, and suggests
| what goes out.
|
| You can say that a folder is two way on a node and read only on
| another - and this is great because tout make decisions locally
| that allow you to build nice things.
|
| The real issue is that you have to check on both sides of the
| pipe what is allowed.
|
| It is very robust and good for LAN and remote sync (the traffic
| is outgoing to a relay server. Note that many of these relay
| servers are also TOR nudes so they can be flagged by your
| severity systems. You can always use your own)
| orbz wrote:
| No need to get fancy, scp or rsync are the tried and true options
| here.
| brudgers wrote:
| Or on Windows, Xcopy.
| PopAlongKid wrote:
| Or robocopy.
| krylon wrote:
| robocopy is nice.
| toomim wrote:
| Use HTTP. For fault tolerance, use resumeable downloads or
| resumeable uploads. There is work at the IETF on resumeable
| uploads right now: https://datatracker.ietf.org/doc/draft-ietf-
| httpbis-resumabl...
| Flimm wrote:
| I've had terrible experiences downloading large files over
| HTTP. I'm not sure why, range requests don't seem to be
| reliable or well supported. Something like BitTorrent is much
| better for large files: it divides the large file into chunks
| and hashes each chunk, and by default the chunks are downloaded
| in random order. BitTorrent seems much more reliable than range
| requests.
| otherme123 wrote:
| Same here. I downloaded some ~100Gb files, and everytime the
| connection broke, after relaunching, the shasum never
| matched. Even when the file size was exactly the same that
| the server reported.
| mynegation wrote:
| What are your latency and bandwidth requirements? How big are the
| files? If you are already looking past obvious TCP-based choices
| like HTTP and FTP, you might be interested in FASP/Aspera
| https://en.m.wikipedia.org/wiki/Fast_and_Secure_Protocol
|
| Edit: I'll leave it here just in case it is useful for others but
| it may or may not be embeddable into your app, especially on the
| phone.
| oschrenk wrote:
| If it's one way (that wasn't quite clear from the requirements to
| me).
|
| take a look at https://tus.io/
| daveidol wrote:
| This is awesome! I am indeed looking for something one way - so
| this looks great. Thanks for sharing.
| rmorey wrote:
| I cannot recommend rclone enough. Been using it to transfer
| petabyte-scale datasets flawlessly. Available as librclone as
| well
| tetris11 wrote:
| it even has a decent Android client in F-droid, RCX. I just
| wish it supported SSHFS
| gazby wrote:
| RCX is approaching abandonware status (last release 2y ago).
| Round Sync is worth a look.
| tetris11 wrote:
| Thanks for the recommendation! I searched F-droid but found
| nothing, and then I realised it was probably.on Izzy
| ksjskskskkk wrote:
| rclone is garbage if you need performance or just copy between
| two points you control.
|
| it shines when you need to sprinkle your data over many
| "clouds"
| esafak wrote:
| So what do you use?
| gazby wrote:
| I'd argue the opposite. Rclone's parallelization options are
| unmatched.
| mongol wrote:
| rclone is no protocol though
| rrix2 wrote:
| I built something on top of the Syncthing API this week after
| using it on its own for years.
|
| A local instance of Syncthing can behave as a robust sync tool +
| inotify API for applications consuming the files:
| https://docs.syncthing.net/rest/events-get.html#get-rest-eve...
|
| i believe there's an embeddable golang library, but if you want
| something easy to use on android check in on syncthing-fork which
| lets you define more granular sync conditions including "just
| turn on 5 minutes every hour"
| https://github.com/Catfriend1/syncthing-android
| pluto_modadic wrote:
| continuous sync - mutagen.io (maybe you could extract some of the
| libraries)
|
| depends on if it's large or small files.
| pvtmert wrote:
| I highly recommend Unison (https://github.com/bcpierce00/unison)
|
| It allows you to sync between 2 machines (bi-directional) over
| TCP or SSH.
|
| Note that TCP way is not encrypted, you may use wireguard as
| transport layer encryption for that purpose...
|
| You can use an external application to copy if file size is
| larger than an arbitrary number. (Eg: use rsync for files > 1gb)
| q0uaur wrote:
| another vote for Unison, been using it for about a year for
| practically everything and it's great. takes a moment to grasp
| the concept, but it's 100% worth it.
|
| that said, i haven't managed to set it up for my android phone
| yet - it's not available in termux and i have NO idea where to
| start if i'd like to package it myself. probably has to be done
| by someone who knows ocaml, since termux' environment is so
| different from normal linux.
|
| unless... maybe i should give proot a shot (chroot in termux,
| lets you run something closer to linux). but it's another layer
| of complexity on top of everything....
| yetanother12345 wrote:
| eh... wget or curl ? or not modern enough for you?
| smackeyacky wrote:
| Grab an S3 bucket on amazon.
|
| Do a 3 way sync with the s3 command line tool.
|
| That way, you have a neat cloud backup as well. Wouldn't take any
| more than 20 minutes total to set up.
___________________________________________________________________
(page generated 2023-12-22 23:02 UTC)