[HN Gopher] Syncthing: A continuous file synchronization program
___________________________________________________________________
Syncthing: A continuous file synchronization program
Author : hello_computer
Score : 179 points
Date : 2023-05-09 19:12 UTC (3 hours ago)
(HTM) web link (syncthing.net)
(TXT) w3m dump (syncthing.net)
| alexbezhan wrote:
| Can I use it to backup SQLite? Like an alternative to Litestream?
| Are there any negative implications for doing this?
| alexbezhan wrote:
| Found relevant discussion here
| https://forum.syncthing.net/t/bahaviour-of-st-on-sqlite-file...
| benbjohnson wrote:
| Litestream author here. I haven't personally tried Syncthing on
| SQLite databases but I suspect it wouldn't work well. Syncthing
| doesn't have any concept of SQLite's locks so even in the best
| case scenario where you only have one writer and the replica is
| read-only, the reader on the replica would have pages changed
| out from under it by Syncthing and that would appear corrupt to
| the reader.
|
| One case where you could make it work is if you sync'd across
| multiple computers but only used one at a time. I mean, it's
| still dangerous but it seems possible.
| alexbezhan wrote:
| What if I do sqlite backup with cron and then sync the
| backup?
| benbjohnson wrote:
| If you're just using the sync'd version as read-only or as
| a backup then I suspect that's fine. It seems kinda
| complicated but you should be ok. :)
| system33- wrote:
| Syncthing "just" syncs files. Near instantly, which could mean
| a delay of a few seconds to maybe even a minute. And it won't
| de-conflict changes beyond saving the "bad" version in a file
| like `sqlite.db-syncthing-conflict-20230101T043044`. (Not an
| issue if you only have one host changing the database file)
|
| I like it and IMO it does what it does well. But litestream can
| "Continuously stream SQLite changes" (pulled from
| https://litestream.io/, I've never heard of it before). It
| seems like that's litestream's killer feature, and I think
| syncthing would leave you wanting.
| qbasic_forever wrote:
| Not really, it can't resolve a conflict inside the file and
| can't enforce a distributed lock on changing the file.
| Ultimately there will be ways for the file to corrupt or get
| out of sync, especially if multiple writers are present. There
| might be a way to make things work better with SQLite's more
| recent WAL mode but I'm not familiar with it.
| aynyc wrote:
| https://www.sqlite.org/howtocorrupt.html
|
| If not careful, I think you'll hit, particularly 1.2, 1.4.
| tech234a wrote:
| I use it and it works well, unfortunately there isn't a free
| client for iOS though.
| mteam88 wrote:
| Been using for a while with my keepass database. Almost never
| noticed it.
| barbs wrote:
| I remember reading somewhere that the best software does what
| it needs to do and then gets out of your way...and it was
| referring specifically to Syncthing!
| ceeam wrote:
| Using resilio (which is NOT open source, of course and is
| actually kinda abandoned) until they implement selective sync
| (also "cloud" files browser). This is THE feature that is the
| deal breaker for me to switch. No, I don't sync hundreds of
| gigabytes from my personal cloud to my fokken phone, or even
| laptop, or even work PC. Still want access to all of them on
| demand.
| ceeam wrote:
| And - yes - a program like Resilio/Syncthing is the paradigm
| shift to how you use your computers, if you're a serious user
| and have more than one machine (incl. mobile devices).
| vorpalhex wrote:
| My workaround with syncthing is to have several smaller folders
| that I only add to devices as I want them, and a common folder
| across all devices. It works well enough.
| tener wrote:
| You do have ignore lists, but the UX is extra bad. OTOH resilio
| ate my files in certain situations, so I had to ditch it.
| zapdrive wrote:
| While syncthing is excellent and I use it backup photos from my
| phone to my private cloud, it can sometimes glitch and delete
| files. I haven't got to the root cause yet, so I can't tell you
| what to do, but there's a way to recover your data in case of
| accidental deletion by syncthing. Make sure you enable "file
| versioning" on both your clients and server. I use the "trashcan"
| option. So in case a file is deleted, syncthing saves it in a
| folder for a pre-selected number of days instead of deleting it.
| karlicoss wrote:
| FWIW, never happened to me with Synching, but either way it's
| best to separate sync software and backup software. Sync might
| work flawlessly, but the user might make a mistake which would
| quickly wipe the files across all devices. I use borg for
| backups, highly recommend
| kyrofa wrote:
| Yikes. Nextcloud used to do that to me. I'd love to find an
| alternative, but I don't want to go back to those dark days.
| charleshan wrote:
| I've been using it for a couple of years. One problem I have is
| that the idle cpu usage goes up to 10-20% after a while on my
| Mac. Drops back to ~1% after restarting the app
| dariosalvi78 wrote:
| I use it for uploading pictures from my phone to my "private
| cloud" (an old RaspberryPi) and it works like a charm. I then
| move the uploaded pictures into organized folders, which also
| makes it delete the pictures from my phone, thus saving space.
| gsich wrote:
| Have they fixed the SD card issue?
| zqna wrote:
| Used syncthing to: - transfer executables from my home to a ship
| (over sattelite link) in Indian Ocean. Much better alternative
| than sftp. - sync filesystems while doing on-going (several
| months of testing) migration from on-prem to mixed cloud
| (google/azure). Requires a bit of understanding why conflicts can
| happen (when using more than 2 hosts), once understood works like
| a charm. Was never disappointed, solid work.
| Joeri wrote:
| Syncthing is excellent. I've used it for years to keep my
| development folders synced between mac, linux and windows, and
| except for some initial figuring out of ignore patterns and the
| occasional hiccup with filenames that windows can't handle it has
| been great at doing what it needs to and getting out of my way.
|
| (I of course also use git, but syncthing lets me choose when I
| push while still having all my work on all my machines.)
| thefz wrote:
| > occasional hiccup with filenames that windows can't handle
|
| My only complaint so far and I've been using it for years. At
| first it started as a way to have my Keepass file in sync
| between computers, and evolved from that.
| raffraffraff wrote:
| I do this too! Feels weird keeping code in sync between
| machines aside from git, but it works really well.
| greenie_beans wrote:
| i always heard it fucks up with git repos?
| mike-cardwell wrote:
| I use it to store most of my private git repos and have been
| doing so for years now without a single issue. I do this:
| cd ~/Sync/git mkdir reponame cd reponame
| git init --bare cd ~/work git clone
| ~/Sync/git/reponame
|
| I guess this could break if I do a push and then shut my
| laptop lid before it syncs, and then do some more work on my
| desktop, push, and then go back to my laptop. That hasn't
| happened though. If it did, it should be trivial to fix.
|
| There was some other comment near here about ignoring .git
| directories. I guess they're using it to store the working
| directory rather than just the repository, as there is no
| .git directory stored in ~/Sync/ in my situation.
| sveinugu wrote:
| [dead]
| generalizations wrote:
| I have an entire folder dedicated to various git repos...only
| thing it doesn't like (so far...) is some compilation
| artifacts. IDK why.
| prophesi wrote:
| I would make sure that there's a `.stignore` file that's
| identical to your `.gitignore` (except for secret
| keys/configs you'd like to sync on your devices) in each of
| those repositories.
| karlicoss wrote:
| not sure what happens if you commit on two devices before
| syncing, but the "worst" that happened to me is I get an
| index conflict, in which case it's easily fixed by 'git
| reset'
| jbverschoor wrote:
| While SyncThjng might be encrypted, it's yet another port or
| server that's open.
|
| We need ubiquitous private networks, such as wire guard,
| zerotier, or the buggy back-to-my-mac.
| bowel wrote:
| You can always turn off NAT, LAN announcement, Relays and STUN
| -- and run SyncThing behind WireGuard. I use Tailscale and it
| works like a charm.
| thefz wrote:
| You can also run your own relay and only allow that if you
| want!
| 0xbadcafebee wrote:
| Private networks are old and busted; zero-trust & E2EE is the
| new hotness. (well, if 14 years is considered new)
|
| The idea of a network as a delineation of security or trust is
| inherently flawed and should be considered an anti-pattern.
| Reliance on "private" networks merely moves attacks to the
| network level, making a very large target and exposing more at
| once. Zero trust is where the whole world should be moving:
| explicit authorization added at a fine-grained level to each
| thing each user wants to access. But people haven't been
| exposed to this idea yet in their day to day lives, so they
| can't let go of the old paradigms of firewalls and networks.
| mholt wrote:
| I keep wanting Tailscale to offer a "tailnet API" whereby you
| can bundle a tailnet into your app, and users can connect to
| other instances of the app (whether their same account or other
| people sharing parts of their app/data) seamlessly: E2E
| encrypted, NAT busting, it just works.
|
| Think about it: mobile and desktop apps no longer have to
| upload things to some third-party service to share with
| yourself or a group of people. You could set up your own server
| of course but that requires technical expertise. The mass
| market could share data, sync, and communicate, straight from
| their devices, without any service hosting their data. It's
| just all on the private network.
|
| There's probably a billion-dollar market there (or more) if
| someone can figure that out for a lay audience: virtually
| unlimited, encrypted private networks on-demand that require no
| setup.
| gnur wrote:
| Pretty sure they offer that already,
| https://tailscale.com/blog/golink/ has more information on
| embedding the tailnet within a program.
| mholt wrote:
| It can be embedded in a program, but it doesn't
| automatically manage devices on a tailnet; i.e. there's no
| way to add all users of an app to the same tailnet (they
| have to explicitly create a Tailscale account and sign in
| and share and join) - AFAIK.
| dmytrish wrote:
| Isn't this litailscale?
| barbs wrote:
| Not sure if this is exactly what you're describing but I
| noticed the open-source Diablo engine DevilitionX has
| Zerotier built in to allow online multiplayer sessions.
|
| https://github.com/diasurgical/devilutionX/wiki/Multiplayer#.
| ..
|
| I thought it was interesting way for an open-source game to
| provide a way to find other players without relying on paid
| dedicated servers.
| mholt wrote:
| Ah yeah, that's pretty cool. Something like that!!
| reaperducer wrote:
| _or the buggy back-to-my-mac._
|
| Back to My Mac was awesome. When it worked.
|
| I had about an 80% success rate with it, dragging my MacBook
| Air to about 30 countries around the world. It was tremendously
| useful at a time when storage was far more expensive and
| internal drives much smaller than they are today.
|
| Often when it didn't work, I could just leave the computer
| running in the hotel while I went out and did touristy stuff,
| and by the time I came back things had sorted themselves out.
| rakoo wrote:
| Private networks are one part of the solution. Syncthing
| synchronizes, and for anyone who has ever tried it, it's not
| just two rsync's back and forth, whether ports are open or not
| Avamander wrote:
| You can disable discovery and hardcode clients to connect to
| each other using hostnames that resolve to whatever private IPs
| you have.
| qbasic_forever wrote:
| If you use wireguard or something similar that gives every
| client their own private and encrypted IP then you don't really
| need syncthing. You can just use rclone or even rsync commands
| that point directly at the appropriate client IP in the mesh
| network. Syncthing is still useful though for its continuous
| syncing support and overall UI.
| thewataccount wrote:
| If you had 10's thousands or millions of files, and several
| terabytes, is syncthing a good option for this?
|
| I haven't seen a ton of great options unfortunately :/
| karlicoss wrote:
| I sync about two hundred thousands of files without any
| problems, especially if your aren't changing them all the time.
| The only issue I can imagine is initial sync with so many files
| might take a while, even if total size isn't huge. As for
| several terabytes, for me it's a bit less than a terabyte in
| total for all synced folders, but also can't see why it
| wouldn't scale up. I guess similar, initial hashing might take
| some time, but otherwise it should handle it well.
| em-bee wrote:
| that is a hard problem. it depends on how frequently the data
| changes, because syncthing or any other backup solution may be
| to slow to keep up. one of my clients eventually switched from
| rsync to gluster. instead of making backups, gluster writes
| multiple storage servers in parallel when files are changed,
| thus ensuring that there are multiple copies right away.
|
| it's not the perfect solution though because when one server is
| offline for a while and needs to catch up that too is very
| slow. the only solution there i find is to have enough
| redundant servers that you can afford to have a server go
| offline and not worry about getting it back online quickly.
| vorpalhex wrote:
| I've been successful with syncthing up to 2tb (never tried
| higher) with the caveat that I only have it function between
| two hosts and only one host is the writer.
| keb_ wrote:
| Been using Syncthing for a year or so to sync plaintext notes
| across my phone and PCs. Worked great, so then I began using it
| to sync Retroarch save files between a spare Pixel 2 I use for
| gaming and my PC. Worked even better! I could play games on my
| PC, and then continue on the go.
|
| Recently, I switched completely off cloud storage to just using
| Syncthing and I love it.
|
| Anyway, this is great software.
| aborsy wrote:
| Best sync software in my view. The only downside is that there is
| no good iOS app.
|
| I think the developers made the right decision focusing the
| software on peer to peer sync (i.e., not expanding to nearby
| "cloud features": backup, photos, videos, office, etc).
| riceart wrote:
| What's wrong with mobius sync?
| aborsy wrote:
| Not available in my region!
| tech234a wrote:
| https://github.com/MobiusSync/MobiusSync/issues/27
| generalizations wrote:
| It doesn't work in the background. But that's not the app's
| fault, it's just an iOS restriction.
| Reptur wrote:
| Yea, I ended up using an IOS SSH app to a server and interact
| with the synced folder. Not perfect, but works.
| ubermonkey wrote:
| "The only downside is that there is no good iOS app."
|
| That's a dealbreaker for iOS folks, though.
|
| I hate the mission creep Dropbox has embraced, and would
| definitely consider something else, but I need access to my
| stuff on Windows, MacOS, and iOS. Anything that doesn't hit all
| three notes isn't a contender.
| hparadiz wrote:
| To me it's a dealbreaker for iOS. iPhones are just toys that
| don't get real work done.
|
| I take a photo with with my phone and it's synced to my Linux
| desktop in 1-2 seconds.
| lost_tourist wrote:
| Strange flex on ios users lol.
| princevegeta89 wrote:
| Is there a way to leverage this software and deploy something
| like a self hosted Google drive that also allows us to generate
| public links for sharing ?
| kornhole wrote:
| Try Nextcloud, Seafile or another server installation.
| [deleted]
| sneak wrote:
| It also phones home by default unless you turn it off (via
| crash and usage reporting), and in the default vendor-provided
| state grants arbitrary remote code execution ability to the
| developer (via unattended auto update which is on by default).
| Fortunately most distro packages and docker containers disable
| this dangerous behavior
| tredre3 wrote:
| Even after disabling crash reporting, auto-update, relays,
| discovery, etc, it will still connect to the internet. It's
| actually very difficult to make it behave.
|
| The only solution is to blank a few URLs in the configuration
| for things that can't be disabled. Then at first sight it
| seems to work... But you check your firewall logs a few days
| later and what do you know? Some dropped connection attempts.
|
| Still, there aren't many alternatives that work as well as
| SyncThing, so you gotta pick your battles...
| celsoazevedo wrote:
| > It also phones home by default unless you turn it off (via
| crash and usage reporting)
|
| Is this new behaviour? I haven't set up a new device
| recently, but it always asked me if I want to enable that or
| not.
| sneak wrote:
| Usage reporting. Crash reporting is on by default without
| asking, as is use of the public relays.
|
| The arbitrary remote code execution vulnerability is much
| more troubling and critical, however. Most people don't
| realize that by running syncthing they grant the syncthing
| developer the ability to execute any code they want on
| their machine (including the ability to read or exfiltrate
| any file on the filesystem).
| JeremyNT wrote:
| > _in the default vendor-provided state grants arbitrary
| remote code execution ability to the developer (via
| unattended auto update which is on by default)_
|
| I find this a curious, borderline alarmist framing of the
| automatic update mechanism. Many pieces of software have
| automatic updates. What you say is no less true for common
| programs like Firefox or Google Chrome.
|
| Obviously you have to consider your threat model, but the
| number of people willing to trust a specific Syncthing binary
| they're downloading from the internet, but who are wary of
| trusting published updates going forward, must be relatively
| small.
|
| (I am a Linux user and use the OS packages rather than
| automatic updates, but automatic updates are quite common
| outside of Linux distro packages)
| berkut wrote:
| It also connects to many "Discovery" servers on the internet
| in its default configuration, which you don't need enabled if
| you're just syncing between machines on an internal network.
| em-bee wrote:
| which tool besides syncthing can do automatic backups of my
| phone? by automatic i mean copy photos and other data to a
| second device of my choosing as soon as they are created.
|
| the primary use for syncthing on my phone is to make sure data
| is backed up immediately.
|
| i have yet to find an app that has the resilience and
| flexibility of syncthing for this purpose.
|
| apart from that the primary issue i have with syncthing is that
| the app on a device can not tell me if all data in a particular
| folder is fully copied to the other device. it only tells me if
| all local folders are complete.
|
| it also gets confusing when you have more than a few devices
| and sync between all of them. i recently discovered that i had
| a set up A<->B<->C but A and C were not connected. i never
| noticed because A and C were always in sync, until device B
| died. but it took me some time to notice that because i didn't
| realize that i missed to connect A to C directly.
|
| it would be nice to have some way to get an overview of the
| whole graph to see what syncs to where.
| kornhole wrote:
| >automatic backups of my phone? Full Android phone backup to
| a drive can be done with Seedvault on a schedule. I think you
| are asking about automatic upload though. I use Nextcloud to
| auto upload new pictures to my server as well as for
| Seedvault backup.
| em-bee wrote:
| yes, automatic upload. schedules are impractical for phones
| that are not connected to wifi on a schedule.
|
| i also occasionally use the phone to take photos of items
| that i want to process on my laptop right away. syncthing
| is faster and more convenient than taking my camera and
| then moving the sd card from the camera to the laptop.
|
| for nextcloud i would need to set up a server that probably
| also needs to be always online. syncthing allows me to
| backup to my laptop and is opportunistic about it, ie
| whenever both are on wifi.
|
| i am also using syncthing to manage my audio files. on the
| laptop i decide what i want to listen to by copying the
| files into a specific folder, and that gets copied to the
| phone. when i am done with a file i delete it from the
| phone, and that gets synced back to the laptop. syncthing
| is perfect here, and so using it for backup in the other
| cases is worth it over having yet another solution.
| ajd1988 wrote:
| I couldn't agree more!
|
| Syncthing is like the reliable, quiet roommate who always does
| their dishes, whereas dealing with different file systems feels
| like trying to figure out who left their half-eaten pizza on
| the couch.
|
| Just needs the iOS app!
| celsoazevedo wrote:
| There's Mobius Sync, but iOS restrictions means that it can't
| work in the background or access folders like it does on
| Android/Windows/macOS/Linux.
| est31 wrote:
| iOS literally doesn't have "just there for storage"
| directories that are not icloud. Either you use the photos
| pool, or you use icloud, or you use per-app directories
| that are local to each app. But then those directories
| cannot be accessed by other apps. Even if access to
| individual files is implemented, it's often very buggy.
|
| Say you have a pictures directory shared via Mobius Sync.
| Then you want to edit one of the pictures with a pictures
| editing app that is not the builtin photos app. The app
| crashes, or cannot save an edited copy of the file into the
| same directory. I encountered this in real life.
| btobolaski wrote:
| Working copy has some of doing this. I had it syncing my
| logseq notes and it worked fine (other than merge
| conflict type things). The setup was fairly convoluted
| though.
|
| This is how it is done
| https://github.com/CharlesChiuGit/Logseq-Git-
| Sync-101/wiki/F...
___________________________________________________________________
(page generated 2023-05-09 23:01 UTC)