[HN Gopher] BTFS: BitTorrent Filesystem
___________________________________________________________________
BTFS: BitTorrent Filesystem
Author : pyinstallwoes
Score : 349 points
Date : 2024-04-16 00:08 UTC (22 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| pyinstallwoes wrote:
| Submitting because I'm surprised why this isn't used more...
| couldn't we build a virtualmachine/OS's as an overlay on BTFS?
| Seems like an interesting direction.
| idle_zealot wrote:
| I'm not sure I see the point. A read-only filesystem that
| downloads files on-the-fly is neat, but doesn't sound practical
| in most situations.
| crest wrote:
| It can be an essential component, but for on-site replication
| you need to coordinate your caches to make the most of your
| available capacity. There're efforts to implement this on top
| IPFS to have mutually trusted nodes elect a leader deciding
| who should pin what to ensure you keep enough intact copies
| of everything in the distributed cache, but like so many
| things IPFS it started out interesting, died from feature
| creep and "visions" instead of working code.
| pyinstallwoes wrote:
| Imagine that any computation is a hash, then every possible
| thing becomes memoized not distinguishing between data/code.
| Then as a consequence you have durability, cache, security to
| an extent, verifiability through peers (could be trusted or
| degrees away from peers you trust).
| gameman144 wrote:
| Is every computation _worth_ memoizing? I can think of very
| few computations I do that others would care about, and in
| those cases there 's already a much more efficient caching
| layer fronting the data anyway.
| pyinstallwoes wrote:
| Why not? I think there is some interesting research here
| at the computationally level / distributed that could
| lead to some interesting architecture and discoveries.
|
| Fully distributed OS's/Virtual Machines/LLM's/Neural
| Networks
|
| If LLM's are token predictors for language, what happens
| when you do token prediction for computation across a
| distributed network? Then run a NN on the cache and
| clustering itself? Lots of potential use cases.
| cyanydeez wrote:
| If you were a billionaire and you wanted some software
| update, you could log into your super computer and have every
| shell mount the same torrent and it should be the fastest
| upload.
| retzkek wrote:
| CVMFS is a mature entry in that space, heavily used in the
| physics community to distribute software and container images,
| allowing simple and efficient sharing of computing resources.
| https://cernvm.cern.ch/fs/
| Jhsto wrote:
| Just the other week I used Nix on my laptop to derive PXE boot
| images, uploaded those to IPFS, and netbooted my server in
| another country over a public IPFS mirror. The initrd gets
| mounted as read-only overlayfs on boot. My configs are public:
| https://github.com/jhvst/nix-config
|
| I plan to write documentation of the IPFS process including the
| PXE router config later at https://github.com/majbacka-
| labs/nixos.fi -- we might also run a small public build server
| for peoples Flake configs, who are interested in trying out
| this process.
| __MatrixMan__ wrote:
| I laughed when I saw that your readme jumps straight into
| some category theory. FYI others might cry instead.
|
| You're doing some cool things here.
| jquaint wrote:
| This is really cool. Plan to take some inspiration from your
| config!
| vmfunction wrote:
| >A prominent direction in the Linux distribution scene lately
| has been the concept of immutable desktop operating systems.
| Recent examples include Fedora Silverblue (2018) and Vanilla
| OS (2022). But, on my anecdotal understanding of the
| timelines concerning Linux distros, both are spiritual
| successors to CoreOS (2013).
|
| Remember in the late 90's booting server off a CD-ROM was the
| thing.
| hnarn wrote:
| Booting off USB sticks is still done all the time and it's
| (almost) literally the same thing. Doing it in combination
| with encrypted persistence support as available in Debian
| for example can be really nice.
| infogulch wrote:
| It's not an overlay provider itself, but uber/kraken is a "P2P
| Docker registry capable of distributing TBs of data in
| seconds". It uses the bittorrent protocol to deliver docker
| images to large clusters.
|
| https://github.com/uber/kraken
| XorNot wrote:
| The problem with being a docker registry is that you're still
| having to double-dip: distribute to the registry, then docker
| pull.
|
| But you shouldn't need to: you should be able to do the same
| thing with a docker graph driver, so there is no registry -
| even daemon should perceive the local registry as "already
| available", even though in reality it's going to just
| download the parts it needs as it overlay mounts the image
| layers.
|
| Which would actually potentially save a ton of bandwidth,
| since the stuff _in_ an image is usually quite different to
| the stuff any given application needs (i.e. I usually base
| off Ubuntu, but if I 'm only throwing a Go binary in there
| plus wanting debugging tools maybe available, then in most
| executions the actual image pulled to the local disk would be
| very small).
| infogulch wrote:
| Maybe something like this?
| https://github.com/containers/fuse-overlayfs
| apichat wrote:
| https://github.com/uber/kraken?tab=readme-ov-
| file#comparison...
|
| "Kraken was initially built with a BitTorrent driver,
| however, we ended up implementing our P2P driver based on
| BitTorrent protocol to allow for tighter integration with
| storage solutions and more control over performance
| optimizations.
|
| Kraken's problem space is slightly different than what
| BitTorrent was designed for. Kraken's goal is to reduce
| global max download time and communication overhead in a
| stable environment, while BitTorrent was designed for an
| unpredictable and adversarial environment, so it needs to
| preserve more copies of scarce data and defend against
| malicious or bad behaving peers.
|
| Despite the differences, we re-examine Kraken's protocol from
| time to time, and if it's feasible, we hope to make it
| compatible with BitTorrent again."
| anacrolix wrote:
| That's exactly the problem my BitTorrent client solves.
| https://github.com/anacrolix/torrent. It's been around
| since 2013.
| phillebaba wrote:
| Kraken is sadly a dead project, with little work being done.
| For example support for Containerd is non-existent or just
| not documented.
|
| I created Spegel to fill the gap but focus on the P2P
| registry component without the overhead of running a stateful
| application. https://github.com/spegel-org/spegel
| thesuitonym wrote:
| Every once in a while, someone reinvents Plan 9 from Bell Labs.
| Solvency wrote:
| so is this like a Dropbox alternative using the bittorrent
| protocol?
| VyseofArcadia wrote:
| No,this is being able to interact with a .torrent file as if
| it's a directory.
|
| The only usecase I see for this is as an alternative to a more
| traditional bittorrent client.
| Solvency wrote:
| got it. but now i kind of want a torrent-based dropbox. i
| have five workstations. would be great to be able to utilize
| them as my own miniature distributed file system without a
| corporate server.
| Zambyte wrote:
| IPFS?
| ggop wrote:
| You could try Syncthing
| https://docs.syncthing.net/intro/getting-started.html
| jtriangle wrote:
| Syncthing experience is greatly improved if you also host
| your own discovery server, and if you can port forward.
|
| Pretty minor to do, but, it's a big speed increase.
| jszymborski wrote:
| Syncthing is great (I use it daily!) but I'm not sure it
| does the Dropbox/NextCloud thing that BTFS does where you
| can see remote files and download them on access.
| Syncthing rather just syncs folders as far as I can tell.
| vineyardmike wrote:
| There have been a bunch of projects that tried to do some
| variant this, and I'd love for it to exist, but I'd almost
| posit it's an impossible problem. Projects either find a
| way to handle the content-addressing, but then fail on
| coordinating nodes, or can coordinate but can't choose
| placement efficiently, or are just vaporware. I think the
| hard part is most personal computers are too unreliable to
| trust, and centralization, even for a homelab experimental
| user, is just too easy.
|
| A few projects that tackled some version of this...
|
| Nextcloud, Owncloud, and generally just NAS can be "dropbox
| but self hosted" but its centralized.
|
| IPFS, Perkeep, Iroh, hypercore (npm), focused on content-
| addressed information, making cataloging and sharing easy,
| but fail to really handle coordination of which node the
| data goes on.
|
| Syncthing, Garage, and of course BitTorrent, and a few
| others can coordinate but they all duplicate everything
| everywhere.
|
| "Bazil.org", and (dead link) "infinit.sh" both sought to
| coordinate distribution and somehow catalog the data, but
| they both seem to have died without achieving their goals.
| I used infinit.sh when it was alive ~2016 but it was too
| slow to use for anything.
|
| I'd love for something like this to exist, but I think its
| an impossible mission.
| Solvency wrote:
| ...wow, kind of fascinating. wish there was a post-mortem
| on failed attempts at this. I would not expect this, of
| all problems, to be unsolvable in 2024.
| kaetemi wrote:
| Resilio Sync (previously BitTorrent sync) is what you're
| looking for.
| baobun wrote:
| There are families of distributed filesystems. Most famous
| would be Ceph and GlusterFS and there are a many newer ones
| - maybe one of them would fit your use-case?
| zed716 wrote:
| There used to be something that worked exactly in this
| manner, called AeroFS -- There was a website portion that
| worked just like Dropbox did, and then a client you could
| install on your systems, and it would distribute items in a
| torrent-like manner between clients. It had a lot of neat
| features. It's a shame that it didn't end up really going
| anywhere (in a very crowded field at the time), because it
| worked great and they had an on-prem solution that worked
| really well.
| r14c wrote:
| tahoe-lafs might do what you need!
| sumtechguy wrote:
| This could be semi interesting for some torrents if it had
| CoW local store? So you could still write to it but keep your
| changes local and it still looks like a coherent directory.
| ChrisArchitect wrote:
| Thoughts from 4 years ago:
|
| https://news.ycombinator.com/item?id=23576063
| mdaniel wrote:
| the top comment <https://news.ycombinator.com/item?id=23580334>
| by saurik _(yes, that one)_ on the previous 121 comment thread
| back in 2020 sums up my feelings about the situation: BTFS is a
| "one CID at a time" version of IPFS
|
| I do think IPFS is awesome, but is going to take some major
| advances in at least 3 areas before it becomes something usable
| day-to-day:
|
| 1. not running a local node proxy (I hear that Brave has some
| built-in WebTorrent support, so maybe that's the path, but since
| I don't use Brave I can't say whether they are "WebTorrent in
| name only" or what
|
| 2. related to that, the swarm/peer resolution latency suffers in
| the same way that "web3 crypto tomfoolery" does, and that latency
| makes "browsing" feel like the old 14.4k modem days
|
| 3. IPFS is _absolutely fantastic_ for infrequently changing but
| super popular content, e.g. wikipedia, game releases, MDN
| content, etc, but is a super PITA to replace "tip" or "main" (if
| one thinks of browsing a git repo) with the "updated" version
| since (to the best of my knowledge) the only way folks have to
| resolve that newest CID is IPNS and DNS is never, ever going to
| be a "well, that's a good mechanism and surely doesn't contribute
| to one of the N things any outage always involves"
|
| I'm aware that I have spent an inordinate amount of words talking
| about a filesystem other than the one you submitted, but unlike
| BTFS, which I would never install, I think that those who click
| on this and are interested in the idea of BTFS may enjoy reading
| further into IPFS, but should bear in mind my opinion of its
| current shortcomings
| SuperNinKenDo wrote:
| Cool concept. I assume that it seeds if and while the files are
| present on your device? Tried to read the manpage but unformatted
| manpage markdown on a phone was too difficult to read.
| xmichael909 wrote:
| No security at all?
| mixmastamyk wrote:
| _Give her some funked-up muzak, she treats you nice_
|
| _Feed her some hungry reggae, she 'll love you twice_
| rakoo wrote:
| What security would be interesting here ?
| blueflow wrote:
| What would be something that needs to be protected against?
| stevefan1999 wrote:
| I found it might be quite useful for huge LLMs since those are
| now hosted on BitTorrent. Of course it is not going to be as
| practical as IPFS since IPFS is content addressable and easier to
| do random access.
| kimixa wrote:
| Is there much use for a partially resident LLM though?
|
| Then I don't see much of an advantage over just vanilla
| bittorrent - if you realistically need a full local copy to
| even start working anyway.
| Kerbonut wrote:
| I dream of having a BTFS that will fix my "damaged" media files.
| E.g. ones I media shift, if my disk was scratched and portions
| are missing, or if the codec options I picked suck, it could
| download the "damaged" portions of my media and fix it
| seamlessly.
| gosub100 wrote:
| another use of this is to share media after I've imported it
| into my library. if I voluntarily scan hashes of all my media,
| if a smart torrent client could offer those files only (so a
| partial torrent because I always remove the superfluous files)
| it would help seed a lot of rare media files.
| Stephen304 wrote:
| This happens to be one of the pipe dream roadmap milestones
| for bitmagnet: https://bitmagnet.io/#pipe-dream-features
|
| I used to use magnetico and wanted to make something that
| would use crawled info hashes to fetch the metadata and
| retrieve the file listing, then search a folder for any
| matching files. You'd probably want to pre-hash everything in
| the folder and cache the hashes.
|
| I hope bitmagnet gets that ability, it would be super cool
| jonhohle wrote:
| I've done a lot of archival of CD-ROM based games, and it's
| not clear to me this is possible without a lot of
| coordination and consistency (there are like 7 programs that
| use AccurateRip, )and those only deal with audio). I have
| found zero instances where a bin/cue I've downloaded online
| perfectly matches (hashes) to the same disc I've ripped
| locally. I've had some instances where different pressings if
| the same content hash differently.
|
| I've written tools to inspect content (say in an ISO file
| system), and those will hash to the same value (so different
| sector data but the same resulting file system). Audio
| converted to CDDA (16-bit PCM) will hash as well.
|
| If audio is transcoded into anything else, there's no way it
| would hash the same.
|
| At my last job I did something similar for build artifacts.
| You need the same compiler, same version, same settings, the
| ability to look inside the final artifact and avoid all the
| variable information (e.g. time). That requires a bit of
| domain specific information to get right.
| gosub100 wrote:
| Sorry I think you misunderstood me. I mean when I download
| a torrent called "Rachmaninov Complete Discography" I copy
| the files to the Music/Classical folder on my NAS. I can no
| longer seed the torrent unless I leave the original in the
| shared folder. But if I voluntarily let a crawler index and
| share my Music folder, it could see the hash of track1.flac
| and know that it associates with a particular file in the
| original torrent, thus allowing others to download it.
| pigpang wrote:
| How you will calculate hash of file, when it broken, to lookup
| for?
| 01HNNWZ0MV43FF wrote:
| You could do a rolling hash and say that a chunk with a given
| hash should appear between two other chunks of certain hashes
| arijun wrote:
| That seems like a recipe for nefarious code insertion.
| everfree wrote:
| Just hash it before it's broken.
| jonhohle wrote:
| Maybe this is a joke that's over my head, but the OP wants
| a system where damaged media can be repaired. They have the
| damaged media so there's no way to make a hash of the
| content they want.
| selcuka wrote:
| Just use the sector number(s) of the damaged parts.
| alex_duf wrote:
| if you store the merkle tree that was used to download it,
| you'll be able to know exactly which chunk of the file got a
| bit flip.
| rakoo wrote:
| You have all the hashes in the .torrent file. All you need is
| a regular check with it
|
| (but then the .torrent file itself has to be stored on a
| storage that resists bit flipping)
| arijun wrote:
| If you're worried about bit-flipping, you could just store
| multiple copies of the hash and then do voting, since it's
| small. If you're worried about correlated sources of error
| that helps less, though.
| Dibby053 wrote:
| >storage [...] bit flipping
|
| As someone with no storage expertise I'm curious, does
| anyone know the likelyhood of an error resulting in a bit
| flip rather than an unreadable sector? Memory bit flips
| during I/O are another thing but I'd expect a modern
| HDD/SSD to return an error if it isn't sure about what it's
| reading.
| halfcat wrote:
| Not sure if this is what you mean, but most HDD vendors
| publish reliability data like "Non-recoverable read
| errors per bits read":
|
| https://documents.westerndigital.com/content/dam/doc-
| library...
| Dibby053 wrote:
| Thanks for the link. I think that 10^14 figure is the
| likelyhood of the disk error correction failing to
| produce a valid result from the underlying media,
| returning a read error and adding the block to pending
| bad sectors. A typical read error that is caught by the
| OS and prompts the user to replace drives.
|
| What I understand by bit flip is a corruption that gets
| past that check (ie the "flips balance themselves" and
| produce a valid ECC) and returns bad data to the OS
| without producing any errors. Only a few filesystems that
| make their own checksums (like ZFS) would catch this
| failure mode.
|
| It's one reason I still use ZFS despite the downsides, so
| I wonder if I'm being too cautious about something that
| essentially can't happen.
| kelchm wrote:
| Not the same as what you are talking about, but your comment
| reminded me of AccurateRip [1] which I used to make extensive
| use of back when I was ripping hundreds of CDs every year.
|
| 1: http://www.accuraterip.com/
| neolithicum wrote:
| Do you have any tricks you can share on how to rip a large
| library of CDs? It would be nice to semi-automate the ripping
| process but I haven't found any tools to help with that. Also
| the MusicBrainz audio tagging library (the only open one I am
| aware of?) almost never has good tags for my CDs that don't
| have to be edited afterwards.
| mavhc wrote:
| https://github.com/whipper-team/whipper
|
| https://github.com/thomas-mc-work/most-possible-
| unattended-r...
|
| Finding a good CD drive to rip them is the first step.
|
| https://flemmingss.com/importing-data-from-discogs-and-
| other...
|
| IME Discogs had the track data most often.
|
| And obviously rip to flac
| neolithicum wrote:
| Great suggestions, I'll have to try these out. Thank you!
| bayindirh wrote:
| I was ripping my CD's with KDE's own KIO interface, which
| also does CDDB checks and embeds original information to
| ID3 tags. Passing through MusicBrainz Picard always gave me
| good tags, but I remember fine tuning it a bit.
|
| Now, I'll start another round with DBPowerAmp's ripper on
| macOS, then I'll see which tool brings the better metadata.
| jonhohle wrote:
| In the distant past iTunes was great at this (really).
| Insert a disc, its metadata is pulled in automatically,
| it's ripped and tagged using whatever coded settings you
| want and when it's done the disc is ejected.
|
| Watch a show do some other work and when the toast pops out
| a new one in.
|
| Ripping DVDs with HandBrake was almost as easy, but it
| wouldn't eject the disc afterwards (though it could have
| supported running a script at the end, I don't recall).
| cheap_headphone wrote:
| It really was. In the early 2000s I had a stack of Mac
| laptops doing exactly this. Made some decent cash
| advertising locally to rip people's CD collections!
| eddieroger wrote:
| This project still seems alive to my pleasant surprise.
|
| https://github.com/automatic-ripping-machine/automatic-
| rippi...
|
| I never had it fully working because the last time I tried,
| I was too focused on using VMs or Docker and not just
| dedicating a small, older computer to it, but I think about
| it often and may finally just take the time to set up a
| station to properly rip all the Columbia House CDs I bought
| when I was a teen and held on to.
| neolithicum wrote:
| Nice, I might install this on my Raspberry Pi.
| Fnoord wrote:
| Distribute parity files together with the real deal, like they
| do on Usenet? Usenet itself is pretty much this anyway. Not
| sure if the NNTP filesystem implementations work. Also, there's
| nzbfs [1]
|
| [1] https://github.com/danielfullmer/nzbfs
| drlemonpepper wrote:
| storj does this
| tarnith wrote:
| Why not run a filesystem that maintains this? (ZFS exists,
| storage is cheap)
| dang wrote:
| Related:
|
| _BTFS - mount any .torrent file or magnet link as directory_ -
| https://news.ycombinator.com/item?id=23576063 - June 2020 (121
| comments)
|
| _BitTorrent file system_ -
| https://news.ycombinator.com/item?id=10826154 - Jan 2016 (33
| comments)
| Cieric wrote:
| I've thought about using this for my media server in the past,
| but in the end I ran into to many issues trying to automate it.
| Then there's all the normal issues, slow downloads can wreck
| havoc on some programs expecting the whole thing to be there and
| I couldn't move files around without breaking connections. It was
| interesting to mess with, but in the end I just decided it would
| be a fun challenge to write my own in zig so I could have
| something "easy" to hack on in the future.
| Cieric wrote:
| I guess giving it some visibility wouldn't be the worst idea.
| https://github.com/ookami125/Zig-Torrent
| all2 wrote:
| Thank you. This answers the first question I have to folks
| who say "I made X that is related to this".
| skeledrew wrote:
| Pretty cool on the surface, but 2 things I can think of right now
| that mostly kills it for me (and maybe many others). 1) the
| things one usually use Bittorrent for tend to need the complete
| files to be all there to be useful, and internet speeds is a
| limiter in that regard. 2) seeder count tends to break down
| pretty quickly, as people delete or even just move the files
| elsewhere for whatever reason, so availability falls.
| elevenz wrote:
| A file index tool can be add to torrent clients, letting it
| scan all files in a user-selected folder and tag them with
| torrent info. Then how can it find the torrent info for a given
| file? Maybe we need a central server, maybe some hacker will
| invent a distributed reverse torrent search network, to list
| what playlists has a specific song. I don't know if someone has
| already invented this. I think it can be the final answer to
| any distributed file system driven by user uploads.
| WhiteOwlLion wrote:
| Aren't torrents block based so file boundaries are not
| observed without hacks?
| rakoo wrote:
| torrents are file-based but in v1 the edge of a file
| doesn't map with the edge of a piece, so you can't easily
| find file's hashes.
|
| In v2 this is solved and it is possible to easily know the
| hash of each file in the torrent, so you can search for it
| in other torrents
| skeledrew wrote:
| IPFS has resolved the indexing issue, with content
| addressing. Overall I'd say it has all the pros of bittorrent
| and fewer of the cons, from a technical perspective. However
| it's more complex, which may be a reason why it isn't more
| adopted.
| ceritium wrote:
| I did something similar some years ago,
| https://github.com/ceritium/fuse-torrent
|
| I had no idea what I was doing, most of the hard work IS done by
| the torrent-stream node package
| Maakuth wrote:
| This is the perfect client for accessing Internet Archive
| content! Each IA item automatically has a torrent that has IA's
| web seeds. Try Big Buck Bunny:
|
| btfs
| https://archive.org/download/BigBuckBunny_124/BigBuckBunny_1...
| mountpoint
| rnhmjoj wrote:
| Even better, try this: btplay https://archive
| .org/download/BigBuckBunny_124/BigBuckBunny_124_archive.torrent
| Maakuth wrote:
| What is that tool?
| palata wrote:
| I guess this:
| https://github.com/johang/btfs/blob/master/scripts/btplay
| haunter wrote:
| I don't know the internal workings of IA and the bittorent
| architecture but if an archive has too many items the torrent
| file won't have them all. I see this all the time with ROM
| packs and magazine archives for example. +1000 items, the
| torrent will only have the first ~200 or so available
| sumtechguy wrote:
| I think for some reason IA limits the torrent size. I have
| seen as low as 50 with a 1000+ item archive.
| sktrdie wrote:
| Or even better store data as an sqlite file that is full-text-
| search indexed. Then you can full-text search the torrent on
| demand: https://github.com/bittorrent/sqltorrent
| rwmj wrote:
| Here's a similar idea for block devices:
| https://libguestfs.org/nbdkit-torrent-plugin.1.html
|
| It lets you create a block device (/dev/nbd0) backed by a
| torrent. Blocks are fetched in the background, or on demand (by
| prioritizing blocks according to recent read requests).
|
| In practice it works - you can even boot a VM from it - but it's
| quite slow unless you have lots of seeds. There's a danger,
| particularly with VMs, that you can hit time outs waiting for a
| block to be read, unless you adjust some guest kernel parameters.
|
| There are some bootable examples in that page if you want to try
| it.
| apichat wrote:
| This tool should be upgrade to use Bittorrent v2 new functions.
|
| https://blog.libtorrent.org/2020/09/bittorrent-v2/
|
| Especially merkle hash trees which enable :
|
| - per-file hash trees - directory structure
| bardan wrote:
| somebody check the test functions...
| logrot wrote:
| Especially the corrupt torrent test file.
| bArray wrote:
| Couldn't help but think it would be epic if it went the other way
| - you throw files into a folder, it's set as read-only and then
| share that as a torrent.
| throwme_123 wrote:
| Someone should integrate:
|
| 1. https://github.com/ipfs/kubo/blob/master/docs/fuse.md
|
| 2. https://ipfscluster.io/
| bArray wrote:
| I'm not sure it's the same, IPFS would be another (valuable)
| thing entirely. I would be tempted to say that the torrent
| filesystem should be kept strictly in the scope of torrents.
| anacrolix wrote:
| https://github.com/anacrolix/torrent has a fuse driver since
| 2013. I'm in the early stages of removing it. There are WebDAV,
| 3rd party FUSE, and HTTP wrappers of the client all doing similar
| things: serving magnet links, infohashes, and torrent files like
| an immutable filesystem. BitTorrent v2 support is currently in
| master.
| raffraffraff wrote:
| So is there a server program to partner this with? Something that
| acts as a torrent file builder, tracker and simple file server
| for the torrent? I can imagine in a large org you could store a
| gigantic quantity of public data on a server that creates a
| torrent whenever the data changes, serves the.torrent file over
| http and also acts as a tracker. You could wrap the FUSE client
| in some basic logic that detects newer torrents on the server and
| reloads/remounts.
|
| Many moons ago I created a Linux distribution for a bank. It was
| based on Ubuntu NetBoot with minimal packages for their branch
| desktop. As the branches were serverless, the distro was self-
| seeding. You could walk into a building with one of them and use
| it to build hundreds of clones in a pretty short time. All you
| needed was wake-on-lan and PXE configured on the switches. The
| new clones could also act as seeds. Under the hood it served a
| custom Ubuntu repo on nginx and ran tftp/inetd and wackamole
| (which used libspread, neither have been maintained for years).
| Once a machine got built, it pulled a torrent off the "server"
| and added it to transmission. Once that was completed the machine
| could also act as a seed, so it would start up wackamole, inetd,
| nginx, tracker etc. At first you seed 10 machines reliably, but
| once they were all up, you could wake machines in greater
| numbers. Across hundreds of bank branches I deployed the image
| onto 8000 machines in a few weeks (most of the delays due to
| change control and staged rollout plan). Actually the hardest
| part was getting the first seed downloaded to the branches via
| their old Linux build, and using one of them to act as a seed
| machine. That was done in 350+ branches, over inadequate network
| connections (some were 256kbps ISDN)
| mdaniel wrote:
| This may interest you, although as far as I know only AWS's S3
| implements it:
| https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjec...
|
| I actually have never used it in order to know if AWS puts
| their own trackers in the resulting .torrent or what
| raffraffraff wrote:
| hmmm, pretty cool. Of course you have to pay the egress tax,
| but still...
___________________________________________________________________
(page generated 2024-04-16 23:01 UTC)