[HN Gopher] Bitmagnet Allows People to Run Their Own Decentraliz...
___________________________________________________________________
Bitmagnet Allows People to Run Their Own Decentralized Torrent
Indexer Locally
Author : mgdigital
Score : 107 points
Date : 2024-02-18 17:05 UTC (5 hours ago)
(HTM) web link (torrentfreak.com)
(TXT) w3m dump (torrentfreak.com)
| jMyles wrote:
| Interesting.
|
| I want to seamlessly distribute my music through bittorrent, but
| since I have a small fan base (and thus, a very small potential
| seeding pool), I found it difficult to connect all the moving
| parts.
|
| I'll give Bitmagnet a try for an Indexer.
|
| I did find that IPFS, with a pinning service (I won't shill the
| one I used in particular), was a bit easier and just worked for
| everyone who tried to use it.
|
| But I'd like to get my bittorrent presence up to the "just works"
| level also.
| 8organicbits wrote:
| Have you considered setting up a web seed?
|
| https://www.ubuntubuzz.com/2021/06/how-to-add-web-seeds-to-t...
| jMyles wrote:
| No I hadn't!
|
| This seems exciting. Have you done it successfully?
| KoftaBob wrote:
| Well for your use case of distributing your own music,
| Bitmagnet wouldn't be necessary.
|
| What a DHT crawler like Bitmagnet does is the following:
|
| 1. Take a few initial "bootstrap" torrents and ping them to see
| which IP addresses are seeding that file 2. Ping those IP
| addresses and ask what other files/torrents they're seeding 3.
| Ping those torrents to see which IP addresses are seeding that
| torrent
|
| Rinse repeat.
|
| So to distribute your music to fans, you'd just want to put
| magnet links to your music on your site.
| gkbrk wrote:
| You don't need to ping anyone to crawl the DHT. You can
| passively wait and you'll get DHT queries in the form of "I'm
| looking for people seeding XYZ. Do you have a list?". You can
| just save those somewhere and you'll accumulate a list of
| active and new torrents.
|
| Writing a DHT crawler is super fun, I suggest everyone to get
| a cheap VM and write/run one.
| pluto_modadic wrote:
| that'd be the slow mode, though
| jMyles wrote:
| It's kinda like the owl with the tootsie pop:
|
| How many cups of coffee does it take to internalize hamming
| distance? :-)
| the8472 wrote:
| That's not correct. Torrent swarms and the DHT are separate.
| Each torrent basically forms its own small network of TCP
| connections to exchange data specific to that one torrent.
| While the DHT is a network shared by all clients that speak
| the protocol and it's carried over short-lived UDP query-
| response exchanges.
| pessimizer wrote:
| You have to be participating in a torrent swarm in order to
| bootstrap yourself into DHT at all. Bittorrent's DHT is not
| a network independent of torrent swarms. You need the
| address of a peer who is already part of the network in
| order to join it, and you have to get a list those
| addresses from somewhere.
| the8472 wrote:
| > You have to be participating in a torrent swarm in
| order to bootstrap yourself into DHT at all.
|
| This is not correct.
|
| > Bittorrent's DHT is not a network independent of
| torrent swarms.
|
| This is also wrong. People have used the DHT for non-
| torrent-related purposes.
|
| > You need the address of a peer who is already part of
| the network in order to join it, and you have to get a
| list those addresses from somewhere.
|
| And nothing dictates that that has to be obtained via the
| bittorrent peer protocol.
| https://stackoverflow.com/a/11089702/1362755
| pluto_modadic wrote:
| they're saying they jump between one and the other. ask
| DHT, then ask swarm, then ask DHT, then ask swarm.
| the8472 wrote:
| That doesn't make sense because those are concurrent
| processes, not serial ones.
| johnchristopher wrote:
| Won't you run into the problem that you will have to generate
| torrents every time you update your music collection ?
| rakoo wrote:
| You don't need an indexer for that. All you need is a static
| list with all your music, and people can click on links to
| download your music. Give them qbittorrent and it'll be running
| smoothly.
| jMyles wrote:
| Sure, but I'd like to go beyond magnet links and make it
| properly decentralized, with the discovery happening in the
| DHT and torrent swarm.
| CharlesW wrote:
| > _I want to seamlessly distribute my music through bittorrent,
| but since I have a small fan base (and thus, a very small
| potential seeding pool), I found it difficult to connect all
| the moving parts._
|
| Out of curiosity, why not just distribute via R2 or similar, or
| archive.org if you don't need AuthN/AuthZ? What's the
| complexity (to you and listeners) of BitTorrent buy you?
| jMyles wrote:
| Well, I do distribute via some centralized platforms
| (including some odious ones like Spotify).
|
| But I'd like to put forward a practice that demonstrates that
| the tools that have been smeared as anti-artist (chiefly but
| not only bittorrent) are actually compelling tools for
| independent distribution.
| felixg3 wrote:
| Post your magnet link and I'll host it on my Amsterdam-based
| server
| rakoo wrote:
| Another alternative is https://github.com/the8472/mldht which,
| contrary to magnetico, strives to be a nice citizen (its author
| is active in the bittorrent community AFAIU)
| mgdigital wrote:
| I have worked with the8472 to get Bitmagnet's BEP5 & BEP51
| implementations working and ensure it's a good citizen on the
| network - there is more to be done and more protocols to be
| implemented, but unlike Magnetico, BM is not simply scraping
| without responding to incoming requests.
| the8472 wrote:
| We had a discussion here https://github.com/bitmagnet-
| io/bitmagnet/issues/11 With the related changes bitmagnet
| shouldn't have the blatant misbehavior of magnetico
| (anymore). Though I haven't looked at its in-the-wild
| behavior, so I can't vouch for how spec-compliant the
| implementation plays in practice.
| mgdigital wrote:
| I'm also open to more feedback, if there are specific areas
| that need attention let me know :)
| swayvil wrote:
| Googled the bittorrent community with the acronym "AFAIU". Lol.
| bangonkeyboard wrote:
| Who remembers Skytorrents
| (https://news.ycombinator.com/item?id=13423629)? Posted as a
| "Show HN" here, it was a DHT-sourced index and stack written in C
| with no JavaScript, no cookies, no ads, no tracking. Skytorrents
| was unbelievably fast, friendly, and complete, and this
| translated into rapid adoption and traffic growth that caused the
| site to shut down due to server costs after just a year
| (https://torrentfreak.com/skytorrents-dumps-massive-
| torrent-d...).
|
| It was a shame that the technology behind Skytorrents was never
| open-sourced; it was the best torrent crawler and site I've ever
| seen, and I would have liked to see how it worked so well.
| mgdigital wrote:
| I guess the takeaway is that a public torrent site can
| (probably) never be sustainable while acting in users' best
| interests?
| evanjrowley wrote:
| Skytorrentz was great, but I never knew it had been coded in C
| nor presented here. Thanks for the background info.
| chopsuey5540 wrote:
| This looks interesting but I'm a bit worried about the CSAM /
| illegal stuff part, could a user get in trouble because he has
| traces of that in his crawled index? Also, how large is the index
| after indexing for a few months?
| rakoo wrote:
| An indexer doesn't download content. The only information
| you'll have is the name of a torrent, potentially its files,
| and who is interested in those files.
|
| But that's the technical view, what happens in court might be
| totally different.
| Scion9066 wrote:
| In order to get the information such as the name of the
| torrent and its files from the hash you do need to connect to
| someone in the swarm to download that metadata. You won't
| know what it is until after you've already connected.
| toyg wrote:
| Connecting to an unknown machine and asking what they have,
| is like knocking on a stranger's door and asking what
| they're selling. Them mentioning something nefarious and
| you leaving in response, is very obviously not a crime.
| rakoo wrote:
| There probably are nefarious content you can see just
| from the filenames but not everything is like that.
| Moreover, you "only" know they distribute it, you don't
| do it yourself.
| MomoXenosaga wrote:
| Considering many countries block torrent sites I wouldn't
| chance it.
| ZoomZoomZoom wrote:
| The real question is: metadata _is_ data, so are there any
| limitations on how much data can be transferred through DHT
| using well-behaving clients /servers so that you can be
| reasonably sure what you download on your machine isn't
| poisoned enough to possibly get you into trouble with the law
| enforcement?
| belorn wrote:
| The really interesting part for me is what this technology might
| lead towards in the end, which is decentralized community based
| curation. An index with white listed curation would be
| indistinguishable from a website, but it would not need a domain
| name nor a ip address to function.
| imtringued wrote:
| The problem with decentralised software is that you don't want
| to host other people's illegal content. I once tried out
| zeronet, which downloads the entire decentralised website and
| anyone can post things to it. Although I have not found CSAM
| directly on their Reddit equivalent, there are people posting
| advertisements to zeronet CSAM sites. The idea that I am
| downloading and automatically redistributing content like that
| is disturbing and zeronet is dead for a good reason. It's a
| pool that is asking to be peed in, even if the abusers
| themselves are a tiny minority.
| mgdigital wrote:
| Bitmagnet may download _metadata about_ CSAM content, which
| is automatically deleted with fairly high accuracy. You would
| _never_ be redistributing it. No outgoing peer protocol is
| currently implemented. This is planned but it will give users
| control of what they 're sharing rather than indiscriminately
| sharing everything.
| jesprenj wrote:
| Nice! I made my own DHT crawler in C for a science project in
| high school.
|
| git: http://ni.4a.si./sijanec/travnik/tree/src/dht.c
| taminka wrote:
| no offence, but this is like the exact type of C code that i
| imagine was dominant before modern C ijbol
| bartvk wrote:
| > modern C ijbol
|
| Can you clarify what this is?
| Rygian wrote:
| https://mashable.com/article/ijbol-explained
| nkohari wrote:
| I have no personal opinion on the code, but it was a high
| school project. Relax.
| jesprenj wrote:
| Implementing sample_infohashes opens your torrent client to
| abuse. UDP responses are much larger than queries and this
| protocol (BEP51) allows attackers that can spoof source IP
| addresses to use a large number of clients as mules for amplified
| distributed denial of service attacks.
| mgdigital wrote:
| Bitmagnet has rate limiting on incoming UDP requests (both
| overall and per-IP) so I don't know that it would be
| vulnerable; if there's anything else that should be done to
| mitigate any risk I'd like to know.
| pluto_modadic wrote:
| outgoing rate limits, and also it could exhaust the overall
| rate with useless traffic.
| mgdigital wrote:
| Incoming and outgoing are both limited, I think the worst
| such an attack could do is prevent responding to legitimate
| incoming queries - this shouldn't slow down the DHT crawler
| in a noticeable way.
| jesprenj wrote:
| I do not mean bitmagnet here, I mean other bittorrent clients
| that respond to sample_infohashes.
| p1mrx wrote:
| Has anyone tried to fix this by making queries bigger, and
| limiting the response to the size of the query?
| UberFly wrote:
| "Unlike well-moderated torrent sites, Bitmagnet adds almost any
| torrent it finds to its database. This includes mislabeled files,
| malware-ridden releases, and potentially illegal content. The
| software tries to limit abuse by filtering metadata for CSAM
| content, however."
|
| This is by far the biggest hurdle to something like this. You'll
| eventually have to end up again with a centralized curator.
| mgdigital wrote:
| Or connect with trusted curators over a decentralized network,
| similar to the fediverse?
| UberFly wrote:
| Or a baked-in a use-based rating system. That would be cool
| if it could be protected from abuse.
| yieldcrv wrote:
| they shouldn't do anything aside from a disclaimer
|
| just opt out of being an arbiter, like public utilities do
| UberFly wrote:
| Unfortunately the majority of it will be malware or illegal
| porn in the blink of an eye without a good way to separate
| it.
___________________________________________________________________
(page generated 2024-02-18 23:00 UTC)