[HN Gopher] Ask HN: I've Built a DHT Torrent Sniffer and Search ...
___________________________________________________________________
Ask HN: I've Built a DHT Torrent Sniffer and Search Engine. Should
I Release?
Recently I was researching about DHTs and developed a DHT Sniffer
in Go which connects to some known DHT Routers and sniffs all the
annoucements. I've quickly added ZincSearch and it is now basically
a search engine which can search for hashes, name or files
contained in the torrents. It is able to index around 5-10k
annoucements per second, so the index grows quite fast. Now, I am
thinking about releasing it as open-source for others to study, but
not sure if I should, because it might be used for "evil".
Author : sylwester
Score : 140 points
Date : 2022-10-23 09:18 UTC (13 hours ago)
| brobinson wrote:
| Let it rip. DHT has been around for so long now that whatever bad
| actors/evil use cases you're imagining have already happened. It
| sounds like a cool project, and I'd be interested to see it.
| sshine wrote:
| I accidentally read this as "Let it R.I.P."
|
| I totally agree, unroll it!
|
| While the file sharing is distributed, the centralised web-
| based indexing is still a game of whack-a-mole.
| MuffinFlavored wrote:
| how is what OP made not basically going to be the p1rat3 bay
| 2.0?
| r3trohack3r wrote:
| It already exists. BTDigg:
| https://en.m.wikipedia.org/wiki/BTDigg
|
| And sibling comments show there is a community of folks
| doing this.
|
| People also do it to Tor's DHT too for discovering hidden
| services: https://donncha.is/2013/05/trawling-tor-hidden-
| services/
|
| You can't really stop open source tech like this from
| existing by choosing to not open source your
| implementation. You just end up staying out of the
| conversation yourself. The community will still build it if
| there is a desire for it.
| haswell wrote:
| One is a repository of code that you or I can run on our
| own machines.
|
| The other is a publicly accessible website that got a kit
| of public attention because it could easily be used by
| anyone including novices to pirate content.
| thomastjeffery wrote:
| Moderation.
|
| TPB (and other trackers) are essentially forums.
| jrm4 wrote:
| _Please_ do. This old-ish timer is kind of blown away by the idea
| that you shouldn 't.
|
| Nothing evil about being a modern archivist/librarian, despite
| what big companies would tell you.
| majestic5762 wrote:
| I wrote a similar solution 8 years ago. I repurposed the system
| to identify IPs owned by the government, and notified them if a
| malicious copy of Windows (but not limited to) was seeded by
| them. Meaning there was a chance that an unknown actor had a
| backdoor in my government's network. If you wanna discuss, I'm
| happy to talk and even contribute towards a commercial solution
| majestic.hn@fastmail.com. I didn't pursue this opportunity for
| money at that time, but I had my fair share of "shoutz". Won't be
| bad to pick this up again
| sascha_sl wrote:
| I mean, it already exists. [1] Always fun to see what my
| neighbors behind the same NAT download.
|
| [1]: https://iknowwhatyoudownload.com/
| insane_dreamer wrote:
| I thought streaming had spelled the demise of torrenting, so
| was v surprised to check out my local hood (quiet suburban,
| skewing older few ppl <40yrs) and see how it's very much alive
| and well. I guess I was naive. (Not condemning anyone, as I did
| my share of peering back in the day, but streaming services are
| convenient and honestly don't cost that much all things
| considered. Torrents (tho I prefer IRC) good for finding
| obscure stuff that's unavailable anywhere else, but that's not
| what my neighbors are downloading; and it's not like any of
| them can't afford to pay. Interesting dynamic.
| oefrha wrote:
| Ah, haven't checked that one for a while. Surprised to find a
| tech-minded neighbor who has downloaded kali-linux-2022.3-live-
| everything-amd64.iso. And no pr0n this time round.
| IYasha wrote:
| You'd better have a good firewall since you have such a tech-
| minded neighbor. )
| imhoguy wrote:
| Checked it for my Transmission host static IP which doesn't
| download anything for a while and it has some completely random
| stuff I never downloaded or searched for. Even shows items from
| today which for sure I haven't requested. Could be some DHT
| spoofing or my Transmission is relaying some DHT requests?
| Klaster_1 wrote:
| It gives quite fascinating results for me. Surprisingly, among
| dozens of torrents only a couple are porn. Most of my neighbors
| using BitTorrent are Russians and Ukrainians (I'm in Cyprus),
| judging by files names and seeder distribution, a couple of
| English-speaking people (probably brits) and a single person
| seeding content in Greek. At least half of Russian-speaking
| neighbors seed IT, engineering and English learning materials.
| And I seed more than everyone together on the same subnet Wish
| I could message them.
| acehw wrote:
| it's a tool. What people use it for is their business.
| zelienople wrote:
| computerfriend wrote:
| I don't know why you think the OP wants to be regarded as a
| professional. But you'd do well to adopt a little
| professionalism in your communication too.
| synctext wrote:
| peterkelly wrote:
| OP said they already have a working system. How _well_ it works
| remains to be seen, until it 's publicly available.
|
| On what basis do you assume he/she has not succeeded without
| even having seen their work?
| synctext wrote:
| We tried too. Several people have made DHT sniffers,
| including our lab. You see lots of spam and get stuck.
| Filtering out the noise is weirdly hard.
| simjnd wrote:
| So you're assuming they cannot possibly have succeeded
| simply because you failed?
| synctext wrote:
| these things need spam measures to work. That was not
| listed as a feature.
| have_faith wrote:
| > We are building a micro-economy without banks, without
| advertisers, and without any government
|
| But with government funding?
| dhaavi wrote:
| Tribler is an interesting project, but not directly comparable
| to what the OP is trying to achieve.
| synctext wrote:
| indeed, bit different. lots of people worked in this field
| and made these tools. Around 2014 the state of the art was
| 20k DHT responses per second, see '100 million DHT replies'.
| https://doi.org/10.1109/P2P.2014.6934318
| dhaavi wrote:
| Thanks for the pointer!
|
| I just read some interesting proposals in the GH issues and
| recognized your username. This is some nice piece of work -
| will definitely return to read more about the underlying
| concepts.
|
| We are working in a similar area with similar problems:
| Building a new Tor/VPN-like privacy network. See
| https://safing.io/spn/
| simjnd wrote:
| Taking this chance to say thank you for Safing Portmaster
| it is awesome!
| pharmakom wrote:
| Who is funding it and why?
| someotherperson wrote:
| If you follow through to the site linked in the Github:
|
| > Tribler is a research project of Delft University of
| Technology
|
| > Work on Tribler has been supported by multiple Internet
| research European grants. In total we received 3,538,609 Euro
| in funding for our open source self-organising systems
| research. Roughly 10 to 15 scientists and engineers work on
| it full-time. Our ambition is to make darknet technology,
| security and privacy the default for all Internet users.
|
| > Vision & Mission ... "Push the boundaries of self-
| organising systems, robust reputation systems and craft
| collaborative systems with millions of active participants
| under continuous attack from spammers and other adversarial
| entities."
| IYasha wrote:
| The "robust reputation systems" made me twitch a bit...
| synctext wrote:
| See the latest 2022 algorithm called MeritRank [1].
| Improves on the prior work developed together with
| Harvard. We deployed roughly 5 generation in the past 17
| years. [1] https://arxiv.org/abs/2207.09950
| BLKNSLVR wrote:
| I'll add to the chorus of people saying "yes, release it".
|
| If you're worried about blowback as a result of "evil" uses /
| users, is there a way to release it (somewhat) anonymously, so
| it's difficult to be traced back to you?
| keroro wrote:
| torrent-paradise [0] is a go project which seems to do the same
| as your project and has existed since 2019. It's since gone down
| but remains up on IPFS [1] but its index hasn't updated since
| January.
|
| [0] https://github.com/urbanguacamole/torrent-paradise
|
| [1] https://cloudflare-
| ipfs.com/ipfs/QmQjsKamNFZRvCMXDvZXQmRYjsm...
| mdaniel wrote:
| > remains up on IPFS [1] but its index hasn't updated since
| January.
|
| But only the "about" page remains up, right? Because both
| https://cloudflare-ipfs.com/ipns/torrent-paradise.ml and
| https://cloudflare-ipfs.com/ipns/12D3KooWB3GY1u6zMLqnf3MJ8zh...
| show a command-line(!) error
|
| If my mental model of IPFS is correct, it means no one is
| seeding the underlying IPFS ... CIDs? ... that those links
| point to
| ehPReth wrote:
| please do!
| icpmoles wrote:
| Is it basically btdig.com ?
| storrgie wrote:
| An advantage here would be people self hosting their own based
| on this project.
| encryptluks2 wrote:
| Btdig is awesome. If this is like an open source version in Go
| then that is a huge contribution.
| steinuil wrote:
| Isn't btdig also open source? I see a "fork me" button on the
| front page and it leads to a repo: https://github.com/btdigg-
| org/dhtcrawler2
| encryptluks2 wrote:
| It is 9 years old and in Erlang.
| the8472 wrote:
| Having seen other indexers before I suspect your implementation
| isn't spec-compliant or well-behaved (perhaps spoofing node-IDs?
| causing more traffic than necessary?)
|
| If you want to build an indexer you should write a normal
| implementation and then use
| http://bittorrent.org/beps/bep_0051.html
| rahimnathwani wrote:
| Related comment:
|
| https://news.ycombinator.com/item?id=14022107
| hardwaresofton wrote:
| Would you mind explainng why you chose ZincSearch? Curious on why
| you picked it over some of the other non-ElasticSearch/OpenSearch
| alternatives (Meilisearch, Typesense)
| berry_sortoro wrote:
| compressedgas wrote:
| Consider that such already is available as open source in Go
| even.
| arthurcolle wrote:
| Magnetissimo did this too, and he's still around.
| Proven wrote:
| navjack27 wrote:
| That would be very useful... Just release the code and building
| instructions.
| thinkmcfly wrote:
| I think you should. From my understanding, use of DHT is already
| dead in the eyes of most torrenters
| qualudeheart wrote:
| Which evil usages are you concerned about? I think it would be
| very useful for the public.
| seqastian wrote:
| copyright holder bots
| Ysx wrote:
| DHT has existed for 17 years - the cat's out of the bag. The
| anti-piracy companies have built their own crawlers.
| aliqot wrote:
| They match swarm IP's and then notify ISP's who voluntarily
| hassle you on their behalf. An IP doesn't equal a person in
| USA.
| netsharc wrote:
| And outside of the US? In Germany there's an "efficient"
| business of law companies acting on behalf of the
| copyright holders, subpoenaing the ISP to get the owner
| of the IP/connection and sending them a cease-and-desist
| with a hefty fee.
|
| There's the next part, law companies specializing in
| "Here's the letter you have to send to say you agree to
| pay them without admitting guilt, that'll be EUR
| please!".
| TillE wrote:
| I think the demand is usually around 1000 Euro, so it's
| way cheaper to just pay for a VPN service, configure
| qBittorrent to only use your VPN interface, and torrent
| whatever the hell you want.
| retonato wrote:
| I have been operating a BitTorrent search engine for a few years
| (it is closed now), here is some advice if you want to launch it
| as a public website:
|
| 1. There is no absence of people opening the kind of torrent
| indexers, which you have in mind. In 2010s there were 5-10 such
| sites launched each year, even nowadays there are at least a few
| new indexers yearly. Almost all are closed within a year or two
| (at most). There are reasons for that, some are less obvious than
| others.
|
| 2. Most likely you will close your site after some time as well,
| here is why:
|
| 2.1. It will be difficult to find a hosting, which will tolerate
| it. Forget about Linode or Digital Ocean or any similar hosting
| providers - they will suspend your server (first) and account
| (later) after receiving a few automated DCMA emails. You can use
| some "second-tier" providers for some time, but eventually even
| they will get tired of you, you will be out the moment they
| receive the first paper letter from some law company, which
| represents some movie company or alike. You can use proxy servers
| or some other arrangement for some time, but eventually you will
| decide that it isn't worth the effort.
|
| 2.2. The same thing is true for domain provider. Yes, there are a
| few, which are more resilient, than others, but don't expect that
| they will protect you and your domain indefinitely for $15 per
| year. If your site is popular enough - the only option may remain
| to register a new domain every few months and hop between them.
| Eventually you will get tired of that. It may take half a year or
| a few years, but you will.
|
| 2.3. There are no money in torrenting nowadays. Forget about ads
| or donations - even thepiratebay cannot earn money that way, you
| will not be able to do that either.
|
| 2.4. Sadly, torrents are not that popular nowadays as they were
| 5-10-15 years ago. Some people still use them, but in most
| developed countries that's more like an exception, than the rule.
| The rule is netflix, spotify and alike. As a result, if you
| imagine large and happy community of users around your site -
| just don't, most likely there won't be any.
|
| 2.5. Don't expect just to launch a site and rest. You will spend
| at least 5-10-20 hours per week on its maintenance (fixing bugs,
| importing/cleaning data, adding new features, etc). It will feel
| fine at first, but more and more tiring as the months and years
| go by. Eventually you will get bored and stop any maintenance.
| Users don't like old/unmaintained sites, so they will be less and
| less interested in it. Eventually you will decide, that it isn't
| worth the effort to run it at all.
| retonato wrote:
| Don't underestimate possible legal problems. Tech people tend
| to simply throw them aside ("I will use a VPN/Cloudflare/Onion,
| no one will find me"), but here are my 5 cents:
|
| 1. Most likely - you site will not gain any significant
| popularity and you will decide to close it on your own.
| Meanwhile you will just be getting some automated DMCA
| emails/complaints from time to time, that's it.
|
| 2. Less likely - you site will be somewhat popular, it will
| become an attractive target for copyright owners and law firms
| (not as attractive as thepiratebay, but still). It usually
| takes just a month or two for some interested law enforcement
| agency to find out who you are (forget about all privacy
| protection methods you know, it is very easy to slip and
| eventually you will slip; you need to be lucky every time, they
| need to be lucky just once). Depending on what country you
| live, you will end up with a few thousand dollar fine (best
| case) or tens/hundreds of thousands + a small jail sentence + a
| criminal record. Not bad for a hobby torrent site with no
| income and a minimal community, is it?
|
| Just to reiterate this one more time - it is VERY easy to find
| out who you are for any interested government agency [in your
| country], most of those 1-2 months will be spent waiting for
| replies for their letters and other similar bureaucracy, not
| for some CSI stuff.
|
| Here is how it might go. First they will check the site domain
| and IP address and ask your hosting/domain providers to reveal
| you payment details (name/address on credit card). That's
| enough in 95% of all cases. Even if you use Cloudflare to hide
| your site IP and even if you use bitcoins to pay for everything
| - that will help only for some time, eventually Cloudflare will
| reveal your server IP and bitcoin transaction will be tracked
| back to your credit card. That's just a question of time and
| whether you are "wanted" enough or not.
| retonato wrote:
| If despite all this you still decide to go forward - don't
| forget to import all the available torrent data on the
| internet (there are tens of dumps here and there), that way
| you will have MUCH larger database, than just by DHT scraping
| alone. This is a good place to start:
| https://archive.org/details/torrent_metadata_archive_sample
|
| Creators of such sites (including me) tend to focus too much
| on the number of torrents, no matter if they are active or
| dead. Regular users are interested mainly in active torrents.
|
| Plus they want to see the current number of seeders/leechers,
| which is very difficult to keep up-to-date for a large
| database.
|
| Plus they want to see a torrent creation/upload date, which
| you cannot get from DHT (you can record the day you found a
| torrent, but it will work only for newer torrents, not for
| historical ones).
| retonato wrote:
| Of course, you can just provide a code for users to run on
| their own computers, but don't expect that anyone will
| really use it (maybe just a few people here and there, I
| really mean it). Everyone, who is really hardcore enough to
| run something on their computers to obtain torrents will
| just use Jackett (https://github.com/Jackett/Jackett). It
| can search through the huge number of torrents, which no
| local DHT scraping/search engine can provide.
| gwnywg wrote:
| I was planning to start learning GO, I'd be definitely interested
| to learn from your project :)
| joeman1000 wrote:
| Please release it. We are in dire need of good torrent search
| without ad-trackers or other nasty stuff.
| boramalper wrote:
| I had been working on this successfully for a couple years in the
| past before I got tired of it and moved on. I still think it's a
| magnificent idea, to be able to host your own torrent site and to
| decentralise the last centralised bit of BitTorrent.
|
| https://github.com/boramalper/magnetico
| rakoo wrote:
| I've built a small cli utility to search the database:
| https://sr.ht/~rakoo/magneticos/. It also asks the DHT and
| known trackers for the liveness of the swarms, so I can sort by
| seeders. It's simpler and more useful than running a full
| webserver with ports and all when I only need an ssh access. It
| has served me well and will be more than enough for a long time
| thomastjeffery wrote:
| The main thing that seems to be missing, in my experience, is
| moderation.
|
| If someone can manage to make a decentralized forum-like
| moderation list, then we wouldn't have any use for centralized
| trackers anymore.
|
| I think it could probably be done by using public GPG keys as
| identities, then keeping changes in a torrent-distributed git
| repo where user-generated data is saved in GPG signed git
| commits.
|
| Then the only centralization left would be what branch/torrent
| people introduce to new users.
| mdaniel wrote:
| > I think it could probably be done by using public GPG keys
| as identities, then keeping changes in a torrent-distributed
| git repo where user-generated data is saved in GPG signed git
| commits.
|
| Congratulations, you won some HN bingo squares by introducing
| both PKI and blockchain to the discussion :-D
|
| But, in all seriousness: moderation against _what_?
| hardwaresofton wrote:
| Thanks for writing that note on the repository -- it was
| thoughtful and well explained -- reading through your small
| manifesto that was linked and that's fun too.
| kevinwang wrote:
| I think the manifesto was Aaron Swartz's
| hardwaresofton wrote:
| oh you're right I didn't even notice -- aaronsw.com...
| First time I've read it.
| 2Gkashmiri wrote:
| yes please. you are not responsible for any "Evil" users of the
| software might do. This is not even about enabling bad stuff,
| this is just natural progression of technology.
| rrwo wrote:
| Nuclear and biological weapons are also part of the "natural
| progression of technology" but it's widely agreed that they
| shouldn't be released to anybody and everybody, and the people
| distributing those technologies have a moral responsibility.
|
| If you think your software would be of more use to "evil" than
| not, then don't release it widely.
| loa_in_ wrote:
| Yeah, it is everyone's moral responsibility to consider
| effects of our actions as far as we're able to.
| effnorwood wrote:
| 0dayz wrote:
| I would recommend writing down the worst and best case scenarios
| that could happen with your software, then determine if you
| notice either that through severity or quantity the software
| outweighs the positives, don't release it.
| tmtvl wrote:
| Just AGPL it, I hear it's an effective ward against Alphabet.
| Eduard wrote:
| Very good suggestion, considering the sniffer could be used by
| copyright claims lawyers.
| MattPalmer1086 wrote:
| Can't see how AGPL would stop anyone using it. Doesn't it
| just prevent them from modifying the source code without
| sharing their changes. ?
| webmobdev wrote:
| AGPL fixes a "bug" in GPL that many tech companies have
| exploited to not release modified source code as the GPL
| requires. Simply put, GPL says that you have to release the
| complete source code (including any changes you have made)
| of a GPL licensed code only if you distribute it to other
| users. Many tech companies thus avoided GPL code. But with
| the growth of Software-as-a-Service, where an application
| only runs on the server and is accessed through a browser
| or an app, many of these same companies created web
| applications with GPL code that they customised. However,
| if a user demanded the source code of the GPL code, along
| with the modifications they made, the tech companies
| refused to provide it claiming they weren't "distributing"
| the application (as in giving you the whole application to
| run on your computer). And since they weren't doing that
| they claimed they had no legal obligation to release the
| complete source code.
|
| AGPL fixes this - it recognizes SaaS web applications too
| as a "distribution" of GPL software. So if a source code is
| licensed under AGPL, anyone who uses it to create web
| applications and makes it available to the public is now
| also legally obliged to provide the complete source code if
| any user requests for it. (And ofcourse, as with GPL the
| user is free to use the source code as they want, freedom
| to improve it, and even start competing SaaS services).
|
| That is why the AGPL is currently the best GNU license to
| ensure that your open source code always remains open
| source.
| Shadowed_ wrote:
| How would someone know if SaaS uses some AGPL code?
| kruuuder wrote:
| In general, you don't, but that's a different question.
| remram wrote:
| It only prevents them from modifying the code and not
| sharing their changes _and making the service available to
| another company_.
|
| You are still free to make private changes to AGPL code and
| run it for yourself (or your company).
| remram wrote:
| AGPL wouldn't stop copyright lawyers from using it or even
| making changes and not sharing. It would only stop _another
| company_ from making changes and not sharing them and selling
| the service to lawyers.
| yieldcrv wrote:
| yeah release it
| pdimitar wrote:
| I imagine any bad actors who store IPs of torrent seeders have
| done so a long time ago already so your software will not do any
| harm that hasn't been done already.
|
| Go for it and open-source it.
| hombre_fatal wrote:
| This is a good example of our hubris as developers. We like to
| think our project will have some impact on the world when in
| reality you're extremely lucky if anybody notices much less
| cares. ;)
| r3trohack3r wrote:
| You should absolutely open source this. Working with the DHT is a
| lot of fun, and a great learning exercise.
|
| I adapted my local "torrent roulette" application to an electron
| app that can be shared. My local version downloads the files, but
| the one I share only fetches the torrents metadata (easy to adapt
| it back to my roulette approach). I call it Taboo:
| https://github.com/retrohacker/taboo
|
| The amount of "evil" on the DHT is pretty low. I've run mine for
| a long time, and very rarely get anything evil. I suspect it's
| because of how poorly BitTorrent plays with privacy tools like
| VPN and Tor, IIUC it's easy to leak identifying information with
| BT and its high bandwidth.
|
| Nearly every "evil" file I've found is either: a honeypot with
| not-evil content or password encrypted (maybe to remove the
| plausible deniability of a random download?). I don't know if the
| encrypted files actually contain evil content, I don't bother
| trying to crack them and promptly gshred them.
|
| What you will find:
|
| * a lot of content illegal under US copyright law
|
| * a lot of porn (also illegal under US copyright law)
|
| * a metric tonne of fascinating content from other cultures you'd
| otherwise not be exposed to (also probably illegal under US
| copyright law)
|
| There is very little "legal" content on the DHT, but most of it
| is falling on the wrong side of intellectual property law.
| Sometimes password/credit card dumps. I once found some very
| sketchy schematics of Eastern European military equipment. But
| that stuff is also really rare in my experience.
|
| If you do play with these systems, I'll leave the same warning I
| left on the Taboo repo:
|
| > Note: I AM NOT A LAWYER! To my knowledge, there aren't any
| other systems doing this that you can run on your laptop. I
| suspect the nuance of how Taboo works isn't going to be
| appreciated by your local law enforcement. If you don't want to
| test the legality of Taboo in court, I'd strongly recommend
| either: running a VPN (less safe) or not using Taboo (most safe).
| If you want to use Taboo with a VPN, may I suggest putting some
| money in an envelope and sending it to Mullvad?
|
| Adding this on after reading your comments elsewhere in the
| threads:
|
| If you're worried about Intellectual Property enforcers using
| this for evil, I wouldn't worry too much about it. BTDigg already
| exists. Not that what you're doing isn't novel or exciting (great
| work on this BTW) - but DHT indexing is an art thats already
| being practiced. Cat is out of the bag.
| Onanymous wrote:
| I like how clearly you distinguish "evil" and "not legal" :)
| tcmb wrote:
| If I may suggest, it would be helpful to have a section in the
| README about how to build/run the application. If I'm not
| familiar with Electron, the repository isn't very helpful, even
| though I might be interested in trying it out.
| tcmb wrote:
| I managed to build and run it, filed an issue for a problem I
| ran into. Hope it's valid, thanks for making this app!
| r3trohack3r wrote:
| Hey, thanks! Let me know what you think of it after using.
| It was a fun project to work on.
| mutant wrote:
| https://github.com/boramalper/magnetico
|
| Someone else did this a while back, universe continues to exist.
___________________________________________________________________
(page generated 2022-10-23 23:01 UTC)