Post B7rYnc5fOA2ezyxQki by galaxis@mastodon.infra.de
 (DIR) More posts by galaxis@mastodon.infra.de
 (DIR) Post #B7rYnc5fOA2ezyxQki by galaxis@mastodon.infra.de
       0 likes, 1 repeats
       
       Also, yet another Fediverse crawler:
       
 (DIR) Post #B7s0JBb87W2QrXxCNc by mirabilos@toot.mirbsd.org
       0 likes, 0 repeats
       
       @galaxis what do I need to defederate and/or user-agent-block to avoid that one?
       
 (DIR) Post #B7s3N2tuhIBYuUGuNk by galaxis@mastodon.infra.de
       0 likes, 1 repeats
       
       @mirabilos It comes with a user agent of  "Mozilla/5.0 (archiv +http://archivnja236kukzwoakbpenv2n65yi3yanupz4tqa54i2nka2ciivid.onion/optout.html)"Apparently you can enter your instance name on that page or "to confirm that this is your instance, add a TXT to the domain: _archiv-optout.@ with value yes" 🙄I blocked the user agent for now.
       
 (DIR) Post #B7s3N36fvqOTY4F6H2 by galaxis@mastodon.infra.de
       0 likes, 1 repeats
       
       @mirabilos Others that were new to me are "Mozilla/5.0+(compatible; FediralBot/1.0; https://www.fediral.com/bot)"Plus the Social Web Foundation crap with tags.pub and their other crawler, "crawler.pub/0.8.0 (https://crawler.pub/; evanp@gatech.edu)"
       
 (DIR) Post #B7s61hq07845MhxR9E by galaxis@mastodon.infra.de
       0 likes, 1 repeats
       
       @mirabilos Also "HolosDiscover/1.10.0 (+https://discover.holos.social)" and "fediway-ingest/0.1 (+https://feeds.fediway.com/actor)"