Post ApvrNZA7lItXDqqehs by sirobsidian@fosstodon.org
 (DIR) More posts by sirobsidian@fosstodon.org
 (DIR) Post #ApvmoKgtAvfGO67MCO by ctrl@alico.nexus
       0 likes, 0 repeats
       
       @dansup@mastodon.social ah it doesn't honor robots.txt currently? 💀
       
 (DIR) Post #Apvqk80gmNZTiYd636 by sirobsidian@fosstodon.org
       0 likes, 0 repeats
       
       @dansup @ctrl Shouldn't anything that basically crawls websites honor robots.txt? Good job anyway making such a useful service though, even if for now it doesn't honor robots.txt.
       
 (DIR) Post #ApvrNZA7lItXDqqehs by sirobsidian@fosstodon.org
       0 likes, 0 repeats
       
       @dansup @ctrl Yeah I know what it's doing, I just used "websites" because well... /.well-known/nodeinfo is an endpoint on the web lol.