Post AlcfbDlsIBDc6U9G7s by justine@mastodon.bsd.cafe
 (DIR) More posts by justine@mastodon.bsd.cafe
 (DIR) Post #AlcfbDlsIBDc6U9G7s by justine@mastodon.bsd.cafe
       2024-09-03T09:28:45Z
       
       1 likes, 0 repeats
       
       Question for all folk that self host their websites. Do you have a robots.txt and if so what's advisable to put in it ???#HomeLab #SelfHosted
       
 (DIR) Post #AlcfbEk8g7QN7OLO3k by a1ba@suya.place
       2024-09-03T09:56:09.954368Z
       
       0 likes, 0 repeats
       
       @justine I just disallow everything.Also it's easier to do within nginx config, so you don't have to modify any of the software.
       
 (DIR) Post #AlcsOGVI7WJSDZXn96 by amerika@annihilation.social
       2024-09-03T12:20:00.235350Z
       
       0 likes, 1 repeats
       
       @justine https://www.cloudflare.com/learning/bots/what-is-robots-txt/Crawl-delay: 8