Post AXbkbdZBMLKHdLGIhU by theruran@hackers.town
 (DIR) More posts by theruran@hackers.town
 (DIR) Post #AXbkbceSlDxKnQj0IC by theruran@hackers.town
       2023-07-12T03:38:00Z
       
       0 likes, 1 repeats
       
       hey so if I can reject Googlebot in robots.txt based on user-agent, then what about OpenAI scrapers? Bing/Microsoft?
       
 (DIR) Post #AXbkbdZBMLKHdLGIhU by theruran@hackers.town
       2023-07-12T03:40:26Z
       
       1 likes, 0 repeats
       
       ah, yes:User-agent: ChatGPT-UserDisallow: /If you want to block it at the IP address level, you need to block the following IP address: 23.98.142.176/28https://webmasters.stackexchange.com/questions/142359/is-it-possible-to-exclude-just-openai-chatgpt-from-scraping-my-website
       
 (DIR) Post #AXbkbePIEb0gExduvQ by theruran@hackers.town
       2023-07-12T03:41:39Z
       
       0 likes, 0 repeats
       
       oh, hmm:Thanks for the pointer! that doesn't seem to be their scraper bot though (quote from the link: "not used for crawling the web"). I was also curious if Common Crawl CCBot, Bingbot or any other bots are helping them as well. – Ivan Balepin, May 17 at 1:48Common Crawl CCBot is another culprit to be banned from my website.