Subj : Web server and BOT's
To   : nelgin
From : MRO
Date : Thu Jun 26 2025 15:24:39

  Re: Web server and BOT's
  By: nelgin to Richard Vonzel on Wed Jun 25 2025 11:26 pm

 >> guess a BOT got on my site and jst downloaded all the file I had
 >> available. Here's the log entry for this,

 ne> I have found gptbot ignores a global disallow rule in robots.txt

 ne> User-agent: *
 ne> Disallow: /

 ne> But you could try adding


dude where have you been? most of those bots ignore robots.txt

google bot, archive.org's bots, all of them.  there's no law that they
have to obey robots.txt
---
 þ Synchronet þ ::: BBSES.info - free BBS services :::

.