Subj : File base with FTP Name with dot in it?
To   : g00r00
From : Bj”rn Wiberg
Date : Sun Mar 20 2022 09:21 pm

Hello g00r00!

I just noticed that Googlebot has begun crawling the anonymous FTP here. :-D

To my surprise, they seem to try to enter a /robots.txt subdirectory before trying to download /robots.txt:

+ 2022.03.20 07:03:04  FTP > Connect on slot 1/10 (66.249.64.92)
+ 2022.03.20 07:03:04  FTP 1-HostName crawl-66-249-64-92.googlebot.com
+ 2022.03.20 07:03:04  FTP 1-Country  United States of America (US)
+ 2022.03.20 07:03:04  FTP 1-C: USER Data: anonymous
+ 2022.03.20 07:03:04  FTP 1-S: 331 User name okay, need password.
+ 2022.03.20 07:03:05  FTP 1-C: PASS
+ 2022.03.20 07:03:05  FTP 1-S: 230 User logged in, proceed.
+ 2022.03.20 07:03:05  FTP 1-Logged in as Anonymous
+ 2022.03.20 07:03:05  FTP 1-C: CWD Data: robots.txt
+ 2022.03.20 07:03:05  FTP 1-S: 550 Directory change failed
+ 2022.03.20 07:03:05  FTP 1-C: TYPE Data: I
+ 2022.03.20 07:03:05  FTP 1-S: 200 All files sent in BINARY mode.
+ 2022.03.20 07:03:05  FTP 1-C: PASV Data:
+ 2022.03.20 07:03:05  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 07:03:05  FTP 1-C: RETR Data: robots.txt
+ 2022.03.20 07:03:05  FTP 1-S: 550 File not found
+ 2022.03.20 07:03:05  FTP 1-C: QUIT Data:
+ 2022.03.20 07:03:05  FTP 1-S: 221 Goodbye
+ 2022.03.20 07:03:05  FTP 1-Connection closed

So I thought that *perhaps* -- *if* they manage to change to that directory -- they'll try to fetch a robots.txt file from within it, i.e. /robots.txt/robots.txt. Although I haven't found any evidence of this in any specs from them (or in the Robots Exclusion Protocol draft/standard)...

So, I added a file base with:

FTP Name     ³ robots.txt

....and allowed anonymous access to it.

But any attempt to switch to that directory fails:

bbs@glimmer:/tmp$ pftp scbbs.nsupdate.info 50021
Connected to scbbs.nsupdate.info.
220 Welcome to Star Collision BBS!
Name (scbbs.nsupdate.info:bbs): anonymous
331 User name okay, need password.
Password:
230 User logged in, proceed.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> prompt
Interactive mode off.
ftp> ls -l
227 Entering Passive Mode (188,149,138,116,198,212).
125 Data connection already open
drwxr-xr-x   1 ftp      ftp             0 Mar 19 17:00 file_list
drwxr-xr-x   1 ftp      ftp             0 Mar 20 20:22 robots.txt
226 Closing data connection.
ftp> cd robots.txt
550 Directory change failed
ftp> ls robots.txt
227 Entering Passive Mode (188,149,138,116,198,212).
125 Data connection already open
drwxr-xr-x   1 ftp      ftp             0 Mar 20 20:22 robots.txt
226 Closing data connection.
ftp> ls robots.txt/
227 Entering Passive Mode (188,149,138,116,198,212).
125 Data connection already open
226 Closing data connection.
ftp> ls robots.txt/robots.txt
227 Entering Passive Mode (188,149,138,116,198,212).
125 Data connection already open
-rw-r--r--   1 ftp      ftp            26 Mar 20 20:24 robots.txt
226 Closing data connection.
ftp> mget robots.txt/robots.txt
local: robots.txt remote: robots.txt
227 Entering Passive Mode (188,149,138,116,198,212).
550 File not found
ftp> quit
221 Goodbye

And the MIS log shows pretty much the same:

+ 2022.03.20 21:11:14  FTP > Connect on slot 1/10 (192.168.1.1)
+ 2022.03.20 21:11:14  FTP 1-HostName router.asus.com
+ 2022.03.20 21:11:14  FTP 1-Country  Unknown (-)
+ 2022.03.20 21:11:16  FTP 1-C: USER Data: anonymous
+ 2022.03.20 21:11:16  FTP 1-S: 331 User name okay, need password.
+ 2022.03.20 21:11:17  FTP 1-C: PASS
+ 2022.03.20 21:11:17  FTP 1-S: 230 User logged in, proceed.
+ 2022.03.20 21:11:17  FTP 1-Logged in as Anonymous
+ 2022.03.20 21:11:17  FTP 1-C: SYST Data:
+ 2022.03.20 21:11:17  FTP 1-S: 215 UNIX Type: L8
+ 2022.03.20 21:11:22  FTP 1-C: PASV Data:
+ 2022.03.20 21:11:22  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 21:11:22  FTP 1-C: LIST Data: -l
+ 2022.03.20 21:11:22  FTP 1-Listing files in /
+ 2022.03.20 21:11:22  FTP 1-S: 125 Data connection already open
+ 2022.03.20 21:11:22  FTP 1-S: 226 Closing data connection.
+ 2022.03.20 21:11:30  FTP 1-C: CWD Data: robots.txt
+ 2022.03.20 21:11:30  FTP 1-S: 550 Directory change failed
+ 2022.03.20 21:11:35  FTP 1-C: PASV Data:
+ 2022.03.20 21:11:35  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 21:11:35  FTP 1-C: LIST Data: robots.txt
+ 2022.03.20 21:11:35  FTP 1-Listing files in /
+ 2022.03.20 21:11:35  FTP 1-S: 125 Data connection already open
+ 2022.03.20 21:11:35  FTP 1-S: 226 Closing data connection.
+ 2022.03.20 21:11:40  FTP 1-C: PASV Data:
+ 2022.03.20 21:11:40  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 21:11:40  FTP 1-C: LIST Data: robots.txt/
+ 2022.03.20 21:11:40  FTP 1-Listing files in robots.txt
+ 2022.03.20 21:11:40  FTP 1-S: 125 Data connection already open
+ 2022.03.20 21:11:40  FTP 1-S: 226 Closing data connection.
+ 2022.03.20 21:11:46  FTP 1-C: PASV Data:
+ 2022.03.20 21:11:46  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 21:11:46  FTP 1-C: LIST Data: robots.txt/robots.txt
+ 2022.03.20 21:11:46  FTP 1-Listing files in robots.txt
+ 2022.03.20 21:11:46  FTP 1-S: 125 Data connection already open
+ 2022.03.20 21:11:46  FTP 1-S: 226 Closing data connection.
+ 2022.03.20 21:11:53  FTP 1-C: PASV Data:
+ 2022.03.20 21:11:53  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 21:11:53  FTP 1-C: NLST Data: robots.txt/robots.txt
+ 2022.03.20 21:11:53  FTP 1-Listing files in robots.txt
+ 2022.03.20 21:11:53  FTP 1-S: 125 Data connection already open
+ 2022.03.20 21:11:53  FTP 1-S: 226 Closing data connection.
+ 2022.03.20 21:11:53  FTP 1-C: TYPE Data: I
+ 2022.03.20 21:11:53  FTP 1-S: 200 All files sent in BINARY mode.
+ 2022.03.20 21:11:53  FTP 1-C: PASV Data:
+ 2022.03.20 21:11:53  FTP 1-S: 227 Entering Passive Mode (188,149,138,116,198,212).
+ 2022.03.20 21:11:53  FTP 1-C: RETR Data: robots.txt
+ 2022.03.20 21:11:53  FTP 1-S: 550 File not found
+ 2022.03.20 21:11:56  FTP 1-C: QUIT Data:
+ 2022.03.20 21:11:56  FTP 1-S: 221 Goodbye
+ 2022.03.20 21:11:56  FTP 1-Connection closed

So, switching to that directory appears impossible, and listing of file details in it only appears to work if you supply the exact file name.

However, if I change the FTP Name to something without a dot in it, things work as expected (but then, that is not the name I want/need for the directory).

Do you know what could be causing this?

And, is there perhaps a better way of getting a /robots.txt in place (i.e. in the root directory) on the FTP server?

Many thanks in advance! =)

Best regards
Bj”rn

--- Mystic BBS v1.12 A48 2022/03/11 (Linux/64)
 * Origin: Star Collision BBS, Uppsala, Sweden (2:201/137)

.