Indy FTP Server Error Question (Could not bind socket) lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- snorkel · 2016-11-30 14:54 Hi, I have a windows service I wrote with the Indy 10 FTP server (which works great by the way) Anyways, I recently had the need for the server to listen on two ports (21 and a non standard port of lets say 30021) So I added the bindings and it indeed is listening on both ports now. the problem seems to be that after I did this once in awhile I get errors like this: FTP server error:(10.0.8.1:60963) Could not bind socket. Address and port are already in use. I did set both bindings to listen on 0.0.0.0 and the server is set to reuse sockets  (property ReuseSocket set to rsTrue) Anyone have any ideas?  should I use the actual IP of the server instead of 0.0.0.0 for the two bindings? ---------------------------------------------------------------------- Remy Lebeau · 2016-11-30 22:12 Quote from: snorkel on November 30, 2016, 03:54:38 pmthe problem seems to be that after I did this once in awhile I get errors like this: FTP server error:(10.0.8.1:60963) Could not bind socket. Address and port are already in use. My guess would be that a client has connected to the server and is performing a passive mode file transfer and the server is trying to bind a new transfer socket to a listening port that is not available.  What do you have the server's PASVBoundPortMin, PASVBoundPortMax, and DefaultDataPort properties set to?  It is important that for Passive transfers, the server has a pool of multiple ports available so multiple transfers can run simultaneously on separate ports. Quote from: snorkel on November 30, 2016, 03:54:38 pmthe server is set to reuse sockets  (property ReuseSocket set to rsTrue) That property is only used for the main command sockets.  Secondary transfer sockets do not use that property. Quote from: snorkel on November 30, 2016, 03:54:38 pmshould I use the actual IP of the server instead of 0.0.0.0 for the two bindings? That is not necessary. ---------------------------------------------------------------------- snorkel · 2016-12-08 21:06 Hi Remy, PASVBoundPortMin = 60000 PASVBoundPortMax = 65000 DefaultDataport = 20 Do you think the passive port range is maybe too large?  Is it possible some other script or whatever is once in awhile grabbing from that range? got another one today:  FTP server error:(10.0.8.1:40913) Could not bind socket. Address and port are already in use. ---------------------------------------------------------------------- snorkel · 2016-12-08 21:16 I did notice that in the FTPSecurityOptions there is a option called: NoReservedRangePort and it's set to true, would that have anything to do with this? ---------------------------------------------------------------------- Remy Lebeau · 2016-12-08 22:00 What you describe does not make sense.  60963 is in your PASV port range, but TIdFTPServer would not raise a bind error on a Passive-mode transfer unless the full range of PASV ports is exhausted (and even then, the error message would be different).  But where is port 40913 coming from?  For that matter, where is your error message actually coming from and what does the call stack look like when the error is raised?  Indy does not add IP/Port information to that error message, so were are you getting it from? The specific error message you have shown is only raised under one condition - failure to bind a socket to a specific port, not to a range of ports.  And the only times that TIdFTPServer binds a socket to a specific port are: each listening Binding.all Active-mode transfers, which are bound to DefaultDataPort.  These are outbound connections, so multiple transfers can be bound to the same port at the same time as long they are connecting to different IP/Port targets.Passive-mode transfers, which are bound to DefaultDataPort when PASVBoundPortMin/Max are 0.  These are inbound connections, so multiple transfers can be bound to the same port at the same time if different IP/Port sources are connecting. Whether or not the PASV range of ports is too large depends on how many simultaneous transfers you want to support, and how many other socket apps are running along your server.  You are using a range of 5001 ports.  When a Passive-mode transfer is requested, TIdFTPServer is going to try opening each port in the range until it finds one that is available. As for NoReservedRangePort, if it is enabled then TIdFTPServer will reject an Active-mode transfer that requests a connection to a port between 1-1024, inclusive.  On some platforms, particular *Nix systems, those ports require admin rights to use. ---------------------------------------------------------------------- snorkel · 2016-12-08 22:56 Remy, I should have mentioned that error msg is for my logging purposes, so port 40913 is the peer port in this case. i'm gettting the error message from the OnException event and in that event I do this for the log:  if assigned(AContext) then           begin                ip:=acontext.Binding.PeerIP;                port:=acontext.Binding.PeerPort;           end;  Then I push the error message into a queue: push_log_msg(format('FTP server error:(%s:%d) %s',[ip,port,AException.Message]),logonid,Error); I looked back in the log and that port already in use error was happening before I added the server bindings for port 10021 and 21, was just coincidence that I noticed it because more users are hitting the server now. The connections coming in are coming from a mainframe and I don't know if the data connections are passive or not. I can't reproduce it when I run the server in the IDE, so no call stack. ---------------------------------------------------------------------- Remy Lebeau · 2016-12-09 17:52 Quote from: snorkel on December 08, 2016, 11:56:12 pmi'm gettting the error message from the OnException event That is odd, because the OnException event is fired only for uncaught exceptions that are fatal to clients (ie, the exception will disconnect the client).  The only socket bindings that a client should be performing is when opening transfer sockets, and any exceptions raised during that process are caught (so errors can be sent back to the client), they shouldn't be firing the OnException event.  I would really need to see the call stack of the exceptions to know what is going on. ---------------------------------------------------------------------- snorkel · 2016-12-12 15:24 I thought the onexception event was the global exception handler for the server? Anyways, what I think is happening is the mainframe is sending each file in a separate connection, i.e. they logon, upload a file and then disconnect and do that for each file they have to send.  Why they can't just logon once and do multiple files under a single logon is beyond me :-) But because of the way they are sending them and the way the Indy FTP server starts at the max passive port for each upload the separate uploads going concurrently must be stepping on each other occasionally, it must be getting a port that is in a timed wait state or something and then when it tries to use it raises that error. I don't know for sure but that seems like what is happening. I wonder if it would be better to grab a random starting point from the passive port range, or maybe from the max to the mid or something like that instead of doing a down to loop starting at max for each upload and connection. ---------------------------------------------------------------------- snorkel · 2016-12-12 17:02 Remy, I got this back from the mainframe folks: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 150 File status okay; about to open data connection.                            EZA2589E Connection to server interrupted or timed out. Waiting for data connectEZA1636I *** I can't open a data-transfer connection:                         426 Data connection closed abnormally. EZA1735I Std Return Code = 27426, Error Code = 00009         This is the error they saw when the binding error occurred on the server. ---------------------------------------------------------------------- Cyrax · 2016-12-12 17:06 Sounds like problem with FTP passive ports and their ranges. They should check their firewall and allow FTP passive ports. ---------------------------------------------------------------------- Remy Lebeau · 2016-12-13 01:55 Quote from: snorkel on December 12, 2016, 04:24:57 pmI thought the onexception event was the global exception handler for the server? It is a server event for uncaught exceptions that terminate client threads on a per-client basis.  The event provides the TIdContext belonging to the client thread that is raising the reported exception. Quote from: snorkel on December 12, 2016, 04:24:57 pmBut because of the way they are sending them and the way the Indy FTP server starts at the max passive port for each upload the separate uploads going concurrently must be stepping on each other occasionally, it must be getting a port that is in a timed wait state or something and then when it tries to use it raises that error. I still don't see how that could cause the error you are seeing.  Even if the ENTIRE passive port range were in TIME_WAIT at the time of a new transfer, a bind error during a new transfer should be caught by the invoking command handler (PASV, etc) and not bubble up to the thread that is managing the client connection.  In case the ENTIRE passive port rage is in TAIME_WAIT, I wonder if using the TIdFTPServer.OnDataPortBeforeBind event to enable the ReuseSocket property (ASender.DataChannel.FDataChannel.Socket.ReuseSocket := rsTrue) on each new data transfer socket would help. Quote from: snorkel on December 12, 2016, 04:24:57 pmI don't know for sure but that seems like what is happening. I can't answer that without seeing the call stack of the exception on the server side.  Can't you please capture that? Quote from: snorkel on December 12, 2016, 04:24:57 pmI wonder if it would be better to grab a random starting point from the passive port range, or maybe from the max to the mid or something like that instead of doing a down to loop starting at max for each upload and connection. Adding some element of randomness would probably help.  But it still needs to loop through the whole range in the end, and it is more overhead to loop through the entire range using random indexes while avoiding duplicates during the looping.  I'll have to figure out an algorithm to do it. Quote from: snorkel on December 12, 2016, 06:02:51 pmI got this back from the mainframe folks: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 150 File status okay; about to open data connection.                            EZA2589E Connection to server interrupted or timed out. Waiting for data connectEZA1636I *** I can't open a data-transfer connection:                         426 Data connection closed abnormally. EZA1735I Std Return Code = 27426, Error Code = 00009         This is the error they saw when the binding error occurred on the server.                                        That is not very helpful without seeing the exact FTP commands and replies leading up to the error.  TIdFTPServer uses 426 for any unexpected error that happens during a data transfer, so there is still a number of things that could be failing to cause that error. ---------------------------------------------------------------------- snorkel · 2016-12-14 15:04 "In case the ENTIRE passive port rage is in TAIME_WAIT, I wonder if using the TIdFTPServer.OnDataPortBeforeBind event to enable the ReuseSocket property (ASender.DataChannel.FDataChannel.Socket.ReuseSocket := rsTrue) on each new data transfer socket would help." That sounds like it might be worth trying. FDatachannel is protected, so currently not accessible, I can add a property to test this out. EDIT:  nope that made it much worse.  Tons of errors during file uploads. ---------------------------------------------------------------------- Remy Lebeau · 2016-12-14 18:46 Quote from: snorkel on December 14, 2016, 04:04:43 pmFDatachannel is protected, so currently not accessible, I can add a property to test this out. Yes, I know it is protected.  A property is not needed, an accessor class will also work. ---------------------------------------------------------------------- snorkel · 2017-01-04 18:10 Remy, This is still any issue and I tracked it down to procedure TIdSocketHandle.Bind in IdSocketHandle. I noticed there is a comment in there about socket in use errors and it has a IDEF for Linux64 and I am compiling for 64bit just not linux. would it hurt to enable the REUSEPORT? ---------------------------------------------------------------------- Remy Lebeau · 2017-01-04 20:26 Quote from: snorkel on January 04, 2017, 07:10:16 pmwould it hurt to enable the REUSEPORT? Windows does not support SO_REUSEPORT.  If you enable that code on Windows, TIdSocketHandle.SetSockOpt() will end up raising a socket error exception. ---------------------------------------------------------------------- snorkel · 2017-01-04 22:07 Yep, I found that out with a bit of googling :-) I don't think the error is coming from the passive data connections but from new client connections and probably related to the SO_REUSEADDR being used on all binds. Shouldn't the SO_REUSEADDR just be used on the listening sockets for the server? It appears that the FTP server  applies SO_REUSEADDR to all sockets if the reuseSocket param is set to rsTrue? Maybe I should try setting it to rsFalse and see what happens.  The worst that could happen is the main server port might not be available for a few minutes right if I shut it down and then start it back up right away. ---------------------------------------------------------------------- snorkel · 2017-01-04 22:38 ok, what I did as a test was to disable socket reuse in the ftp server properties and then do this in the bind procedure in IDSocketHandle:   if (self.Port = 21) or (port = 10021) then      SetSockOpt(Id_SOL_SOCKET, Id_SO_REUSEADDR, Id_SO_True) else   SetSockOpt(Id_SOL_SOCKET, Id_SO_REUSEADDR, LValue); So if the port is one of the listening ports I enable the socket reuse so the service can start up again fast and then all other binds have it disabled via the property setting. The server is not that busy that I would exhaust ports that are not in a time_wait state. ---------------------------------------------------------------------- Remy Lebeau · 2017-01-05 00:21 Quote from: snorkel on January 04, 2017, 11:07:46 pmI don't think the error is coming from the passive data connections but from new client connections bind() is not called on new client connections, only on data transfer connections. Quote from: snorkel on January 04, 2017, 11:07:46 pmand probably related to the SO_REUSEADDR being used on all binds. SO_REUSEADDR is only enabled if: 1. ReuseSocket is rsTrue. 2. ReuseSocket is rsOSDependent (the default) and GOSType is otUnix (which it is not in your case). ENABLING SO_REUSEADDR would not be causing your bind errors. DISABLING it might. Quote from: snorkel on January 04, 2017, 11:07:46 pmShouldn't the SO_REUSEADDR just be used on the listening sockets for the server? In FTP, bind() is used for both listening and connecting sockets.  Inbound passive-mode transfer sockets are bound to the passive port range, or to the server's DefaultDataPort if no range is specified.  Outbound active-mode transfer sockets are bound to the DefaultDataPort. Quote from: snorkel on January 04, 2017, 11:07:46 pmIt appears that the FTP server  applies SO_REUSEADDR to all sockets if the reuseSocket param is set to rsTrue? Yes (on a per-socket basis). Quote from: snorkel on January 04, 2017, 11:07:46 pmMaybe I should try setting it to rsFalse and see what happens. It was not set to rsTrue by default to begin with.  So you had to have set it to rsTrue at some point. Quote from: snorkel on January 04, 2017, 11:07:46 pmThe worst that could happen is the main server port might not be available for a few minutes right if I shut it down and then start it back up right away. Or, the server ports might not be available for a few minutes after each data transfer. Quote from: snorkel on January 04, 2017, 11:38:10 pmok, what I did as a test was to disable socket reuse in the ftp server properties and then do this in the bind procedure in IDSocketHandle: You should not be modifying Indy's source code like that.  TIdFTPServer has published OnBeforeBind and OnDataPortBeforeBind events.  The first event is for the main listening sockets that clients connect to for commands, the second event is for the data transfer sockets. ---------------------------------------------------------------------- snorkel · 2017-01-05 16:03 Hi Remy, Thanks for the Advice :-) yes, you are correct, I must have set the ReuseSockets to true at some point.  I don't remember why I enabled it, it was almost a year ago. I was already in the source code, so I just added that as a quick test, I already modified my copies source code by adding after put and after get events. The mainframe people already sent their files for the morning run with that quick mod and no bind errors, granted it was happening like once a week. I will use the events if this works.  because the bind error happens so rarely I was just guessing that it must be picking up a socket that is in a wait state, but still connected and I was reading that if SO_REUSEADDR is set it is possible to get a socket that is in that wait state and still connected and hence the error address and port already in use. There is also a SFTP server running on this particular server plus lots of Python scripts that are using Paramiko SFTP client to upload and get files.  It is definitely called when the listening sockets are created, so now with that slight mod only the listening sockets on port 21 and 10021 have  SO_REUSEADDR. ---------------------------------------------------------------------- snorkel · 2017-01-17 17:01 Just following up. Been almost two weeks and not a single socket binding error since I made that change to only do the socket reuse for the listening sockets. I don't remember why I enabled the socket reuse in the first place, but I am guessing it's because the service would not start up right way if it was restarted because the listening socket was still in a wait state.