"Impossible to create epoll, too many open files" error and lNet
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
washburn_it · 2021-01-05 00:13

Hi,

some year ago I developed a program to control my fish tank with a
Raspberry Pi...it controls water temperature, pH, lights.
Some days ago I implemented also to receive power consumption data of
my house from a 2nd Raspberry that is connected to a SDM120C.
Those data are displayed to the "fish tank" Raspberry through a 7" LCD
display.
Data are received through a TCP communication (using lNet components)
where the 2nd raspberry is the TCP Server and the "fish tank"
raspberry is the TCP client, data are shown on request (by clicking on
a button on the main Form) where a TTimer is activated and every 10
seconds request data to the TCP server.
The "polling cycle" is the following:

- TCP Client is created
- it connects to the TCP server
- acquires data
- closes the connection
- the "TLTCP" derived object is freed.
- waits 10 seconds and the cycle start again

Now the problem....if I keep the "power" monitoring active, after a
certain time the error in the subject appears then I must close the
program and open it again or reboot the "fish tank" raspberry.
Trying to find the problem, I launched the command "ls -l
/proc/24130/fd" (where "24130" is the PID of the program) after
activating "power" monitoring and I found that there are an increasing
number of lines "anon_inode:[eventpoll]" that stops only when I stop
the TCP connection (thus the "power" monitoring).
This the code of the TCP Client:

Class definition

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Type 
TPwrThread= class    private      IP: string;      status: string;   
 datiStr: string;      porta: integer;      FCon : TLTcp; //client
TCP      procedure Scrivi2Log(str: string);      function
GetAppDirec(): string;      procedure OnDs(aSocket: TLSocket);
 //disconnessione dal server      procedure OnRe(aSocket: TLSocket);
 //ricezione da server      procedure OnEr(const msg: string; aSocket:
TLSocket); //errori      procedure ElaboraDati();      procedure
SetStatus();    protected      procedure Run;    public     
Constructor Create(IPAddr: string; port: integer);      destructor
Destroy; override;  end;  

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i constructor
TPwrThread.Create(IPAddr: string; port: integer);begin   
//costruttore classe     IP:= IPAddr;    porta:= port;     //TCP
client creation    FCon := TLTCP.Create(nil);     FCon.OnError :=
@OnEr; // assign callbacks    FCon.OnReceive := @OnRe;   
FCOn.OnDisconnect := @OnDs;    FCon.Timeout := 100; // responsive
enough, but won't hog cpu     //inherited
Create(CreateSuspended);end; destructor TPwrThread.Destroy;begin  try 
  FCon.Free; // free the connection  except    //  end;  inherited
Destroy;end; procedure TPwrThread.OnDs(aSocket: TLSocket);begin 
//Scrivi2Log('Perso connessione a server');  status:= 'non connesso'; 
frmMain.lbTCPSrv.Caption:= status; 
//Synchronize(@SetStatus); end; procedure TPwrThread.OnRe(aSocket:
TLSocket);var  s: string;begin  //ricezione dati da server  if
aSocket.GetMessage(s) > 0 then  begin    datiStr:= s;    //flDati:=
true;  end; end; procedure TPwrThread.OnEr(const msg: string; aSocket:
TLSocket);begin  //errori vari  Scrivi2Log(msg); // if error occured,
write it    frmMain.lbTCPSrv.Caption:= msg; 
//Synchronize(@SetStatus); end; procedure TPwrThread.Run;begin   
//esecuzione thread     //connetto al TCP server    if
FCon.Connect(IP, porta) then    begin      //attende connessione al
TCP server      repeat        FCon.CallAction; // wait for
"OnConnect"        Sleep(1);        Application.ProcessMessages;     
until FCon.Connected;       status:= 'connesso';     
frmMain.lbTCPSrv.Caption:='connesso';     
//Synchronize(@SetStatus);       //invio richiesta dati al TCP server 
    datiStr:='';      FCon.SendMessage('*d');  //sends the data
request to the server      FCon.CallAction; // eventize lNet loop    
//elaboro i dati ricevuti      ElaboraDati;    end;    
FCon.Disconnect(true);    FCon.Free;    //Scrivi2Log('Thread TCP
Client terminato');end;  
Main program timer to read data

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TfrmMain.tmPwrTimer(Sender: TObject);begin    //power data reading   
tmPwr.Enabled:= false;     PWR := TPwrThread.Create('192.168.1.122',
5050);    PWR.Run;    PWR.Free;     tmPwr.Enabled:= true;end; 
Data are reading in the "TPwrThread" class, basically it's a string
that contains values as voltage, current, power.
The procedure "ElaboraDati" takes the "data string" received in
"OnRe(aSocket: TLSocket)" and assigns each value to some labels to the
main Form.
What could be causing that error?
It seems that the socket or the connection is not closed and in some
way that allocates "open files" until the limit is reached, then the
error.
Thank you

Roberto

----------------------------------------------------------------------
washburn_it · 2021-01-06 16:11

I fixed the problem by myself...the problem was due to how I was using
the component, freeing it after every receiving of the data caused the
problem.
Now I "encapsulated" it into a thread and it is freed only when the
thread is terminated, leaving the connection active for the whole
running time of the thread: this fixed the problem.
Regards,


Roberto

----------------------------------------------------------------------
MarkMLl · 2021-01-06 16:48

Quote from: washburn_it on January 06, 2021, 05:11:48 pmI fixed the
problem by myself...the problem was due to how I was using the
component, freeing it after every receiving of the data caused the
problem.
Now I "encapsulated" it into a thread and it is freed only when the
thread is terminated, leaving the connection active for the whole
running time of the thread: this fixed the problem.

Sorry, didn't spot that earlier and am not going through your code
right now. I had something very similar a few weeks ago where I was
neglecting to close a UDP socket... just because it's connectionless
doesn't mean that every open doesn't have to be paired with a close.

Not sure if Lnet is suffering from the same problem... I've put work
in the past into trying to understand it and concluded it wasn't worth
the effort for the things I needed to do.

MarkMLl

----------------------------------------------------------------------
washburn_it · 2021-01-06 17:11

I didn't try the "UDP" connection but it's supported by LNet, like
also http, smtp and ftp protocols.
It was my mistake but it's pretty easy to implement a simple TCP
client/server as I was trying to do.
Consider that package in the future if you still have to implement
networking functionality to your programs.
Regards,


Roberto

----------------------------------------------------------------------
MarkMLl · 2021-01-06 17:21

Quote from: washburn_it on January 06, 2021, 06:11:32 pmI didn't try
the "UDP" connection but it's supported by LNet, like also http, smtp
and ftp protocols.
It was my mistake but it's pretty easy to implement a simple TCP
client/server as I was trying to do.
Consider that package in the future if you still have to implement
networking functionality to your programs.

I've wasted plenty of time on Lnet in the past, and have very little
intention of doing so in the future. It's only marginally supported by
its original authors, is vastly overcomplex since by and large it
predates threads, and has some very nasty failure behaviour.

As I said in a different thread a few days ago: simple jobs like TCP
streams and UDP datagrams can be handled easily enough without a
dedicated library, and while its support for Telnet is useful the
associated complexity (and nasty failures when one end or the other
drops the connection without warning) mean that a rewrite is badly
needed.

MarkMLl

----------------------------------------------------------------------
washburn_it · 2021-01-06 17:31

Thanks for the advice about LNet, I'm using it just since a week and
for a "hobbyst" purpose at home.
For my job I use different languages (mainly C#) that already
integrates libraries for almost everything.
Regards,


Roberto

----------------------------------------------------------------------
MarkMLl · 2021-01-06 17:53

Quote from: washburn_it on January 06, 2021, 06:31:50 pmThanks for the
advice about LNet, I'm using it just since a week and for a "hobbyst"
purpose at home.
For my job I use different languages (mainly C#) that already
integrates libraries for almost everything.

Remember also that there's two variants: the one on Github and the one
shipped with FPC (for the use of installation utilities). However I've
no particular reason to believe that one of those is substantially
more or less reliable than the other.

I got some patches into the latter variant perhaps five years ago to
improve the way it handled Telnet subcommands, specifically to support
a terminal emulator (i.e. the Telnet client end) that I was attempting
to connect to something with slightly quirky Telnet terminal-type
handling. I've also used it more recently to implement a Telnet server
in a couple of things for debugging purposes... /not/ the sort of
thing I'd expect a customer to see.

I've put much more time into trying to debug the Telnet code over the
last few months than it would have taken to rewrite it from scratch,
or at least get in a good foundation onto which various terminal
handlers could be layered.

As far as other libraries go: there's things like Synapse, although in
my experience once one starts doing anything that needs a custom
protocol (i.e. not HTTP, FTP etc.) one's better off going to the
underlying (Berkeley) sockets API.

MarkMLl

----------------------------------------------------------------------
MarkMLl · 2021-04-06 09:54

Quote from: MarkMLl on January 06, 2021, 06:53:35 pmI've also used it
more recently to implement a Telnet server in a couple of things for
debugging purposes... /not/ the sort of thing I'd expect a customer to
see.

I've put much more time into trying to debug the Telnet code over the
last few months than it would have taken to rewrite it from scratch,
or at least get in a good foundation onto which various terminal
handlers could be layered.

I've just come across my note regarding the Telnet problem in the
variant of Lnet bundled with FPC, I don't know whether it affects the
one on Github.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i (*  * At
console, ATTACH: SPO command issued. Port e.g. 5500 now listening.   
*)(*  * First Telnet user attaches to port 5500 successfully.        
           *)(*  * Second user attempts to attach to port 5500,
rejected.                    *)(*  * First user detaches while second
user's socket is in TIME_WAIT state.     *)(*  * There are now two
sockets in TIME_WAIT state, which confuses the server.  *)(*  * No
user can attach without a detach/attach at the main console, which   
*)(*    won't work while sockets are in the TIME_WAIT state.         
           *) 
I'll be reimplementing this for my own use at some point, based on a
separate thread which splits incoming Telnet traffic into a stream for
data and separate FIFOs for commands/results and physical status
changes. Any more complex protocols I will obviously be using
something like Synapse for, but Telnet is sufficiently useful for
programs which aren't primarily network-oriented (and sufficiently
neglected by mainstream libraries) to be treated as a special case.

MarkMLl

----------------------------------------------------------------------
washburn_it · 2021-04-06 17:09

After some months, some weeks ago, I had to stop using LNet since
every now and then the program got "frozen" so I had to close it and
launch it again.
At first I thought it was due to the power supply because sometime a
yellow "flash" appeared on the screen, then I changed it but the
program kept stopping working.
So I recalled your advice about LNet and I tried to disable the thread
that was using that component: two weeks have passed without problems.
I permanently cancelled the thread from the source code and the
component.
Now data are displayed by an ESP8266 NodeMCU (maybe you know it but,
if not, it's a micro equipped with wifi) with a 2.8" LCD.
Regards,


Roberto