1000+ threads and Synapse or Indy = VERY unstable on Linux 64
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 02:28

Hi all,

I try to catch this bug last 5 days and I give up... 

First of all my enviroment:

Lazarus 1.6.4 + FPC 3.0.2 (also test (with no success) with Lazarus
trunc + FPC trunc) on CentOS 6 x64
Lazarus 1.6.4 + FPC 3.0.2 on Windows 10 x64
CentOS 6.0 works in VMWare Workstation 12. Host OS is Windows 10 x64

My app needs to connect to large number of email accounts via Imap or
Pop3 protocol and download emails for processing (they analyze
feedback-loop FBL messages for several thousands domains).

So, I use Synapse's TPOP3Send + TIMAPSend in threads for login,
download and analyze emails.

On Linux64 large number of threads makes my app completely unstable !
With 50-100 threads they can work for several hours, but with 700-1000
threads they crashes in several seconds or minutes.  But it's works
fine without any problems on Win32/Win64 !

App craches (randomly) with SIGSEGV or ERangeError or EAccessViolation
or EObjectCheck in folowing units/routines:

- inside netdb unit in name-to-ip resolving routines (various AV and
RangeErrors).
- inside synapse's blocksock.pas in InternalCanRead or
InternalCanWrite (EObjectCheck exception)
- also, they can affect database connection (sqlite via ZEOSdbo) -
sqlite Disk I/O errors.

So, looks like this bug randomly affects file and/or socket handles in
random places. Looks like valid handles (randomly) becomes invalid if
number of threads is more than few houndreds.

Well, I was think it's probably Synapse bug, so I re-write my classes
to use Indy insted Synapse. But... same problems.

It's should be FPC bug or some CentOS limitation about handles or
sockets (I was try to increase various OS limits with no success).

I realy hope anybody can give me some workaround (cthreads is first
used unit of course). I MUST make it works under Linux )
Maybe some FPC or OS setting must be set ?

I write minimal example project to reproduce bug (see attachemt). Here
is their code:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i unit
Main; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, FileUtil,
Forms, Controls, Graphics, Dialogs, StdCtrls,  pop3send, imapsend,
smtpsend, dnssend; type   { TMainFrm }  TMainFrm = class(TForm)   
CancelBtn: TButton;    StartBtn: TButton;    Label1: TLabel;   
ReqLbl: TLabel;    procedure CancelBtnClick(Sender: TObject);   
procedure StartBtnClick(Sender: TObject);  private    ThreadsClosed:
Integer;    procedure OnThreadTerminated(Sender: TObject);  public   
{ public declarations }  end;  TTestThread = class(TThread)private   
ReqDone:              Integer;    Pop3:                 TPOP3Send;   
Imap:                 TIMAPSend;    Smtp:                 TSMTPSend; 
 Dns:                  TDNSSend;    DnsResult:          
 TStringList;    procedure             ResolveMX;    procedure       
    ResolveA;    procedure             TestPop3;    procedure        
   TestImap;    procedure             TestSmtp;    procedure         
  UpdateLabel;protected    procedure             Execute;
override;public    constructor          
Create(CreateSuspended:Boolean);    destructor            Destroy;
override;end; var  MainFrm: TMainFrm; implementation uses
ssl_openssl; {$R *.lfm} { TMainFrm } procedure
TMainFrm.StartBtnClick(Sender: TObject);var  i:integer;  tmpThread:TTe
stThread;beginStartBtn.Enabled:=false;CancelBtn.Enabled:=true;ThreadsC
losed:=0;self.Tag:=0;for i:=0 to 1499 do  begin 
tmpThread:=TTestThread.Create(true); 
tmpThread.FreeOnTerminate:=true; 
tmpThread.OnTerminate:=@OnThreadTerminated;  tmpThread.Start; 
end;end; procedure TMainFrm.CancelBtnClick(Sender: TObject);beginself.
Tag:=1;CancelBtn.Enabled:=false;end;//--------------------------------
---------------------------------------------- procedure
TMainFrm.OnThreadTerminated(Sender:
TObject);beginInc(ThreadsClosed);if ThreadsClosed=1500 then  begin 
ShowMessage('All threads terminated');  StartBtn.Enabled:=true;  end;e
nd;//-----------------------------------------------------------------
------------- constructor TTestThread.Create(CreateSuspended:Boolean);
beginReqDone:=0;Pop3:=TPOP3Send.Create;Pop3.Timeout:=10000;Pop3.Sock.C
onnectionTimeout:=10000; Imap:=TIMAPSend.Create;Imap.Timeout:=10000;Im
ap.Sock.ConnectionTimeout:=10000; Smtp:=TSMTPSend.Create;Smtp.Timeout:
=10000;Smtp.Sock.ConnectionTimeout:=10000; Dns:=TDNSSend.Create;Dns.Ti
meout:=10000;Dns.TargetHost:='8.8.8.8';Dns.TargetPort:='53'; DnsResult
:=TStringList.Create; Inherited Create(CreateSuspended);end;//--------
----------------------------------------------------------------------
 destructor TTestThread.Destroy;beginPop3.Free;Imap.Free;Smtp.Free;Dns
.Free;DnsResult.Free;Inherited Destroy;end;//-------------------------
----------------------------------------------------- procedure
TTestThread.Execute;beginwhile not Terminated do  begin  ResolveMX; 
ResolveA;  TestPop3;  TestImap;  TestSmtp; 
Synchronize(@UpdateLabel);  end;end;//--------------------------------
---------------------------------------------- procedure
TTestThread.ResolveMX;beginif Terminated then  Exit;DnsResult.Clear;if
Dns.DNSQuery('gmail.com',QTYPE_MX,DnsResult) then  Inc(ReqDone);end;//
----------------------------------------------------------------------
-------- procedure TTestThread.ResolveA;beginif Terminated then 
Exit;DnsResult.Clear;if
Dns.DNSQuery('pop.gmail.com',QTYPE_A,DnsResult) then  Inc(ReqDone);if
Dns.DNSQuery('imap.gmail.com',QTYPE_A,DnsResult) then  Inc(ReqDone);en
d;//------------------------------------------------------------------
------------ procedure TTestThread.TestPop3;beginif Terminated then 
Exit;Pop3.TargetHost:='pop3.any-test-server.com';if Random(2)=0 then 
begin  Pop3.TargetPort:='995'; // test ssl  Pop3.FullSSL:=true; 
Pop3.AutoTLS:=false;  endelse  begin  Pop3.TargetPort:='110'; // no
ssl  Pop3.FullSSL:=false;  Pop3.AutoTLS:=false; 
end;Pop3.UserName:='no-matter@server.com';Pop3.Password:='no-matter-pa
ssword';Pop3.Login;Pop3.Logout;Inc(ReqDone);end;//--------------------
---------------------------------------------------------- procedure
TTestThread.TestImap;beginif Terminated then 
Exit;Imap.TargetHost:='imap.any-test-server.com';if Random(2)=0 then 
begin  Imap.TargetPort:='993';   // test ssl  Imap.FullSSL:=true; 
Imap.AutoTLS:=false;  endelse  begin  Imap.TargetPort:='143';    // no
ssl  Imap.FullSSL:=false;  Imap.AutoTLS:=false; 
end;Imap.UserName:='no-matter@server.com';Imap.Password:='no-matter-pa
ssword';Imap.Login;Imap.Logout;Inc(ReqDone);end;//--------------------
---------------------------------------------------------- procedure
TTestThread.TestSmtp;beginif Terminated then 
Exit;Smtp.TargetHost:='smtp.any-test-server.com';if Random(2)=0 then 
begin  Smtp.TargetPort:='465';  // test ssl  Smtp.FullSSL:=true; 
Smtp.AutoTLS:=false;  endelse  begin  Smtp.TargetPort:='25';   // no
ssl  Smtp.FullSSL:=false;  Smtp.AutoTLS:=false; 
end;Smtp.UserName:='no-matter@server.com';Smtp.Password:='no-matter-pa
ssword';Smtp.Login;Smtp.Logout;Inc(ReqDone);end;//--------------------
---------------------------------------------------------- procedure T
TestThread.UpdateLabel;beginMainFrm.ReqLbl.Caption:=IntToStr(StrToInt(
MainFrm.ReqLbl.Caption)+ReqDone);ReqDone:=0;if MainFrm.Tag=1 then  Ter
minate;end;//---------------------------------------------------------
---------------------end.
On Win64 they works for hours with no problems.
On Linux64 they chrashes after several seconds or minutes.

UPD: same problems on CentOS 7 x64... 

UPD2: exceptions in netdb unit happens if Synapse needs to resolve
host hame to IP e.g. if Pop3.TargetHost:='pop3.any-test-server.com'.
Of course I try to test with IP address instead of host name to avoid
resolving in netdb unit. In this case I got EObjectCheck exception in
blocksock.pas in InternalCanRead or InternalCanWrite routines after
line x := synsock.Select(FSocket + 1, @FDSet, nil, nil, TimeVal);

----------------------------------------------------------------------
Blestan · 2017-08-07 05:18

under linux the problem is the number of threads not fpc or synapse...
i remember several years ago the max thread count per process was
373... any. way its insane to have more threads than cores... and you
can use async io... pop3 and dns resolv is ideal for async...
Modify message
Report to

----------------------------------------------------------------------
AmatCoder · 2017-08-07 06:34

Linux has a limit on the total number of processes (or threads) on the
system.
To see this limit:

Code: [Select]$ cat /proc/sys/kernel/threads-max
Here on Arch Linux gives me 63482.
But I think that on Centos 6/7 the default limit is 1024.

Edit: Increase CentOS 6.x Thread Limit

----------------------------------------------------------------------
Thaddy · 2017-08-07 06:46

Ah, that gives me 14629 of Raspbian

Anyway: indeed so many threads is not a good idea. It is also
necessary to fine tune the resources that a thread uses and keep it to
a minimum (stacksize, tls) will definitely help!
Also note that I prefer to use forking (fibers) over threading (or a
combination of both) with so many connections and dedicate a core to a
thread if that thread performs heavy processing. In general the
instability is caused by hardware limitations or resource limitations,
not by the software perse. You will experience the same issues with
software written in other languages if you do not manage the thread
resources efficiently. That also goes for the threads themselves: use
a thread pool.

----------------------------------------------------------------------
SymbolicFrank · 2017-08-07 06:57

There are two different kinds of threads: blocked threads waiting on
I/O, and active threads. If you have Windows 10, you can see in the
Task Manager how many thousands of threads are currently active.

On Linux you need to use unit cthreads, and probably cmem as well.

Edit: the main problem is likely this:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
Synchronize(@UpdateLabel); 
If you have many threads, this single line essentially serializes your
application. If there are too many active (ie. not blocked), bad
things will happen.

----------------------------------------------------------------------
Blestan · 2017-08-07 07:04

@thaddy:
QuoteAlso note that I prefer to use forking (fibers) over threading
(or a combination of both)
im looking for a crossplatform fiber implementation for years ... do
you have any viable working code?

----------------------------------------------------------------------
Thaddy · 2017-08-07 07:16

No. On windows I use the fiber api. On unix I sometimes use forks if
necessary, although threads are much less of an issue compared to
Windows.

Note that this code:Code: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i MainFrm.ReqLbl.Caption:=IntToStr(StrToInt(MainFrm.ReqLbl.Captio
n)+ReqDone); is an absolute howler, of course.
Calling synchronize on that will cause problems. Why not use a class
var integer or global var integer and use
InterlockedIncrement/InterlockedDecrement.
You can then update the label in the Application.OnIdle event. No need
to do all that string conversions and call synchronize.
Note that on most platforms an integer is atomic anyway, so the
counter can probably be read straight from the TThread class var w/o
syncing.

----------------------------------------------------------------------
Blestan · 2017-08-07 07:29

@thaddy: yes on windows the fiber api is good and pretty easy to use
just convert the main thread to fiber ... the best part is that I/O
Completion Ports can be used with fibers and this is a very powerful
async io approach atleast as fast as epoll and kqueue on u*x
btw: the code that you mentioned is awesome   -  strtoint is my
favorite

----------------------------------------------------------------------
Thaddy · 2017-08-07 08:50

Quote from: Blestan on August 07, 2017, 09:29:26 am@thaddy: yes on
windows the fiber api is good and pretty easy to use just convert the
main thread to fiber ... the best part is that I/O Completion Ports
can be used with fibers and this is a very powerful async io approach
atleast as fast as epoll and kqueue on u*x
btw: the code that you mentioned is awesome   -  strtoint is my
favorite 

Not mine  , nowadays I would write Code: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i with
MainFrm.ReqLbl do Caption := (Caption.ToInteger+ReqDone).ToString;   .
Just as slow of course, and should also not be used with
Synchronize....

----------------------------------------------------------------------
SymbolicFrank · 2017-08-07 09:32

Using PostMessage would work to update the GUI, but as there are
already lots of server threads, it might be best not to show it in the
GUI and have another server thread handle the bookkeeping.

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 09:35

QuoteLinux has a limit on the total number of processes (or threads)
on the system.
cat /proc/sys/kernel/threads-max reports 29971 on my CentOS6, so it's
not thread limit problem.

QuoteAlso note that I prefer to use forking (fibers) over threading
(or a combination of both) with so many connections and dedicate a
core to a thread if that thread performs heavy processing. In general
the instability is caused by hardware limitations or resource
limitations, not by the software perse.
But it's works with no problem on Win10 x64 on SAME hardawre ! My
CentOS runs under VMWare workstation.

QuoteOn Linux you need to use unit cthreads, and probably cmem as
well.
Both used in project .lpr file. First is cthreads, then cmem.

Quotethe main problem is likely this: Synchronize(@UpdateLabel);
No. It's used just to see is example program alive/working. I not have
any Syncronize in my real code. Also, commenting this line not solve
problem.

QuoteCalling synchronize on that will cause problems. Why not use a
class var integer or global var integer and use
InterlockedIncrement/InterlockedDecrement.
No, see above. Commenting Synchronize line not solve problem.

But after another night of testing I found interesting things: looks
like problem happens if DNSSend (udp based) and Imap/Pop3/Smtp send
(tcp based) lives in same thread !

If comment out Resolve functions - they works with no problems for
hours:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i while not
Terminated do  begin  //ResolveMX;  //ResolveA;  TestPop3;          //
working with no problems  TestImap;  TestSmtp; 
//Synchronize(@UpdateLabel);  end;end;
If uncomment ResolveMX, ResloveA but comment TestPop3, TestImap,
TestSmtp - they again working for hours:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i while not
Terminated do  begin  ResolveMX; // working with no problems 
ResolveA;  //TestPop3;            //TestImap;  //TestSmtp; 
//Synchronize(@UpdateLabel);  end;end;
But if ResloveMX and any of TestPop3, TestImap, TestSmtp works in
single threads - they hangs in serveral seconds:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i while not
Terminated do  begin  ResolveMX;  //ResolveA;  TestPop3;           //
this buggy  TestImap;  TestSmtp;  //Synchronize(@UpdateLabel); 
end;end;
It's looks like some interference between udp and tcp sockets in same
thread ! Or maybe some additional synchronization required somewhere
inside Synapse code. I will also try to write to synapse mailing
list...

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 09:41

QuoteUsing PostMessage would work to update the GUI, but as there are
already lots of server threads, it might be best not to show it in the
GUI and have another server thread handle the bookkeeping.
Problem is not realted to Syncrhonize and GUI update. It's just to see
is example program working. Commenting this line not solve problem.
Also, test routines try to connect to non-existent servers pop3.any-
test-server.com this relatively slow and Syncrhonize calls not to
frequently. Of course with real servers I have same problems. During
days of testing I try almost all possible (and impossible) things )))

----------------------------------------------------------------------
Blestan · 2017-08-07 10:01

first check all the parameters for thread creation on both platform
... may be you are allocating too much memory for stack tls ect for
each thread on linux ...
secord try a non threaded solution - use lnet for async socket io
third update the gui only on onidle event to avoid syncronize

----------------------------------------------------------------------
AmatCoder · 2017-08-07 10:12

Quote from: Max V. Terentiev on August 07, 2017, 11:35:27 amQuoteLinux
has a limit on the total number of processes (or threads) on the
system.
cat /proc/sys/kernel/threads-max reports 29971 on my CentOS6, so it's
not thread limit problem.

On CentOS6, the file /etc/security/limits.d/90-nproc.conf has a soft
limit (soft nproc 1024) that override that setting.

I'll put again this link: Increase CentOS 6.x Thread Limit

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 10:13

Quotefirst check all the parameters for thread creation on both
platform ... may be you are allocating too much memory for stack tls
ect for each thread on linux ...
Default paramters on both platforms (OS default stack size for
threads). Because I use x64 OS and I have 32 gb ram - it's must be not
problem.

Quotesecord try a non threaded solution - use lnet for async socket io
Lnet not contains needed DNS components. I can write it my self by
converting Synapse code but I not sure it's helps.

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 10:32

QuoteOn CentOS6, the file /etc/security/limits.d/90-nproc.conf has a
soft limit (soft nproc 1024) that override that setting.
I was 4096 here. Changed to 65535. But it's not helps (

----------------------------------------------------------------------
Blestan · 2017-08-07 10:58

look anyway the problem is not fpc/lazarus or synapse/indy releated
the problem is the multi-threading support on linux ...
try to make a non  socket program in c/c++ with 1000+ threads and you
will get the same craches ...
1000+ threads on 8 core cpu is insane and counter productive

----------------------------------------------------------------------
SymbolicFrank · 2017-08-07 11:19

Unless all those threads are listening on a socket in your full
application, I agree: just do it serial. Much faster as well.

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 11:48

Quote1000+ threads on 8 core cpu is insane and counter productive
Of course I not use 1000+ threads in my app ! I use only 100-150.

But. This problem may happens on 100-150 threads after several hours.

So, to find and fix bug I write test app with 1000+ threads where
problem happens in several seconds. It's just to avoid waiting for
hours during debugging )

Anyway, with 1000 or 5000 threads apps must works stable if OS have
enouch resources. App may work very slow or ineffective, use to much
memory, etc but it's MUST works stable, right ? )

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 11:51

Quotelook anyway the problem is not fpc/lazarus or synapse/indy
releated the problem is the multi-threading support on linux ...
You mean multi-threading support in Linux OS ? Or multi-threading
support in FPC->Linux targets ?

OS should not have any problems with threads. For example, nginx can
serve thousands requests with no problems.

----------------------------------------------------------------------
Blestan · 2017-08-07 11:59

i repeat for the 10th time this is NOT fpc problem!!!!!
nginx is single threaded async io based and does not use more than the
main thread     Please read.alttle bit more about socket
programing!!!!!!

----------------------------------------------------------------------
Max V. Terentiev · 2017-08-07 12:04

Ok, how many threads is safe and 100% stable on Linux ?

----------------------------------------------------------------------
Blestan · 2017-08-07 12:18

im using 8 or 16  depending the core... also tested up to 64 just for
the benchmarks ... but my code is very tight not even doing memory
allocation and string creation... every thing is using mem pools per
thread.i get under 2ms per simple http request/reponse so for 8 cores
it runs at 5-20k requests

----------------------------------------------------------------------
SymbolicFrank · 2017-08-07 12:21

The optimal amount of active threads (ie. threads that are running
code) is twice the amount of cores. Three threads running at full
speed on each core will make the computer unresponsive until it
crashes.

That is not OS specific, it is purely hardware related.

----------------------------------------------------------------------
SymbolicFrank · 2017-08-09 19:56

Ok, to explain this a bit more:

1. It is very hard to get threads to run at full speed, because
they're probably waiting a long time to read or write stuff to and
from the slow memory.
2. Intel tells us, that hyper-threading effectively doubles the amount
of threads you can run.
3. The Windows threading model and the *nix one are vastly different.

A long time ago (in a galaxy far away?) we had Unix, which was multi-
user and multi-tasking, and Windows 95, which only pretended to be
those things. And, starting a process in Windows takes a long time.
So, Microsoft went the way of the least resistance, and made it
possible to just start threads at random, without any support
structure whatsoever. "Now we are multi-whatever as well!!!"

The *nix guys had to laugh at that: they passed that stage more than
twenty years ago. It wouldn't work. You need separation and messages
to make it work. And they were right.

Then again, when Microsoft spends many millions on promoting it, who
are you going to believe? Those old nerds? Nah, of course not!
Microsoft is the winning team! Jump on the bandwagon!

So, for twenty years, people believed Microsoft, and tried to get it
to work. And if they succeeded, profiling was not the way to go in
most cases, as it showed that their multi-threaded application was
actually slower than the single-threaded one. WTF!!!

And, of course, Linux now has threads and all the other crap as well:
we have to copy the winning team!


Nowadays, most programmers who use multi-whatever a lot, have taken a
step back and they don't use most of the things Microsoft tells them
to. Like, shared memory. Or keeping pointers to other threads. Or,
actually the whole programming model as they learned it on school.
Because, it doesn't work!

Then again, how do you do it, in a way that it actually works? Can I
get an education for that? Not yet, probably. Or, not any more.


So, when I made a fast, working, multi-platform threading library a
few years ago (in C++, unfortunately), I did have to figure it all out
myself. And test it as well. And I can state, that my testing showed,
that both on ARM and Intel processors, in multiple variants and on
multiple platforms (Windows, Linux, OS-X, iOS and Android), two
threads for each core, that are designed to run at full speed (they
use very little memory) are fine, while three will crash the whole box
after a while.

But, as stated, it is very hard to make useful threads/processes that
do more useful things than just benchmarking, that run at full speed.
Most of those threads/processes spend the majority of their time
idle/suspended/waiting for memory or I/O.