Gracefully shutdown TFPHTTPServer in a thread
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
tudi_x · 2017-07-20 14:31

hi All,
please see attached and screen capture.
i am getting an exception ' Could not accept a client connection on
socket' when stopping the TFPHTTPServer server. also there are unfreed
memory blocks.
please advise what i am missing.

thank you

----------------------------------------------------------------------
Thaddy · 2017-07-20 15:04

If port 148 is not available.....

----------------------------------------------------------------------
tudi_x · 2017-07-20 15:10

i think tcp port 148 is the jargon port according to IANA.
so you are saying that socket 148 is actually port 148?

would port 148 be related to port 8001 on which i am starting the HTTP
server?

----------------------------------------------------------------------
rvk · 2017-07-20 15:24

For me it's hanging again after _T.Server.Active := False;
It seems it doesn't cancel the server.
I thought this was solved but somehow with console and all these
threads it's not working again.

(A few days ago the same code with Server.Active := false worked fine,
sighhh)

----------------------------------------------------------------------
rvk · 2017-07-20 15:31

The exception is probably correct:
10004:
The WSACancelBlockingCall function has been removed in compliance with
the Windows Sockets 2 specification, revision 2.2.0.
https://msdn.microsoft.com/en-
us/library/windows/desktop/ms741547(v=vs.85).aspx

So when you set Server.Active := false the code calls
CancelBlockingCall and that generates an exception.
You can handle that exception as if it was correct (if the code was
10004 of course).

----------------------------------------------------------------------
tudi_x · 2017-07-20 15:37

thank you rvk. i was not aware of the WSACancelBlockingCall function.

please advise on the unfreed memory options...

----------------------------------------------------------------------
rvk · 2017-07-20 15:38

And the leak is because you have fphttpapp in your http_m_listen.

When you terminate your program I don't think the Finalization-part of
fphttpapp is called and you get a leak.

You could try just commenting it out:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses 
Classes, SysUtils, {fphttpapp,} fphttpserver;

----------------------------------------------------------------------
tudi_x · 2017-07-20 15:45

i took out fphttpapp - still get the leak.

on a theoretic side - as WSACancelBlockingCall was removed from
windows sockets 2, on windows TFPHTTPServer should have been using a
different mechanism?

----------------------------------------------------------------------
rvk · 2017-07-20 15:58

Quote from: tudi_x on July 20, 2017, 05:45:39 pmi took out fphttpapp -
still get the leak.
on a theoretic side - as WSACancelBlockingCall was removed from
windows sockets 2, on windows TFPHTTPServer should have been using a
different mechanism?
Then the leak is probably due to the exception itself.

When there is an exception in the socket-code not all memory might be
freed.

What happens if you replace the sleep(500) in the main with readln.
And then run the program, wait a few seconds and press enter.

(I'm still puzzled why it's not working for me anymore in Windows.
Maybe I'll have to reinstall trunk completely again.)

----------------------------------------------------------------------
rvk · 2017-07-20 21:35

Well, I tried a fresh trunk 32 bit at home and there is also didn't
respond to the _T.Server.Active := False;
Next I tried a 64 bit laz 1.6.4 and there it worked. I also got your
exact result.
I tried some things but also couldn't figure out why you get the
exception.
The Active := false triggers a CloseSocket which results in that
exception.

Even with some simple console-code with just the THTTPServerThread
(and without your TLink-thread) it gave the same exception.
I'm sure the following leak is due to that exception.

Frankly, I find TFPHttpServer in combination with a thread and setting
Active := false quite unstable. In 32 bit trunk it doesn't even work.
My preferred goto is Synapse for communications. (I already
implemented it in my own Delphi program for Google authentication and
redirect to local)

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var  _T:
THTTPServerThread;begin  writeln('execute 1');  _T :=
THTTPServerThread.Create(8001);  writeln('execute 2');  sleep(5000); 
writeln('execute 3');  _T.Server.Active := False;  writeln('execute
4');  sleep(5000);   _T.WaitFor;  writeln('execute 5');  _T.Destroy; 
writeln('execute 6');end.
Code: [Select]execute 1
create http
execute 2
execute http
execute 3
execute 4
exception http: Could not accept a client connection on socket: 296,
error 10004
finish http
execute 5
destroy http
execute 6
Heap dump by heaptrc unit
86 memory blocks allocated : 4309/4600
84 memory blocks freed     : 4133/4424
2 unfreed memory blocks : 176
True heap size : 196608
True free heap : 196384
Should be : 196176
Call trace for block $00000000001307D0 size 24
  $000000010000A702
  $00000001000084F7
  $0000000100031F86
  $0000000100032603
  $0000000100032DE0
  $0000000100032C43
  $0000000100031711
Call trace for block $0000000000126550 size 152
  $000000010000A702
  $00000001000084F7
  $0000000100032C0B
  $0000000100031711
  $00000001000312FE
  $000000010002DB4C line 59 of http_m_listen.pas
  $000000010001A49D

----------------------------------------------------------------------
Josh · 2017-07-20 21:55

Hi
Not sure if it relates to this scenario, but I recently had odd issue
with trunk using gdb on win 7 64 bit running 32bit app in ide, running
64bit app in ide was fine. My issue was not with threads, if I run the
32bit app in ide without debugger it worked fine.

Its purely a shot in the dark, probably nothing to do with your issue,
but thought i would mention just in case.

----------------------------------------------------------------------
rvk · 2017-07-20 21:59

Not a bad shot. But in this case, because it was a console program, I
compiled it in Lazarus IDE and started it in a command prompt. So
GDB.exe was not used in my case.

Thanks for thinking along.

----------------------------------------------------------------------
Josh · 2017-07-20 22:01

ok, was wondering if heaptrc maybe exhibiting the same, does the
console app throw an exception without heaptrc.

A big shot in dark, and my last odd thought of the day.

----------------------------------------------------------------------
rvk · 2017-07-20 22:04

Quote from: josh on July 21, 2017, 12:01:47 amok, was wondering if
heaptrc maybe exhibiting the same, does the console app throw an
exception without heaptrc.Yup, the exception is still there without
the heaptrc. (So probably the leak too)

Code: [Select]execute 1
create http
execute 2
execute http
execute 3
execute 4
exception http: Could not accept a client connection on socket: 288,
error 10004
finish http
execute 5
destroy http
execute 6

----------------------------------------------------------------------
Josh · 2017-07-20 22:24

I have downloaded the code, but cannot get it work past free resource
api,
I was going to try using cmem,

----------------------------------------------------------------------
rvk · 2017-07-20 22:30

Quote from: josh on July 21, 2017, 12:24:15 amI have downloaded the
code, but cannot get it work past free resource api,
Yep, that was my problem too with a fresh trunk 32 bit install.
Somehow the Server.Active := false doesn't work and the server isn't
stopped.
That was also a issue some time ago in another topic.

But in Laz 1.6.4 64 bit it worked for me here at home.

----------------------------------------------------------------------
rvk · 2017-07-20 22:37

With cmem the leak does seem to be resolved.
(no idea why)

Code: [Select]execute 1
create http
execute 2
execute http
execute 3
execute 4
exception http: Could not accept a client connection on socket: 288,
error 10004
finish http
execute 5
destroy http
execute 6
Heap dump by heaptrc unit
0 memory blocks allocated : 0/0
0 memory blocks freed     : 0/0
0 unfreed memory blocks : 0
True heap size : 65536 (224 used in System startup)
True free heap : 65312
The hang after "free resources api" on a 32 bit trunk isn't resolved
with that.

----------------------------------------------------------------------
Josh · 2017-07-20 22:38

I set active to false, and added cmem to all files.
I get this result, what should I be getting.

Code: [Select]Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\josh>t:

T:\>api
main thread
create api
execute api
create http
execute http
finish http
destroy api
free resources api
destroy http

Heap dump by heaptrc unit
0 memory blocks allocated : 0/0
0 memory blocks freed     : 0/0
0 unfreed memory blocks : 0
True heap size : 65536 (64 used in System startup)
True free heap : 65472

T:\>

----------------------------------------------------------------------
rvk · 2017-07-20 22:46

Quote from: josh on July 21, 2017, 12:38:30 amI set active to false,
and added cmem.
I get this result, what should I be getting.
In http_m_listen you should leave the Server.Active := True;
It activates the server.
After that the sleep(500) is executed in the main program.
After that t.Destroy is called in which _T.Server.Active is set to
false.
In the http_m_listen the action should continue after Server.Active :=
true (because that was "blocking")

For me on 32 bit trunk _T.Server.Active := False; doesn't do anything
and the program hangs there waiting for the thread in http_m_listen to
continue.
On 64 bit the _T.Server.Active := False in api.pas does call
closesocket (so the code continues after Server.Active := true) but
closesocket generates an exception.

So even with cmem with laz 1.6.4 64 bit (in the project in opening
post) you get the exception:
(but no more leak)
Code: [Select]main thread
create api
execute api
create http
execute http
destroy api
free resources api
exception http: Could not accept a client connection on socket: 300,
error 10004
finish http
destroy http

Heap dump by heaptrc unit
0 memory blocks allocated : 0/0
0 memory blocks freed     : 0/0
0 unfreed memory blocks : 0
True heap size : 65536 (224 used in System startup)
True free heap : 65312

----------------------------------------------------------------------
molly · 2017-07-21 00:01

Quote from: rvk on July 20, 2017, 05:31:37 pmThe exception is probably
correct:
10004:
The WSACancelBlockingCall function has been removed in compliance with
the Windows Sockets 2 specification, revision 2.2.0.
https://msdn.microsoft.com/en-
us/library/windows/desktop/ms741547(v=vs.85).aspx

So when you set Server.Active := false the code calls
CancelBlockingCall and that generates an exception.
You can handle that exception as if it was correct (if the code was
10004 of course).
I'm in limbo on that assessment, because the winxp32 i tested:

Code: [Select]File type: WINDOWS EXECUTABLE

File Header:
  Machine:                       0x014C (Intel x86)
  Number of Sections:            4
  Time Date Stamp:               0x411096F2 (4-8-2004 9:57:38)
  Pointer to Symbol Table:       0x00000000
  Number of Symbols:             0
  Size of Optional Header:       0x00E0 (224)
  Characteristics:               0x210E
    File is executable.
    Line numbers stripped from file.
    Local symbols stripped from file.
    32 bit word machine.
    File is a DLL.

Optional Header:
  Magic:                         0x010B
  Linker Version:                7.10
  Size of Code:                  0x00012200 (74240)
  Size of Initialized Data:      0x00001E00 (7680)
  Size of Uninitialized Data:    0x00000000 (0)
  Address of Entry Point:        0x00001273
  Base of Code:                  0x00001000
  Base of Data:                  0x00012000
  Image Base:                    0x71AB0000
  Section Align:                 0x00001000
  File Align:                    0x00000200
  Operating System Version:      5.01
  Image Version:                 5.01
  Subsystem Version:             4.10
  Size of Image:                 0x00017000 (94208)
  Size of Headers:               0x00000400 (1024)
  Checksum:                      0x0002030B
  Subsystem:                     0x0003 (Windows character subsytem)
  DLL Characteristics:           0x0000
  Size of Stack Reserve:         0x00040000 (262144)
  Size of Stack Commit:          0x00001000 (4096)
  Size of Heap Reserve:          0x00100000 (1048576)
  Size of Heap Commit:           0x00001000 (4096)
  Loader Flags:                  0x00000000
  Number of Rva and Sizes:       16

Data Directory:
  Export directory:               VA: 0x00001404  Size: 0x000011ED
(4589)
  Import directory:               VA: 0x000125C8  Size: 0x00000078
(120)
  Resource directory:             VA: 0x00015000  Size: 0x00000408
(1032)
  Exception directory:            VA: 0           Size: 0
  Security directory:             VA: 0           Size: 0
  Base relocation table:          VA: 0x00016000  Size: 0x00000DC8
(3528)
  Debug directory:                VA: 0x000130D4  Size: 0x00000038
(56)
  Architecture-specific data:     VA: 0           Size: 0
  RVA of global pointer:          VA: 0           Size: 0
  Thread local storage directory: VA: 0           Size: 0
  Load configuration directory:   VA: 0x0000BC78  Size: 0x00000048
(72)
  Bound import directory:         VA: 0x00000288  Size: 0x00000074
(116)
  Import address table:           VA: 0x00001000  Size: 0x000001D4
(468)
  Delay load import descriptors:  VA: 0x00012510  Size: 0x00000040
(64)
  COM run-time descriptor:        VA: 0           Size: 0
  (unknown directory entry):      VA: 0           Size: 0

Export Table:
  Name:                          WS2_32.dll
  Time Date Stamp:               0x41107EDA (4-8-2004 8:14:50)
  Version:                       0.00
  Ordinal Base:                  1
  Number of Functions:           500
  Number of Names:               117

  Ordinal   Entry Point   Name
       27   0x00002B0B    FreeAddrInfoW
       24   0x00002899    GetAddrInfoW
       25   0x0000C4EC    GetNameInfoW
      500   0x00012105    WEP
       28   0x00011CA7    WPUCompleteOverlappedRequest
       29   0x00010DA9    WSAAccept
       30   0x000091F6    WSAAddressToStringA
       31   0x000062B2    WSAAddressToStringW
      102   0x0000EA2B    WSAAsyncGetHostByAddr
      103   0x0000E985    WSAAsyncGetHostByName
      105   0x0000EAD5    WSAAsyncGetProtoByName
      104   0x0000E2AB    WSAAsyncGetProtoByNumber
      107   0x0000E84F    WSAAsyncGetServByName
      106   0x0000E8FB    WSAAsyncGetServByPort
      101   0x00010979    WSAAsyncSelect
      108   0x0000E32F    WSACancelAsyncRequest
      113   0x0000D3F2    WSACancelBlockingCall 
<<<<<<<<<<<<<<<<<<<<<<<<<<
      116   0x00004428    WSACleanup
       32   0x00004682    WSACloseEvent
       33   0x00010C69    WSAConnect
       34   0x000045F7    WSACreateEvent
       35   0x0000DB82    WSADuplicateSocketA
       36   0x0000DAF4    WSADuplicateSocketW
       37   0x0000F9C9    WSAEnumNameSpaceProvidersA
       38   0x0000FA29    WSAEnumNameSpaceProvidersW
       39   0x00004617    WSAEnumNetworkEvents
       40   0x0000DC47    WSAEnumProtocolsA
       41   0x0000848C    WSAEnumProtocolsW
       42   0x00004573    WSAEventSelect
      111   0x000094DC    WSAGetLastError
       43   0x00010D03    WSAGetOverlappedResult
       44   0x0000F458    WSAGetQOSByName
       45   0x00010428    WSAGetServiceClassInfoA
       46   0x00010141    WSAGetServiceClassInfoW
       47   0x0000FD87    WSAGetServiceClassNameByClassIdA
       48   0x0000FF69    WSAGetServiceClassNameByClassIdW
       49   0x0000BCC9    WSAHtonl
       50   0x0000BDB1    WSAHtons
       58   0x000103A9    WSAInstallServiceClassA
       59   0x0000FBB1    WSAInstallServiceClassW
       60   0x00004489    WSAIoctl
      114   0x0000D441    WSAIsBlocking
       61   0x00010F58    WSAJoinLeaf
       62   0x000052AA    WSALookupServiceBeginA
       63   0x00003307    WSALookupServiceBeginW
       64   0x00003226    WSALookupServiceEnd
       65   0x0000570E    WSALookupServiceNextA
       66   0x00002E99    WSALookupServiceNextW
       67   0x00004D06    WSANSPIoctl
       68   0x0000BCC9    WSANtohl
       69   0x0000BDB1    WSANtohs
       70   0x0000881F    WSAProviderConfigChange
       71   0x00004318    WSARecv
       72   0x0000F5D6    WSARecvDisconnect
       73   0x0000F652    WSARecvFrom
       74   0x0000FC9C    WSARemoveServiceClass
       75   0x0000949F    WSAResetEvent
       76   0x00006233    WSASend
       77   0x00010A0A    WSASendDisconnect
       78   0x00010A95    WSASendTo
      109   0x0000D494    WSASetBlockingHook
       79   0x000094B0    WSASetEvent
      112   0x00002A5E    WSASetLastError
       80   0x000104BA    WSASetServiceA
       81   0x00010269    WSASetServiceW
       82   0x00008769    WSASocketA
       83   0x000039CB    WSASocketW
      115   0x0000664D    WSAStartup
       84   0x000090D8    WSAStringToAddressA
       85   0x0000281E    WSAStringToAddressW
      110   0x0000D4F0    WSAUnhookBlockingHook
       86   0x0000948E    WSAWaitForMultipleEvents
       26   0x00011EC9    WSApSetPostRoutine
       87   0x000119B1    WSCDeinstallProvider
       88   0x0000EE41    WSCEnableNSProvider
       89   0x00007761    WSCEnumProtocols
       90   0x00008A27    WSCGetProviderPath
       91   0x0000F135    WSCInstallNameSpace
       92   0x0001164D    WSCInstallProvider
       93   0x0000F301    WSCUnInstallNameSpace
       94   0x00011279    WSCUpdateProvider
       95   0x0000F001    WSCWriteNameSpaceOrder
       96   0x00011519    WSCWriteProviderOrder
      151   0x00004544    __WSAFDIsSet
        1   0x00011028    accept
        2   0x00003E00    bind
        3   0x00009639    closesocket
        4   0x0000406A    connect
       97   0x00002B0B    freeaddrinfo
       98   0x00002A6F    getaddrinfo
       51   0x0000E479    gethostbyaddr
       52   0x00004FD4    gethostbyname
       57   0x000050C8    gethostname
       99   0x0000C671    getnameinfo
        5   0x00010B50    getpeername
       53   0x0000E176    getprotobyname
       54   0x0000E0B4    getprotobynumber
       55   0x0000E6EB    getservbyname
       56   0x0000E59D    getservbyport
        6   0x0000951E    getsockname
        7   0x000046C9    getsockopt
        8   0x00002BC0    htonl
        9   0x00002B66    htons
       11   0x00002BF4    inet_addr
       12   0x00003F41    inet_ntoa
       10   0x00004519    ioctlsocket
       13   0x000088D3    listen
       14   0x00002BC0    ntohl
       15   0x00002B66    ntohs
       16   0x0000615A    recv
       17   0x00002D0F    recvfrom
       18   0x00002DC0    select
       19   0x0000428A    send
       20   0x00002C69    sendto
       21   0x00003EA1    setsockopt
       22   0x00010BDE    shutdown
       23   0x00003B91    socket

And, also there i am confronted with the error:
Code: [Select]linkedin_console.exe
main thread
create api
execute api
create http
server.active = no
execute http
destroy api
free resources api
exception http: Could not accept a client connection on socket: 1920,
error 10004
finish http
destroy http

Heap dump by heaptrc unit
90 memory blocks allocated : 3003/3232
88 memory blocks freed     : 2895/3120
2 unfreed memory blocks : 108
True heap size : 196608
True free heap : 196464
Should be : 196368
Call trace for block $000B1378 size 12
  $0042BC8B
  $0042C37E
  $0042C1F0
  $0042B01F
  $0042AD6D
  $00417095
  $0040C5CE
Call trace for block $000A0118 size 96
  $0042B01F
  $0042AD6D
  $00417095
  $0040C5CE
  $7C80B50B
  $00417095
  $0040C5CE

But, i did not had a good look yet at all units/headers involved.

----------------------------------------------------------------------
tudi_x · 2017-11-25 09:37

opened https://bugs.freepascal.org/view.php?id=32727