(Indy) Web application get page error lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- xinyiman · 2018-11-07 08:39 Hi guys, I created a web app with integrated web server (indy 10) to replicate errors. I would like your opinion on this. What's wrong? I have to read a json string through an http call and I tried 3 ways. The first works, the other two do not. But I would like to understand what is different. For me it is absurd. Who can help me? With indy, reading this string generates an error that is not always replicable: HTTP / 1.1 200 OK Connection: close Content-Length: 397 While synapse the version that I expect to work does not work, while the one written by me. Tips? I enclose an example. This is my browser result: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i correct: { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUt TUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuL zdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk1 1RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkRqd WxlbmNRMWo2bWlVK1kzRi9sbnEwU2R6Q3BHV3JaR3gwTGYrblpwUmNEaVc1UEdDdC9mQTV UbGs2OUtOanloUlJWN1ZIR2JydlRYQnh0V3ljbz0=" }error 1: { "token" : "ZVRo QWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWl Fscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWto Vjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWT hJQThxaTBrLy9MQlBhU1loeTVCYzerror 2: HTTP/1.1 200 OK Connection: close Content-Length: 397 Server: NGIT Bootstrap Application Set-Cookie: IDH TTPSESSIONID=nh2ari4EVMF2sxBnh2ari4EVMF2sxBnh2ari4EVMF2sxBnh2ari4EVMF2 sxBnh2ari4EVMF2sxBnh2ari4EVMF2sxB; path=/ { "token" : "ZVRoQWN0aFF6UEx kdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZd XlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ 6blFDY ---------------------------------------------------------------------- xinyiman · 2018-11-07 11:01 It was pointed out to me that I left dcpcrypt among the dependencies. You can remove it, it is not necessary for the use of the test program ---------------------------------------------------------------------- Thaddy · 2018-11-07 11:16 The error is obviously a terminating zero because it expects a PChar. Will test after lunch. ---------------------------------------------------------------------- Remy Lebeau · 2018-11-07 18:34 Quote from: xinyiman on November 07, 2018, 09:39:39 amHi guys, I created a web app with integrated web server (indy 10) to replicate errors. I would like your opinion on this. What's wrong? Some mistakes I see: - your TIdHTTPServer.OnCommandGet event handler is catching and discarding all raised exceptions.  Don't do that!  Let TIdHTTPServer handle exceptions so it can send error messages back to the HTTP client.  If you are going to catch exceptions for your own logging, at least re-raise them when you are done using them.  Otherwise, when the OnCommandGet event handler exits cleanly, TIdHTTPServer will send whatever reply data is currently in the the TIdHTTPResponse object (if it has not already been sent manually by the OnCommandGet event handler), and that will default to sending an HTTP 200 reply. - your TNGITBootstrapApplication constructor should be initializing its FHTMLDir and FTemplate members before activating the TIdHTTPServer, so they are ready for use before any clients can connect and potentially access them before they are ready. - TNGITController.GetHTML() is not protecting itself from exceptions when version is 2.  You need to add try..finally blocks to free the TIdHTTP and TMemoryStream objects correctly.  TIdHTTP.Get() will raise an exception if it receives an error from the server (unless you disable that option in the TIdHTTP.HTTPOptions property). ---------------------------------------------------------------------- xinyiman · 2018-11-08 07:23 Thank you Remy Lebea. One question, your suggestion is like my attachment? Thank you ---------------------------------------------------------------------- xinyiman · 2018-11-08 11:10 Quote(unless you disable that option in the TIdHTTP.HTTPOptions property) How? I searched but only HTTPOptions value are: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   // Protocol options  TIdHTTPOption = (hoInProcessAuth, hoKeepOrigProtocol, hoForceEncodeParams);  TIdHTTPOptions = set of TIdHTTPOption; ---------------------------------------------------------------------- Remy Lebeau · 2018-11-08 19:19 Quote from: xinyiman on November 08, 2018, 08:23:42 amOne question, your suggestion is like my attachment? I don't see my suggested changes applied to that code.  Especially the OnCommandGet change in particular.  It is still catching and discarding all exceptions. Quote from: xinyiman on November 08, 2018, 12:10:10 pmHow? I searched but only HTTPOptions value are: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   // Protocol options  TIdHTTPOption = (hoInProcessAuth, hoKeepOrigProtocol, hoForceEncodeParams);  TIdHTTPOptions = set of TIdHTTPOption; You claim to be using Indy 10, but are you maybe using Indy 9 instead?  What you show above is what TIdHTTPOption looks like in Indy 9.  In the current Indy 10 release, TIdHTTPOption has MANY more options: Code: [Select]  TIdHTTPOption = (hoInProcessAuth, hoKeepOrigProtocol, hoForceEncodeParams,     hoNonSSLProxyUseConnectVerb, hoNoParseMetaHTTPEquiv, hoWaitForUnexpectedData,     hoTreat302Like303, hoNoProtocolErrorException, hoNoReadMultipartMIME,     hoNoParseXmlCharset, hoWantProtocolErrorContent, hoNoReadChunked     ); When I mentioned configuring TIdHTTP.Get() to not raise an exception on a server error, I was referring to the hoNoProtocolErrorException option. ---------------------------------------------------------------------- xinyiman · 2018-11-08 22:45 I see but lazarus have install indylaz v10.2.0.3 but i have only: hoInProcessAuth, hoKeepOrigProtocol, hoForceEncodeParams where I find the version of indy 10 that contains all the options you say?! ---------------------------------------------------------------------- xinyiman · 2018-11-09 08:31 Ok, i find a correct version of indy. Now i change with your suggestion. But now my program closed when into browser page run fast. If you try to quickly reload the page without waiting for the page to load before the program closes without giving any kind of error. Why? ---------------------------------------------------------------------- Remy Lebeau · 2018-11-09 21:27 Quote from: xinyiman on November 08, 2018, 11:45:09 pmI see but lazarus have install indylaz v10.2.0.3 but i have only: hoInProcessAuth, hoKeepOrigProtocol, hoForceEncodeParams That is an extremely old version of Indy.  The current version is 10.6.2.5485. Quote from: xinyiman on November 08, 2018, 11:45:09 pmwhere I find the version of indy 10 that contains all the options you say?! You can download Indy from its official download mirrors, which includes Lazarus's own Online Package Manager (though, I don't know which version of Indy is currently in OPM.  IIRC, it does sync up fairly often). Quote from: xinyiman on November 09, 2018, 09:31:50 amOk, i find a correct version of indy. Now i change with your suggestion. But now my program closed when into browser page run fast. If you try to quickly reload the page without waiting for the page to load before the program closes without giving any kind of error. Why? I have no idea what you are referring to.  Please clarify. ---------------------------------------------------------------------- xinyiman · 2018-11-09 21:50 Have you tried to download and run my last attached example? Have you tried to run it? You'll see that if you try to run my program and quickly load the page in the browser, the http server closes itself without giving an error. ---------------------------------------------------------------------- Remy Lebeau · 2018-11-09 22:46 Quote from: xinyiman on November 09, 2018, 10:50:17 pmHave you tried to download and run my last attached example? Have you tried to run it? No, because I'm not a FreePascal/Lazarus user, so I couldn't compile and run your project even if I wanted to. Quote from: xinyiman on November 09, 2018, 10:50:17 pmYou'll see that if you try to run my program and quickly load the page in the browser, the http server closes itself without giving an error. It is your responsibility to debug your own project.  I can't help you with that. ---------------------------------------------------------------------- xinyiman · 2018-11-09 23:27 But Is not a Project problem, because whit old version of Indy not exists this problem. For me Is a library problem. You know a Lazarus developer from Indy team?! ---------------------------------------------------------------------- xinyiman · 2018-11-10 15:56 I testes my project example on other O.S. and this is the result 1. Windows: no problem, my project it did not close unexpectedly 2. Linux: problem, my project it closed unexpectedly 3. Mac OSX: problem, my project it closed unexpectedly Into unix like OS is problem ---------------------------------------------------------------------- xinyiman · 2018-11-10 16:58 I found the place where the problem is generated, in the IdHTTP unit there is the function Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TIdCustomHTTP.Get(AURL: string  {$IFDEF STRING_IS_ANSI}; ADestEncoding: IIdTextEncoding = nil{$ENDIF}  ): string;begin  Result := Get(AURL, []{$IFDEF STRING_IS_ANSI}, ADestEncoding{$ENDIF});end; It happens that if the server receives two simultaneous calls it closes. This is on linux and on mac osx. Only I do not understand how to solve. Some idea? ---------------------------------------------------------------------- asdf121 · 2018-11-10 17:45 Did you define a TIdCompressorZLib for using gzip? I think there is an issue because my old code does not work anymore after updating Indy to latest version. Maybe it's related to the change from Updating various DLL related functions to use THandle instead of HMODULE? If I remove the support for gzip from my TIdHTTP object, it works flawless. ---------------------------------------------------------------------- xinyiman · 2018-11-11 00:10 how? if like this not found idhttp1.Compressor:=nil; ---------------------------------------------------------------------- xinyiman · 2018-11-11 15:33 Ok, I've done other tests and I understand that the problem is not the GET call, the problem is that the http server crashes when idhttpserver1.OnCommandGet is called the second time but has not finished running the first one yet. I modified the source, if you open it you will see that now there is a sleep (4000); in order to give you time to load the page in the browser the second time. Can someone tell me what's wrong? I do not understand. It should be managed multithreaded so it should not happen. Ideas? ---------------------------------------------------------------------- xinyiman · 2018-11-12 10:20 No suggestions? Has anyone at least managed to replicate my problem? ---------------------------------------------------------------------- xinyiman · 2018-11-12 12:34 In the face of other tests I understood that the problem exists only if the requests are made by the same PC on which the http server runs. If I do it with another computer, or the smartphone or tablet, the server does not crash. Ideas? ---------------------------------------------------------------------- Remy Lebeau · 2018-11-12 19:22 Quote from: xinyiman on November 10, 2018, 12:27:50 amYou know a Lazarus developer from Indy team?! I'm the only developer left on the team.  Other remaining members are admins. Quote from: xinyiman on November 10, 2018, 05:58:23 pmI found the place where the problem is generated, in the IdHTTP unit there is the function Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TIdCustomHTTP.Get(AURL: string  {$IFDEF STRING_IS_ANSI}; ADestEncoding: IIdTextEncoding = nil{$ENDIF}  ): string;begin  Result := Get(AURL, []{$IFDEF STRING_IS_ANSI}, ADestEncoding{$ENDIF});end; It happens that if the server receives two simultaneous calls it closes. This is on linux and on mac osx. Only I do not understand how to solve. Some idea? What 2 calls are you referring to?  There is only 1 GET request in that code.  Just one overloaded Get() calling another overloaded Get() with extra parameters.  But only 1 GET request is sent to the server. Quote from: xinyiman on November 11, 2018, 04:33:14 pmOk, I've done other tests and I understand that the problem is not the GET call, the problem is that the http server crashes when idhttpserver1.OnCommandGet is called the second time but has not finished running the first one yet. What kind of crash EXACTLY? What is your logging telling you? TIdHTTPServer (like all of Indy's TCP servers) is a multi-threaded component.  The OnCommandGet event (and other events) is triggered in the context of a worker thread for a particular client (represented by the AContext parameter).  OnCommandGet can be called multiple times in parallel, but only when receiving requests from multiple clients.  For a given client, the events are serialized, and HTTP is a command/response protocol, so it is simply not possible for OnCommandGet to be called for a client while it is already running for that same client.  But, it can certainly be called for another client.  A browser can (and frequently does) make multiple TCP connections to a server.  That is perfectly normal.  It is your responsibility to make sure the code in your event handlers is thread- safe. Quote from: xinyiman on November 11, 2018, 04:33:14 pmI modified the source, if you open it you will see that now there is a sleep (4000); in order to give you time to load the page in the browser the second time. That merely blocks a client from responding for 4 seconds.  If you ask your browser to make a new request while the previous request is still blocked, the browser will simply open a new TCP connection to the server (thus running a new thread in the server) to make that new request. ---------------------------------------------------------------------- xinyiman · 2018-11-13 07:13 I meant two calls from the clients. If I open the browser from the same computer it is the server and very quickly launch two pages the program closes without daring errors. Can you explain why? On windows it does not happen, only on unix-like systems (I tried mac osx and ubuntu). ---------------------------------------------------------------------- Remy Lebeau · 2018-11-13 19:51 Quote from: xinyiman on November 13, 2018, 08:13:35 amIf I open the browser from the same computer it is the server and very quickly launch two pages the program closes without daring errors. Then you are doing something wrong in your code.  You need to debug it to see how it reacts with multiple browser requests being made at the same time. ---------------------------------------------------------------------- xinyiman · 2018-11-14 07:11 My code is the one I have above. It comes from one of the examples released with the package. I do not doubt it is a mistake. I'm asking for help precisely because I do not understand where I'm wrong. ---------------------------------------------------------------------- Remy Lebeau · 2018-11-14 19:58 Quote from: xinyiman on November 14, 2018, 08:11:40 amI do not doubt it is a mistake. I'm asking for help precisely because I do not understand where I'm wrong. Well, you didn't answer my earlier questions yet: QuoteWhat kind of crash EXACTLY? What is your logging telling you? You are the one running the server.  How does it behave exactly?  HOW is it crashing?  WHAT is being reported to you when the crash is detected? ---------------------------------------------------------------------- xinyiman · 2018-11-15 07:24 As I have already told you, it does not report any errors, simply the program closes itself. ---------------------------------------------------------------------- Remy Lebeau · 2018-11-19 20:34 Quote from: xinyiman on November 15, 2018, 08:24:38 amAs I have already told you, it does not report any errors, simply the program closes itself. As I stated earlier, you need to step through your code with a debugger and find out exactly what is happening.  I can't help you with that.