TFPHTTPClient debugging lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- SymbolicFrank · 2022-05-20 14:23 It works, it's pretty simple to get going, but it is hard to see what is happening and it likes raising exceptions (there are 15). Like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TFPCustomHTTPClient.Get(const AURL: String; Stream: TStream);begin  HTTPMethod('GET',AURL,Stream,[200]);end; The HTTPMethod is the main function to execute the request. Which ends up in: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Function TFPCustomHTTPClient.ReadResponse(Stream: TStream;  const AllowedResponseCodes: array of Integer; HeadersOnly: Boolean): Boolean;  //...  if not CheckResponseCode(FResponseStatusCode,AllowedResponseCodes) then    Raise EHTTPClient.CreateFmt(SErrUnexpectedResponse,[ResponseStatusCode]); Where that "[200]" is the AllowedResponseCodes. So, if the result is not 200, you don't get to see the response, because it isn't read. Also, seeing what message (and headers!) is actually sent is quite hard. Is the intended use of this class that you override virtual methods until you can see what is happening? ---------------------------------------------------------------------- SymbolicFrank · 2022-05-23 13:09 How do you add parameters? ---------------------------------------------------------------------- Thaddy · 2022-05-23 17:04 200 is not the only legal response code. 301 and 302 are too (redirections, there are more...like for old security protocols) and quite very common nowadays. https://en.wikipedia.org/wiki/HTTP_301 https://en.wikipedia.org/wiki/HTTP_302 So if you expect 200 rightaway you are making a programmer error. Simply set AllowRedirects to true. Eventually you should get 200. ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-05-23 18:00 Hey SymbolicFrank, Quote from: SymbolicFrank on May 20, 2022, 04:23:03 pmAlso, seeing what message (and headers!) is actually sent is quite hard. If you look at the code (https://gitlab.com/freepascal.org/fpc/source/ -/blob/main/packages/fcl-web/src/base/fphttpclient.pp#L373) you can see that TFPHTTPClient exposes these, among many: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   TFPHTTPClient = Class(TFPCustomHTTPClient)  Published    { Shortened to make a point }    Property RequestHeaders;    Property RequestBody;    Property ResponseHeaders;    Property ResponseStatusCode;    Property ResponseStatusText;    Property Cookies;    Property Proxy;  end; If you're looking to inspect the Request Headers before you perform a method, that would be RequestHeaders. If you're looking to inspect the Request Body, usually empty if you don't place anything there, that would be RequestBody. If you want to inspect the Response Headers after you've performed a method, that would be ResponseHeaders. Cheers, Gus ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-05-23 18:05 Hey SymbolicFrank, Quote from: SymbolicFrank on May 23, 2022, 03:09:33 pmHow do you add parameters? If you're looking for a way to add parameters like one is used to on Postman, or some other HTTP Client object implementations you will not find any. It's up to you to add the question mark symbol and the name, value pairs in the URL string that you pass to any method call. Same with the anchor side of things: it's up to you to put the hash symbol and the name of the anchor if you want to have that in. But I do agree that it would be a good Quality of Life type of feature that could be added without much effort. Cheers, Gus ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-05-23 18:16 Hey SymbolicFrank, AHHH, sorry, I completely forgot that you can also hook up on these events to have access to what's happening during the actual method call: https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl- web/src/base/fphttpclient.pp#L373 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   TFPHTTPClient = Class(TFPCustomHTTPClient)  Published    {Shortened to make a point}    Property OnRedirect;    Property OnPassword;    Property OnDataReceived;    Property OnDataSent;    Property OnHeaders;    Property OnGetSocketHandler; Cheers, Gus ---------------------------------------------------------------------- Thaddy · 2022-05-23 19:23 Indeed TFPCustomHTTPClient exposes more (inheritance) and even more from what TFPCustomHTTPClient inherits. Follow the breadcrumps. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 06:59 Quote from: Thaddy on May 23, 2022, 07:04:08 pm200 is not the only legal response code. 301 and 302 are too (redirections, there are more...like for old security protocols) and quite very common nowadays. https://en.wikipedia.org/wiki/HTTP_301 https://en.wikipedia.org/wiki/HTTP_302 So if you expect 200 rightaway you are making a programmer error. Simply set AllowRedirects to true. Eventually you should get 200. Yes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 07:02 Quote from: Gustavo 'Gus' Carreno on May 23, 2022, 08:00:46 pmHey SymbolicFrank, Quote from: SymbolicFrank on May 20, 2022, 04:23:03 pmAlso, seeing what message (and headers!) is actually sent is quite hard. If you look at the code (https://gitlab.com/freepascal.org/fpc/source/ -/blob/main/packages/fcl-web/src/base/fphttpclient.pp#L373) you can see that TFPHTTPClient exposes these, among many: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   TFPHTTPClient = Class(TFPCustomHTTPClient)  Published    { Shortened to make a point }    Property RequestHeaders;    Property RequestBody;    Property ResponseHeaders;    Property ResponseStatusCode;    Property ResponseStatusText;    Property Cookies;    Property Proxy;  end; If you're looking to inspect the Request Headers before you perform a method, that would be RequestHeaders. If you're looking to inspect the Request Body, usually empty if you don't place anything there, that would be RequestBody. If you want to inspect the Response Headers after you've performed a method, that would be ResponseHeaders. Cheers, Gus Yes, but there are a lot of rules they have to conform to for them to be added to the message. So you have to override multiple methods to be able to see what is sent just before it is. That is the problem.  Ok, you can check the ResponseHeaders, unless you got something else than a 200. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 07:09 Quote from: Gustavo 'Gus' Carreno on May 23, 2022, 08:05:35 pmHey SymbolicFrank, Quote from: SymbolicFrank on May 23, 2022, 03:09:33 pmHow do you add parameters? If you're looking for a way to add parameters like one is used to on Postman, or some other HTTP Client object implementations you will not find any. It's up to you to add the question mark symbol and the name, value pairs in the URL string that you pass to any method call. Same with the anchor side of things: it's up to you to put the hash symbol and the name of the anchor if you want to have that in. But I do agree that it would be a good Quality of Life type of feature that could be added without much effort. Cheers, Gus Yes, and I did. The TFPHTTP client expects a server, a path and a file to get. For most modern sites the URL is a function to call and a set of parameters. 20 years ago that would have been totally fine. Today, not so much. And I did add and fix it all. This is just my feedback on what definitely could (and probably should) be improved. ---------------------------------------------------------------------- Thaddy · 2022-05-24 07:28 Quote from: SymbolicFrank on May 24, 2022, 08:59:26 amYes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was. No, TFpHttpClient does NOT throw exceptions on 301/302 if redirects are on. It will only throw an exception if the error is not a redirect or if the last in the chain is not 200. You can test this against e.g. google.com "[200, 301,302]"  are AllowedResponseCodes ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-05-24 07:34 Hey SymbolicFrank, Quote from: SymbolicFrank on May 24, 2022, 09:02:13 amYes, but there are a lot of rules they have to conform to for them to be added to the message. I'm really sorry, you lost me here. Could you please elaborate on what you call rules and what you call message? Quote from: SymbolicFrank on May 24, 2022, 09:02:13 amSo you have to override multiple methods to be able to see what is sent just before it is. That is the problem.  From the quick look that I gave the code, I think that TFPHTTPClient is not a complicated HTTP Client. It does HTTP 1.0 or 1.1 and that's pretty much it. I think it does not do HTTP 2.0, nor does it do compressed or chunked transport. But I could be wrong on these last ones. It pretty much sends what you can gather from RequestHeaders, RequestBody and that's pretty much it, no magic behind the curtain. So, when you say that you have to override multiple methods, I really don't understand your objective. Quote from: SymbolicFrank on May 24, 2022, 09:02:13 amOk, you can check the ResponseHeaders, unless you got something else than a 200. If I'm not mistaken, apart from some 500 errors, you always get content back on the ResponseHeaders, so I don't understand your argument here, sorry. In the end, what I think is that I really don't understand your debugging needs. And that's normal since you actually never specified them. You just stated that you can't debug it. So please, in the quest to better help you, could you please give more specifics on what you're expectations for the debug are? Could you provide us with an example of what you input and then what you're expecting to get back on such debug? This would really help us give you a more precise and focused help. Cheers, Gus ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 07:51 Quote from: Thaddy on May 24, 2022, 09:28:22 amQuote from: SymbolicFrank on May 24, 2022, 08:59:26 amYes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was. No, TFpHttpClient does NOT throw exceptions on 301/302 if redirects are on. It will only throw an exception if the error is not a redirect or if the last in the chain is not 200. You can test this against e.g. google.com "[200, 301,302]"  are the AllowedResponseCodes Yes, that's what I said. What I also said, was that it generates an exception if you get a 400+, and that you are unable to see the error message, because it isn't read. ---------------------------------------------------------------------- Thaddy · 2022-05-24 07:56 then should map the code against https://en.wikipedia.org/wiki/List_of _HTTP_status_codes#4xx_client_errors Those are descriptive enough. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 07:57 Quote from: Gustavo 'Gus' Carreno on May 24, 2022, 09:34:25 amHey SymbolicFrank, .. From the quick look that I gave the code, I think that TFPHTTPClient is not a complicated HTTP Client. It does HTTP 1.0 or 1.1 and that's pretty much it. I think it does not do HTTP 2.0, nor does it do compressed or chunked transport. But I could be wrong on these last ones. It pretty much sends what you can gather from RequestHeaders, RequestBody and that's pretty much it, no magic behind the curtain. .. Cheers, Gus Exactly. It's for simple stuff, and it either works, or it doesn't. I always used Synapse for client & server in the past, but as this is the default FP client, I used it this time. But it needs work to get it to do pretty basic stuff. That's it. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 08:00 Quote from: Thaddy on May 24, 2022, 09:56:48 amthen should map the code against https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4 xx_client_errors Those are descriptive enough. It's a set, it won't fit For now, I did it like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TRestClient.HTTPMethod(const AMethod, AURL: String; Stream: TStream;  const AllowedResponseCodes: array of Integer);var  p: TParameter;  s: string;begin  s := '';  if Parameters.Count > 0 then  begin    for p in Parameters do    begin      s += '&' + p.Name + '=';      if (not VarIsEmpty(p.Value)) and (not VarIsNull(p.Value)) then        s += VarToStr(p.Value);    end;    s[1] := '?';  end;   inherited HTTPMethod(AMethod, AURL + s, Stream, [200, 201, 400, 401]);end; And I'll add new errors when I encounter them. ---------------------------------------------------------------------- Thaddy · 2022-05-24 08:01 Quote from: SymbolicFrank on May 24, 2022, 09:57:16 amQuote from: Gustavo 'Gus' Carreno on May 24, 2022, 09:34:25 amHey SymbolicFrank, .. From the quick look that I gave the code, I think that TFPHTTPClient is not a complicated HTTP Client. It does HTTP 1.0 or 1.1 and that's pretty much it. I think it does not do HTTP 2.0, nor does it do compressed or chunked transport. But I could be wrong on these last ones. It pretty much sends what you can gather from RequestHeaders, RequestBody and that's pretty much it, no magic behind the curtain. .. Cheers, Gus Exactly. It's for simple stuff, and it either works, or it doesn't. I always used Synapse for client & server in the past, but as this is the default FP client, I used it this time. But it needs work to get it to do pretty basic stuff. That's it. What exactly do you miss in fcl-web? It is actually in the same family as synapse and some code is shared or derived (SSL). I rewrote all my web client code to use fcl-web/net instead of synapse, because it is better maintainted and seems to allow more connections. ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-05-24 08:05 Hey SymbolicFrank, Quote from: SymbolicFrank on May 24, 2022, 09:09:58 amYes, and I did. Ok, good, you handled the parameters, nice Quote from: SymbolicFrank on May 24, 2022, 09:09:58 amThe TFPHTTP client expects a server, a path and a file to get. For most modern sites the URL is a function to call and a set of parameters. 20 years ago that would have been totally fine. Today, not so much. For the verb/method GET, most API sites need an endpoint, some sort of authentication and some parameters in some cases. For the verbs/methods POST, PUT, DELETE, most API sites need an endpoint, some sort of authentication, some content on the RequestBody and maybe some parameters. For the endpoint, it's the piece of the URL that comes after the site's domain and you put that on the URL string.For the verbs themselves, apart from GET and POST, depending on how the site gets the others, usually it's via a Header added to the RequestHeaders.For the authentication: - if it's basic, you have Username and Password properties. - if it's something else, you have the RequestHeaders and/or parameters.For the RequestBody you have a stream type of thing, so you can put anything in there, even plain old JSON. You usually then get JSON or XML as the result of calling one of the verbs/methods, so you can then process that elsewhere in your app. From a modern perspective, I don't see what more you need in terms of added functionality to TFPHTTPClient. If so, and I guess I could've missed something, could you please be a bit more specific on what you feel is missing? Quote from: SymbolicFrank on May 24, 2022, 09:09:58 amAnd I did add and fix it all. You've lost me here again. What did you add and fix? Quote from: SymbolicFrank on May 24, 2022, 09:09:58 amThis is just my feedback on what definitely could (and probably should) be improved. Ermm, again, pardon me for asking, but I don't recall you mentioning or specifying any improvements. What I recall is you mentioning a lot of things that are either wrong or missing according to your opinion. Cheers, Gus ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-05-24 08:15 Hey SymbolicFrank, Quote from: SymbolicFrank on May 24, 2022, 09:57:16 amExactly. It's for simple stuff, and it either works, or it doesn't. I was still writing an answer to the other reply and Thaddy responded to this one in the meantime. And I have to say that I agree with him!! So, maybe the problem is with the way you're trying to use the component. From my experience there are 2 ways: The simple way, but you forgo some controlThe complete way and you have full control Could you please give us some examples of your code so we could maybe point you in the right direction? Quote from: SymbolicFrank on May 24, 2022, 09:57:16 amI always used Synapse for client & server in the past, but as this is the default FP client, I used it this time. But it needs work to get it to do pretty basic stuff. I have the same experience as Thaddy in this aspect. I've used both synapse and Indy in the past for lack of an internal option. But ever since they released TFPHTTPClient that I've been using it and figuring out the kinks when using SSL across multiple FPC versions. So again, I beg of you, could you please give us some examples of you code and what issues you're stumbling on? Cuz me and Thaddy have been rather happy with it for quite a while and here you come saying it doesn't work, so you confuse use, a lot!! Cheers, Gus ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 08:58 Quote from: Gustavo 'Gus' Carreno on May 24, 2022, 10:15:01 amHey SymbolicFrank, .. So again, I beg of you, could you please give us some examples of you code and what issues you're stumbling on? Cuz me and Thaddy have been rather happy with it for quite a while and here you come saying it doesn't work, so you confuse use, a lot!! Cheers, Gus Hi Gus, I think I did, see my post above. ---------------------------------------------------------------------- Warfley · 2022-05-24 08:59 Quote from: SymbolicFrank on May 24, 2022, 08:59:26 amYes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was. Thats wrong, it is possible, you can simply ask the client afterwards: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  cli: TFPHTTPClient;  res: TStringStream;begin  cli := TFPHTTPClient.Create(nil);  res := TStringStream.Create;  try    try    cli.Get('http://google.com/ThisPathDoesNotExist', res);    except    res.Seek(0, soBeginning);      WriteLn(cli.ResponseStatusCode, ' ', cli.ResponseStatusText);      WriteLn(cli.ResponseHeaders.Text);      WriteLn(res.DataString);    end;  finally    cli.Free;    res.Free  end;  ReadLn;end.This gives the complete HTTP response Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 404 Not FoundContent-Type: text/html; charset=UTF-8Referrer-Policy: no- referrerContent-Length: 1581Date: Tue, 24 May 2022 08:54:19 GMTConnection: close  That said, it is incredibly clumsy, as it does not make use of the main feature of exceptions (that they are objects that can hold abitrary data), so you have to get your data from different sources, with the actual exception being so void of information that in the code above I did not even need to catch it but a general except block is enough. Also this means that this does only work with an initialized HTTP client and not with the helper functions, and therefore requires a lot of boilerplate. This would be so much more comfortable: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i try  Res := TFPHTTPClient.Get(url);except on E: EInvalidHTTPResponse dobegin  WriteLn(E.StatusCode, ' ', E.StatusText);  WriteLn(E.ResponseHeaders.Text);  WriteLn(E.ResponseBody);end;end; So the current design is bad, no question, but at least it is possible to handle this sort of stuff ---------------------------------------------------------------------- Thaddy · 2022-05-24 09:01 Another option is to use libcurl which has full http/2 support. It is not as easy to use as synapse or fcl-web, but it is not THAT difficult and comes with FPC as standard: ./packages/libcurl In windows you need to obtain libcurl.dll. I fixed windows support a couple of years ago and Florian merged my patch. ---------------------------------------------------------------------- Warfley · 2022-05-24 09:04 Quote from: SymbolicFrank on May 24, 2022, 10:00:05 amIt's a set, it won't fit For now, I did it like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TRestClient.HTTPMethod(const AMethod, AURL: String; Stream: TStream;  const AllowedResponseCodes: array of Integer);var  p: TParameter;  s: string;begin  s := '';  if Parameters.Count > 0 then  begin    for p in Parameters do    begin      s += '&' + p.Name + '=';      if (not VarIsEmpty(p.Value)) and (not VarIsNull(p.Value)) then        s += VarToStr(p.Value);    end;    s[1] := '?';  end;   inherited HTTPMethod(AMethod, AURL + s, Stream, [200, 201, 400, 401]);end; And I'll add new errors when I encounter them. First, this code is potentially buggy, as you need to URL encode your parameters. Second, I think you shouldn't encode the parameters at this stage, the parameter encoding should be part of the URL with which the function gets called. I know that fpweb is missing some easy URI handling that makes the use of queries, fragments and co easy, but still this sort of stuff should already be part of the URI that gets in, it has nothing todo with HTTP and should therefore be not part of the HTTPClient class ---------------------------------------------------------------------- Thaddy · 2022-05-24 09:09 Links for libcurl on Windows are here: https://curl.se/windows/ In most nixes it comes as standard, but if not you can obtain it from the same website: curl.se Note that windows 10/11 come with curl.exe as standard, but without the libcurl.dll because Microsoft has statically linked it. Also note that the example header still says it needs modifications for windows. This is not the case after my fix, except you may want to rename libcurl-x64.dll to libcurl.dll for 64 bit windows. Static linking works too with FPC. Curl is very complete, stable, secure and very up-to-date. (At the cost of steep learning curve for programmers if you want all its features) ---------------------------------------------------------------------- Thaddy · 2022-05-24 09:27 Oh, wait!, I see the libcurl.pp in the fpc distribution needs an update... I will create one. ---------------------------------------------------------------------- MvC · 2022-05-24 09:29 About "parameters": As the maintainer of fphttpclient, I can only agree with warfly. The HTTP protocol expects a URL.  How this URL is constructed is not its business. But I agree that a class (or advanced record) for easy URL constructing would be useful, but it simply must not be part of the HTTPClient class. I have made a note to add this to the URIParser or HTTPProtocol unit. About exceptions: I think it is indeed possible to improve EHTTTPClient to contain the error document, and I will add this. It has already fields for status code and text, I will check whether they are already filled. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 09:30 @Warfley: I agree. This is for testing, and I certainly am going to URLEncode them and add a lot of other code to make it more bullet- proof, before it is production ready. I just wasn't sure if TFPHHTPClient was the way to go. ---------------------------------------------------------------------- SymbolicFrank · 2022-05-24 09:32 Quote from: MvC on May 24, 2022, 11:29:52 amAbout "parameters": As the maintainer of fphttpclient, I can only agree with warfly. The HTTP protocol expects a URL.  How this URL is constructed is not its business. But I agree that a class (or advanced record) for easy URL constructing would be useful, but it simply must not be part of the HTTPClient class. I have made a note to add this to the URIParser or HTTPProtocol unit. About exceptions: I think it is indeed possible to improve EHTTTPClient to contain the error document, and I will add this. It has already fields for status code and text, I will check whether they are already filled. Thanks!