I need help debugging FPHTTPClient
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-26 14:19

Hey Y'all,

Ok, here's my situation, and it's quite particular to my setup, since
it only occurs when I'm tethering to my phone:
ANY HTTPS download using FPHHTPClient will prematurely end at around
74%

Using Ikel's cli-fp I've done this to try and find out what's
happening:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i unit
CMD.FPHTTPCli; {$mode ObjFPC}{$H+}{$J-} interface uses  Classes,
SysUtils, Common.Utils    // Utilities for formatting, CLI.Interfaces
 // Core interfaces, CLI.Progress    // Optional: Progress indicators,
CLI.Command     // Base command implementation, CLI.Console     //
Optional: Colored console output; type { TCmdFHTTPCli }  TCmdFHTTPCli
= class(TBaseCommand)  private    FSpinner: IProgressIndicator;   
FStream: TMemoryStream;     FSize: Int64;     procedure
OnDataReceived(Sender: TObject; Const ContentLength, CurrentPos:
Int64);  protected  public    { Main execution method that
orchestrates the greeting display      @return 0 for success, 1 for
errors }    function Execute: Integer; override;  published  end; var 
CmdFHTTPCli: TCmdFHTTPCli; implementation uses  fphttpclient   // Free
Pascal HTTP Client, opensslsockets; const  codes: array of Integer =
(    200,    400,    401,    403,    404,    405,    500,    501,   
502,    503,    504  ); { TCmdFHTTPCli } procedure
TCmdFHTTPCli.OnDataReceived(Sender: TObject; const ContentLength, 
CurrentPos: Int64);begin  FSpinner.Update(0);  TConsole.Write(Format('
Data Received: %9s of %9s (%6.2f%%); Stream: %9s',[   
FormatBytes(CurrentPos),    FormatBytes(FSize),    Percent(FSize,
CurrentPos),    FormatBytes(FStream.Position)  ]));end; function
TCmdFHTTPCli.Execute: Integer;var  http: TFPHTTPClient;  url: String; 
index: Integer;begin  Result:= 0;   if GetParameterValue('--url', url)
then  begin    http:= TFPHTTPClient.Create(nil);    try      FStream:=
TMemoryStream.Create;      try        try          FSpinner :=
CreateSpinner(ssDots);          FSpinner.Start;          
http.AllowRedirect:= True;          http.KeepConnection:= False;     
    FSpinner.Update(0);          TConsole.Write(' Calling HEAD');    
     http.HTTPMethod('HEAD', url, nil, []);          FSize := 0;     
   for index := 0 to Pred(http.ResponseHeaders.Count) do         
begin            if LowerCase(http.ResponseHeaders.Names[index]) =
'content-length' then            begin              FSize:=
StrToInt64(http.ResponseHeaders.ValueFromIndex[index]);             
FSpinner.Update(0);              TConsole.Write(Format(' Got Size: %d;
Status(%d): %s', [                FSize,               
http.ResponseStatusCode,                http.ResponseStatusText      
      ]));              break;            end;          end;         
Sleep(2000);           FSpinner.Update(0);          TConsole.Write('
Calling GET                                    ');         
http.OnDataReceived:=@OnDataReceived;          http.HTTPMethod(      
    'GET',            url,            FStream,            codes      
  );          FSpinner.Stop;         
TConsole.WriteLn(Format('Status(%d): %s', [http.ResponseStatusCode,
http.ResponseStatusText]));        except          on E:Exception do 
       begin            Result:= 1;           
TConsole.WriteLn('ERROR: ' + E.Message, ccRed);          end;       
end;      finally        FStream.Free;      end;    finally     
http.free;    end;  end  else  begin    Result:= 1;   
TConsole.WriteLn('ERROR: No URL to download', ccRed);  end;end; end.
As you can see I have:
Created a codes variable with a ton of HTTP Status codes.Put a call to
HEAD because the first 4 or 5 calls to OnDataReceived come with the
total at zeroA try..except around the entire transfer code
While testing it, no matter the speed that my provider's allowing me,
I get consistent download drop at around 74%, I know this is a clue,
just not figured what it means yet.
This only happens in HTTPS not HTTP. I've tested both and the HTTP
completes every time, no issue.
The call to HTTPMethod never triggers any exception and always returns
200 OK.
It only happens on files above 10MB. I've tested with files of ~1MB
and it downloads it fine, but with files of 11MB, 16MB and 17MB it
drops.
I've tested with aria2c, wget and cURL. They download the file without
any issue.

One of the things that bothers me is the fact that FPHHTPClient
swallows any/all exception. I don't think that we get anything going
bad if it happens at the SSL level. Which is where my suspicions are
at the moment. I don't think it's chunking, but I'm not sure.

My suspicions, at the moment are: It's either an SSL issue, or a
chunking issue.

Since I don't want to post an issue on GitLab to something that can
only be reproduced with my own phone's tethering, I need some help to
debug. I'm quite unfamiliar with the code and the hierarchy of the
Classes that it's just a black box at the moment.

Why all this? Welp, I did a GUI downloader that is now used on
fpcupdeluxe, and it fails miserably  . I need to get to the bottom of
this in order to not disappoint it's author!!

Can someone give me some pointers to where I should dig in?
Many thanks in advance!!

Cheers,
Gus

----------------------------------------------------------------------
dbannon · 2025-06-27 01:17

Gee Gus, I just do this, first few line important. Most of the code is
excepton handling and reporting. As you can see, fpHTPclient generates
lots of exceptions for me. I have, during testing, seen then all.

The downloaded file, in my case, text (md) or xml, ends up in
Somestring and is never truncated.

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TMistySync.Downloader(URL : string; out SomeString : String; const
ConType : TContentType; const Header : string = '') : boolean;var   
Client: TFPHTTPClient;begin    if DebugMode then
debugln('TMistySync.Downloader URL is ' + URL);    Client :=
TFPHttpClient.Create(nil);    Client.UserName := UserName;   
Client.Password := Password; //
'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';     eg a github token   
Client.AddHeader('User-Agent','Mozilla/5.0 (compatible; fpweb)');   
case ConType of        ctXML : Client.AddHeader('Content-
Type','application/xml; charset=UTF-8');        ctText :
Client.AddHeader('Content-Type','application/text; charset=UTF-8');  
    ctJSON : Client.AddHeader('Content-Type','application/json;
charset=UTF-8');        ctHTML : Client.AddHeader('Content-
Type','application/HTML; charset=UTF-8');    end;   
Client.AllowRedirect := true;    Client.ConnectTimeout := 8000;    
 // mS ?  was 3000, I find initial response from github very slow
....    Client.IOTimeout := 4000;           // mS ? was 0   
SomeString := '';    try        try            SomeString :=
Client.Get(URL);        except            on E: EHTTPClient do begin 
                                       // eg, File Not Found, we have
asked for an unavailable file                ErrorString :=
'TMistySync.Downloader - EHTTPClient Error ' + E.Message             
     + ' ResultCode ' + inttostr(Client.ResponseStatusCode);         
     exit(SayDebugSafe(ErrorString));            end;            on E:
ESocketError do begin                ErrorString :=
'TMistySync.Downloader - SocketError ' + E.Message     // eg failed
dns, timeout etc                    + ' ResultCode ' +
inttostr(Client.ResponseStatusCode);                SomeString :=
'Fatal, is server availale ?';               
exit(SayDebugSafe(ErrorString));                end;            on E:
EInOutError do begin                ErrorString := 'TMistySync
Downloader - InOutError ' + E.Message;                // might
generate "TGithubSync Downloader - InOutError Could not initialize
OpenSSL library"                SomeString := 'Failed to initialise
OpenSSL';           // is error message translated ?               
exit(SayDebugSafe(ErrorString));                end;            on E:
ESSL do begin                ErrorString := 'TMistySync.Downloader -
SSLError ' + E.Message;         // eg openssl problem, maybe FPC
cannot work with libopenssl                SomeString := 'Failed to
work with OpenSSL';                exit(SayDebugSafe(ErrorString));  
            end;            on E: Exception do begin               
ErrorString := 'TMistySync.Downloader Unexpected Exception ' +
E.Message + ' downloading ' + URL;                ErrorString :=
ErrorString + ' HTTPS error no ' +
inttostr(Client.ResponseStatusCode);               
exit(SayDebugSafe(ErrorString));                end;        end;     
 Result := Client.ResponseStatusCode = 200;        // My version of
this in GitHubSync then did some processing of Client.ResponseHeaders,
 not necessary here.    finally        Client.Free;    end;end;      
     

Davo

----------------------------------------------------------------------
440bx · 2025-06-27 02:37

@Davo,

I've been looking for some documentation for TFPHttpClient but, it
looks like I may have looked in all the wrong places.

Based on the code you posted above, it seems like you may know of
where it is documented and, if so, I'd really appreciate it if you
shared where that documentation is found.

Thank you in advance.

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-27 10:07

Hey Davo,

Wow, that's a freekin' humongous list of Exceptions, that is LOL!!!

Thank you so very much for sharing that code!! That does inform me of
a lot of stuff and I'll look at it better and compare with my code!!

Nonetheless, the thing that frustrates me the most is the fact that I
get NO exceptions at all, and to add injury to insult, it returns 200
OK.

What I'm guessing is that the SSL layer, somehow drops the stream, but
for some reason the issue does not bubble up to Pascal land. Somehow,
Pascal land interpretes the stream drop as a completion and then
happily spits out a 200 OK 

My connection, during peek times is abysmal. That has given me the
opportunity to memorize a lot of errors that other apps dish out when
they can't download.
A lot of errors telling me that the SSL handshake timed out, or that
the SSL layer had an unexpected EOF and much more.
With that knowledge, I think a premature EOF is happening at the SSL
layer, but for some mysterious reason, it doesn't bubble up to Pascal
land...
Cuz, you have to remember that it always drops at ~74%. And that's
consistent !! So, the connection goes through, it pull about 74% of
data and then all of a sudden BOOM, drops...

Welp, I've ranted enough about this...

Cheers,
Gus

----------------------------------------------------------------------
dbannon · 2025-06-28 00:39

Well Gus, all the exceptions listed there do happen and do pop up as
an exception. I got the basic thing happening and deliberately
provoked errors, caught them. That included using, at the time, a very
flaky network connection, while totally out of control, useful for
testing. I cannot imagine why yours fail silently. The fpc httpserver
does handle some mystery exceptions internally, you see it when
running under the debugger. Maybe same thing ?

440bx, no, no secrets I am afraid. What you see there is from the
examples in the compiler (use fpc324-branch, not fpc322) and the wiki.
And a lot of experimenting, especially, as noted, the error handling.

That code was first used in a Github API sync,
https://github.com/tomboy-notes/tomboy-
ng/blob/master/source/transgithub.pas and it now reused in another
unit that will talk directly to the fpc httpserver. That is a work in
progress.

Davo

----------------------------------------------------------------------
440bx · 2025-06-28 00:48

Quote from: dbannon on June 28, 2025, 02:39:49 am440bx, no, no secrets
I am afraid. What you see there is from the examples in the compiler
(use fpc324-branch, not fpc322) and the wiki. And a lot of
experimenting, especially, as noted, the error handling.

That code was first used in a Github API sync,
https://github.com/tomboy-notes/tomboy-
ng/blob/master/source/transgithub.pas and it now reused in another
unit that will talk directly to the fpc httpserver. That is a work in
progress.

Davo
I now see your knowledge was the product of genuine sweat (just like
most of the little bit of knowledge I have about it.)  Thank you for
clarifying the source of your knowledge in that area.

----------------------------------------------------------------------
Thaddy · 2025-06-28 06:24

The real TFPHTTPClient does not swallow exceptions. Why do you think
so? That is the first thing that draws my attention. That statement is
plainly not true. It is written by the core team and they do not
swallow exceptions because that is bad coding.

The cause can be a problem with the frame size (I encountered that in
the past) but that would be a problem in ssockets or sockets, not
FPHttpClient. Although the bug I referred to above was actually that
the remote connection used a non-standard frame size (1500 instead of
1512)

----------------------------------------------------------------------
MarkMLl · 2025-06-28 09:43

Quote from: Thaddy on June 28, 2025, 08:24:57 amAlthough the bug I
referred to above was actually that the remote connection used a non-
standard frame size (1500 instead of 1512)

And OP (Gus) specifically says that he gets the problem with a
tethered 'phone: I've previously seen non-standard MTU/frame size in
this context because the carrier is using a tunneling protocol.

MarkMLl

----------------------------------------------------------------------
Thaddy · 2025-06-28 11:45

Googling a bit deeper, it seems the most likely case. In my example
1500 is the payload, excluding overhead so the frame becomes too
small. Usually PMTUD should solve this and is usually implemented.
If not, you get frame drift and that brings MTU down eventually.
That will also explain the ~75% mark depending on size.
(Because of (1500/1512)/ div 2)
So we need to focus on that, but then I need some more code.
If Path MTU isn't working, packages get dropped mid-stream.
I think that is what is happening here.

This is a noughty one.... Although it is known in theory and practice.

Do you have an exact URL to which I can test? And what are the exact
versions of THttpClient and opensslsockets you are using?

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 12:07

Hey Thaddy,

Quote from: Thaddy on June 28, 2025, 08:24:57 amThe real TFPHTTPClient
does not swallow exceptions. Why do you think so? That is the first
thing that draws my attention. That statement is plainly not true. It
is written by the core team and they do not swallow exceptions because
that is bad coding.

You're absolutely right!! This was made evident when Davo dropped his
code.

My only excuse, and it's a really bad one, is that I never got an
exception in the past. So, yeah, mea culpa all the way !!!

Quote from: Thaddy on June 28, 2025, 08:24:57 amThe cause can be a
problem with the frame size (I encountered that in the past) but that
would be a problem in ssockets or sockets, not FPHttpClient. Although
the bug I referred to above was actually that the remote connection
used a non-standard frame size (1500 instead of 1512)

Hummm.... This is above my pay grade, and I'll need a bit more info so
I can look at the appropriate code lines to test that...

Cheers,
Gus

----------------------------------------------------------------------
Thaddy · 2025-06-28 12:12

It is likely solvable by using a smaller MTU frame size. That would be
slightly slower, but then the whole protocol chain will be such that
it can easier recover. Try specifying e.g. 512.
As I wrote: I encountered this issue only once but it is not uncommon.
And many lost a lot of hair, pulled out, and now need a whig.

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 12:18

Hey Thaddy,

Quote from: Thaddy on June 28, 2025, 01:45:28 pmGoogling a bit deeper,
it seems the most likely case. In my example 1500 is the payload,
excluding overhead so the frame becomes too small. Usually PMTUD
should solve this and is usually implemented.
If not, you get frame drift and that brings MTU down eventually.
That will also explain the ~75% mark depending on size.
(Because of (1500/1512)/ div 2)
So we need to focus on that, but then I need some more code.
If Path MTU isn't working, packages get dropped mid-stream.
I think that is what is happening here.

I'll take your word for it

Quote from: Thaddy on June 28, 2025, 01:45:28 pmThis is a noughty
one.... Although it is known in theory and practice.

Indeed it is !!

Quote from: Thaddy on June 28, 2025, 01:45:28 pmDo you have an exact
URL to which I can test? And what are the exact versions of
THttpClient and opensslsockets you are using?

This has been happening for some years now, so I'm gonna say any
version.
I've been doing my testing with Lazarus [3.6, 4.0, 4.99] and FPC
[3.2.2, 3.3.1].
I'm using this URL: https://github.com/LongDirtyAnimAlf/fpcupdeluxe/re
leases/download/darwin_arm64_crossbins_all/Embedded_riscv32_unknown_V2
40.zip in particular, but I suspect any SSL one will do.

What baffles me the most is that I've now tested: cURL, wget, aria2c,
a program in Go, a program in Zig and they all work with no issues.
I was unable to test Indy because it errors out with the LCL on a CLI
only program. I did try on a GUI app, and ran into OpenSSL versioning
shenanigans
I was unable to test with synapse 40.1 because of OpenSSL versioning
shenanigans.
Both Indy and synapse 40.1 are from the current OPM versions.
It's only FPHTClient that drops the ball at ~%74, consistently, in the
case of the above file size ~16MiB.
Also of note: If I DON'T use my phone as the tether, I have no issue
what so ever !!

Hope that helps!!

EDIT: I'm adding the code that I'm using for the tests.

Cheers,
Gus

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 12:20

Hey Thaddy,

Quote from: Thaddy on June 28, 2025, 02:12:47 pmIt is likely solvable
by using a smaller MTU frame size. That would be slightly slower, but
then the whole protocol chain will be such that it can easier recover.
Try specifying e.g. 512.
As I wrote: I encountered this issue only once but it is not uncommon.
And many lost a lot of hair, pulled out, and now need a whig.

Please provide me the line of code where I can change the MTU. I'm not
quite sure where that is. I'm guessing in the Socket somewhere, but
I', too ignorant to be sure!!

Cheers,
Gus

----------------------------------------------------------------------
Thaddy · 2025-06-28 12:57

Hm. Seems to need a patch or two...
We have to adapt TInetSocket a bit, or I can prepare a hack for it.
Come back later. It is not a big fix.

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 13:34

Hey Thaddy,

I remember that Ubuntu has a way to set the MTU on the wired
connection settings.

I'll do some testing with the 512 value you proposed. I'll relay the
results once my connection is not capped at ~20KiB/s 

Cheers,
Gus

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 13:35

Hey Thaddy,

Quote from: Thaddy on June 28, 2025, 02:57:13 pmHm. Seems to need a
patch or two...
We have to adapt TInetSocket a bit, or I can prepare a hack for it.
Come back later. It is not a big fix.

Don't really understand what you're saying here, but I'll wait for
your instructions nonetheless!!

Cheers,
Gus

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 14:18

Hey Thaddy,

Look like Ubuntu doesn't like an MTU of 512.

When I set it to 1500, it does report that it's at 1500.

When I set it to 512, this is what it gives me from ip a:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i 2: enp4s0:
<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UP
group default qlen 1000    link/ether 74:56:3c:6b:85:50 brd
ff:ff:ff:ff:ff:ff    altname enx74563c6b8550    inet 10.42.0.100/24
brd 10.42.0.255 scope global noprefixroute enp4s0       valid_lft
forever preferred_lft forever    inet6 fe80::7656:3cff:fe6b:8550/64
scope link noprefixroute        valid_lft forever preferred_lft
forever
It shows me it's at 1280 and not 512. Curiously, the same value as
when I set it to automatic 

I also tried the download at 1500 and got cut off...

Any other suggestion?

Cheers,
Gus

----------------------------------------------------------------------
Thaddy · 2025-06-28 14:19

Not yet, but I am on Ubuntu too.

----------------------------------------------------------------------
Gustavo 'Gus' Carreno · 2025-06-28 16:04

Hey Thaddy,

Quote from: Thaddy on June 28, 2025, 04:19:59 pmNot yet, but I am on
Ubuntu too.

AWESOME SAUCE!!

BTW, have I thanked you for your effort? If I haven't then:
Thank you soooooooooooooooooooooooooooo very much for picking this
up!!!!
Especially when this is a thing that only happens on my phne's
tethering. It's not like it would affect anyone but me...

Again: YOU DA MAN!! 

Cheers,
Gus