Problem with POST [BUG discovered]
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Vodnik · 2021-01-21 18:21

Hello!
This is my first step attempt to use REST API.
I've used this example as a base:
https://wiki.lazarus.freepascal.org/fphttpclient#Posting_JSON
Here is my code:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
http;uses  classes,  // TStringList  sysutils,  fphttpclient,
opensslsockets;var  Client: TFPHTTPClient;  Response:
TStringStream;begin  try    writeln('Initializing HTTP client...');  
Client := TFPHTTPClient.Create(nil);    Client.AllowRedirect := True; 
  writeln('HTTP client successfully initialized.');   
writeln('Looking for REST SDK API...');    try      if Pos('Rest SDK
API',Client.Get('https://10.10.111.211/restsdk'))<>0 then begin      
writeln('Rest SDK API is detected.');        writeln('Registering
application...');        Client.AddHeader('Content-
Type','application/json; charset=utf-8');       
Client.AddHeader('Cache-Control','no-cache');       
Client.AddHeader('Host','10.10.111.211');       
Client.RequestBody:=TRawByteStringStream.Create('{ "webhookURL" :
"https://enilpc2l0jwu.x.pipedream.net", "application" : { "name" :
"TestRestSdkClient", "token" :
"0e59b20476fca9042496084c99a1dc8f00f55a963146be13f6a9d6db0ee309af",
"busUnitName" : "DEFAULT" } }');        Response:=
TStringStream.Create('');        try          Client.Post('https://10.
10.111.211/restsdk/webapi/main/registerApplication',Response);       
 writeln('Response Code is ' + inttostr(Client.ResponseStatusCode)); 
       writeln(Client.ResponseStatusText);          if
Client.ResponseStatusCode=0 then begin            writeln('Application
registered');          end else begin            writeln('Application
not registered');          end;        except          on E: exception
do            writeln(E.Message);        end;      end else
writeln('Rest SDK API is not detected.');    except      on E:
exception do         writeln(E.Message);    end;  finally   
Client.RequestBody.Free;    Client.Free;    Response.Free  end; 
writeln('Press ENTER to exit');  readln();end.Problem is that I always
get response code 400.
I guess that body format may be not good, but the same REST request
from Postman works fine.
I tried to check this with Wireshark, but communication is encrypted.
Please help with ideas for debug.

----------------------------------------------------------------------
Vodnik · 2021-01-24 13:38

Just replacing Post procedure with simplePost solves the problem!
Can somebody explain the difference between them, please?

----------------------------------------------------------------------
Vodnik · 2021-01-24 16:18

Well, SimplePost just calls Post, setting KeepConnection to False:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i class
procedure TFPCustomHTTPClient.SimplePost(const URL: string;  const
Response: TStream); begin  With Self.Create(nil) do    try     
KeepConnection := False;      Post(URL,Response);    finally     
Free;    end;end;   
So how KeepConnection may cause "Bad request" response?
Is it the same as setting header "Connection":"keep-alive"?

----------------------------------------------------------------------
Leledumbo · 2021-01-25 00:56

Your endpoint either doesn't like the content-type or the host header.
It's endpoint specific behavior so you must consult the server side
documentation.

----------------------------------------------------------------------
Vodnik · 2021-01-25 14:51

You are right, problem is with the host header.
If I remove
        Code: Pascal  [Select][+][-]window.onload = function(){var x1
= document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Client.AddHeader('Host','10.10.111.211');then I guess host header is
created somehow automatically, and then my Post request succeeds.
Host header for this API is documented exactly as I use it.
I have compared the same Post requests to RequestBin.com from Lazarus
and from Postman but I didn't find any difference between the host
headers in that case.

----------------------------------------------------------------------
Vodnik · 2021-01-31 14:29

Well, dear friends, seems there is a bug in
TFPCustomHTTPClient.SendRequest procedure.
It adds "Host" header without any check for existing one.
Look at line 30: header "Host" is added without any check.
Later (line 38) it adds all headers, defined by programmer.
I'm also upset with "Cookies" header (line 48), which is added empty
in my case (I've cleared them with         Client.Cookies.Clear).
Resulting string S (with 2 "Hosts" headers, empty "Cookies" header and
two CRLF at the end) causes exception when calling FSocket.WriteBuffer
procedure.
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
TFPCustomHTTPClient.SendRequest(const AMethod: String; URI:
TURI); Var  PH,UN,PW,S,L : String;  I : Integer;  AddContentLength :
Boolean;  begin  S:=Uppercase(AMethod)+' '+GetServerURL(URI)+'
'+'HTTP/'+FHTTPVersion+CRLF;  UN:=URI.Username;  PW:=URI.Password;  if
(UserName<>'') then    begin    UN:=UserName;    PW:=Password;   
end;  If (UN<>'') then    begin    S:=S+'Authorization: Basic ' +
EncodeStringBase64(UN+':'+PW)+CRLF;   
I:=IndexOfHeader('Authorization');    If I<>-1 then     
RequestHeaders.Delete(i);    end;  if Assigned(FProxy) and
(FProxy.Host<>'') then    begin    PH:=FProxy.GetProxyHeaders;    if
(PH<>'') then      S:=S+PH+CRLF;    end;  S:=S+'Host: '+URI.Host;  If
(URI.Port<>0) then    S:=S+':'+IntToStr(URI.Port);  S:=S+CRLF; 
AddContentLength:=Assigned(RequestBody) and (IndexOfHeader('Content-
Length')=-1);  If AddContentLength then    AddHeader('Content-
Length',IntToStr(RequestBody.Size));  CheckConnectionCloseHeader;  For
I:=0 to FRequestHeaders.Count-1 do    begin    l:=FRequestHeaders[i]; 
  If AllowHeader(L) then      S:=S+L+CRLF;    end;  If
AddContentLength then   
FRequestHeaders.Delete(FRequestHeaders.IndexOfName('Content-
Length'));  if Assigned(FCookies) then    begin    L:='Cookie: ';   
For I:=0 to FCookies.Count-1 do      begin      If (I>0) then       
L:=L+'; ';      L:=L+FCookies[i];      end;    if AllowHeader(L) then 
    S:=S+L+CRLF;    end;  FreeAndNil(FSentCookies); 
FSentCookies:=FCookies;  FCookies:=Nil;  S:=S+CRLF;  if not Terminated
then    FSocket.WriteBuffer(S[1],Length(S));  If
Assigned(FRequestBody) and not Terminated then   
FSocket.CopyFrom(FRequestBody,FRequestBody.Size);end;          So
workaround for "Host" is: never add "Host" header manually!
How to completely clear "Cookies", I don't know.
In older examples I can see method Client.Clear, which is not
available in current version (Lazarus 2.0.10, FPC 3.2.0).
I expect this was the way to clear the body and headers completely.

----------------------------------------------------------------------
PascalDragon · 2021-02-01 08:15

Quote from: Vodnik on January 31, 2021, 03:29:00 pmSo workaround for
"Host" is: never add "Host" header manually!
Please file a bug report so that the class will at least complain if
someone adds Host manually.

Quote from: Vodnik on January 31, 2021, 03:29:00 pmHow to completely
clear "Cookies", I don't know.
Please file a bug report for this as well.

Quote from: Vodnik on January 31, 2021, 03:29:00 pmIn older examples I
can see method Client.Clear, which is not available in current version
(Lazarus 2.0.10, FPC 3.2.0).
I expect this was the way to clear the body and headers completely.

I can't find a Clear method it TFPHTTPClient back till at least 2.6.4.
I also doubt that it would have done what you expect as the headers
are built upon the request.

----------------------------------------------------------------------
Vodnik · 2021-02-01 09:08

I will do with bug report.
Quote from: PascalDragon on February 01, 2021, 09:15:49 amI can't find
a Clear method it TFPHTTPClient back till at least 2.6.4. I also doubt
that it would have done what you expect as the headers are built upon
the request.
Seems you are right. This is method of httpsend unit from Synapse.
So I have no idea how to clean the http client object between
responses in FPHTTPClient.

----------------------------------------------------------------------
lucamar · 2021-02-01 11:39

Quote from: Vodnik on January 31, 2021, 03:29:00 pmHow to completely
clear "Cookies", I don't know.
Untested, but:Code: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i FreeAndNil(MyHTTPClient.Cookies)should do the trick

----------------------------------------------------------------------
Vodnik · 2021-02-01 19:09

No, this causes Error: Can't take the address of constant expressions.

----------------------------------------------------------------------
lucamar · 2021-02-01 21:19

Quote from: Vodnik on February 01, 2021, 08:09:25 pmNo, this causes
Error: Can't take the address of constant expressions.
Then try the alternatives, either:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
MyHTTPClient.Cookies.Free;MyHTTPClient.Cookies := Nil;or:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i { var
Strings: TStrings; }Strings :=
MyHTTPClient.Cookies;Strings.Free;MyHTTPClient.Cookies := Nil;

----------------------------------------------------------------------
PascalDragon · 2021-02-02 08:06

If you look at the code you'll see that accessing the getter of
Cookies will recreate the FCookies field.

----------------------------------------------------------------------
Vodnik · 2021-02-06 20:04

Quote from: PascalDragon on February 02, 2021, 09:06:18 amIf you look
at the code you'll see that accessing the getter of Cookies will
recreate the FCookies field.
Yes, if FCookies=Nil.
But if previous (GET) request returns Cookies, it will not be Nil.
And after Client.Cookies.Free it will not be Nil.
As a result, next POST request contains empty Cookies header, causing
exception.

@lucamar
MyHTTPClient.Cookies := Nil    causes SIGSEGV error, I guess because
FCookies is declared private.

The very bad solution that I have found to make my application working
with REST API without errors is to call MyHTTPClient.Free after each
request.

----------------------------------------------------------------------
devEric69 · 2021-02-07 09:50

Overall, from memory, we do not really need to explicitely destroy the
cookie object (TCookie) itself, with fpWeb. It's fpWeb that does it
for us: for example, when we want to close a web session, we have to
send back its corresponding cookie empty, towards the browser, by
calling:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Self.Session.Terminate; Self.Session.InitResponse(AResponse); { it
returns an empty web session cookie to the client browser }
Or, we can put its value with ''.




However, inversely, we may need to create a cookie, if there are none
coming from the client in the Cookies "channel", and that it's no
longer contextually normal:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
[snip]o_cookie:= AResponse.Cookies.FindCookie('fpWebSessionName'); {
my fpWeb web session name; in Php, it would have been PHP_SESSID }if
o_cookie=nil then begin    o_cookie:= AResponse.Cookies.Add; { ask for
a new cookie }    o_cookie.Name:= 'fpWebSessionName';endelse begin  if
FbSession_AnonymousOrNamed_started then begin { i'm using an inherited
from TCustomSession }    o_cookie.Value:= GetGUID; { or anything else
}    o_cookie.Path:= '/';  { if it must cover all paths on our web
Domain Name Server }  end  else If
FbSession_AnonymousOrNamed_Terminated then begin    o_cookie.Value:=
''; { send an invalid web-cookie-session }    o_cookie.Path:=
'';end;[snip]

----------------------------------------------------------------------
Vodnik · 2021-02-07 19:08

Bug report 0038450: "Host" header is duplicated in POST request if one
is already set.

----------------------------------------------------------------------
devEric69 · 2021-02-08 08:50

@Vodnik, @..., sorry for my off-topic answer: I read too fast your
topic, not paying attention that you have a cookie problem, but on the
HTTP client side (with TFPCustomHTTPClient.SendRequest), not on the
response side i.e. on the server (with TFPWebModule or TCustomSession,
dealing with ARequest: TRequest and AResponse: TResponse;).

----------------------------------------------------------------------
Vodnik · 2021-02-08 13:13

@devEric69, no problem, finally I guessed that you are talking about
server side.