[SOLVED] FPHTTPClient + GitHub = :)
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
Trenatos · 2018-04-11 03:49

I'm working on a CLI tool, it's working nicely but I've run into a
strange issue.

I'm using FPHTTPClient to download various files, works fine, except
with GitHub.

This works: WriteLn(
TFPHTTPClient.SimpleGet('https://httpbin.org/uuid') );
This doesn't: WriteLn(
TFPHTTPClient.SimpleGet('https://api.github.com/repos/OpenBD/openbd-
core/releases/latest') );

I have the OpenSSL DLLs and that part seems fine, but when I try
hitting the GitHub API nothing seems to work.

ResponseStatusCode returns 0, and data just spits out a blank line.

I'm lost..  Anyone seen this behaviour before?

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i  
FHTTPClient := TFPHTTPClient.Create(nil);  try   
FHTTPClient.AllowRedirect:=true;    data := FHTTPClient.Get(
'http://api.github.com/repos/OpenBD/openbd-core/releases/latest' );  
WriteLn(FHTTPClient.ResponseStatusCode);    WriteLn( data );  finally 
  FHTTPClient.Free;  end;
The url works fine in my browser, it works fine in Insomnia (Desktop
REST utility)

----------------------------------------------------------------------
Cyrax · 2018-04-11 03:59

Maybe setting user agent variable helps?

----------------------------------------------------------------------
balazsszekely · 2018-04-11 04:39

Try this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
GetJSON(const AURL: String; out AJSON: TJSONStringType): Boolean;var 
Ms: TMemoryStream;  HTTPClient: TFPHTTPClient;begin  Result := False; 
Ms := TMemoryStream.Create;  try    HTTPClient :=
TFPHTTPClient.Create(nil);    try      HTTPClient.AllowRedirect :=
True;      HTTPClient.AddHeader('User-Agent', 'Mozilla/5.0
(compatible; fpweb)');      HTTPClient.AddHeader('Content-Type',
'application/json');      HTTPClient.HTTPMethod('GET', AURL, MS, []); 
    if HTTPClient.ResponseStatusCode = 200 then      begin        if
Ms.Size > 0 then        begin          MS.Position := 0;         
SetLength(AJSON, MS.Size);          MS.Read(Pointer(AJSON)^,
Length(AJSON));          Result := Length(AJSON) > 0;        end;    
end;    except      Result := False;    end;  finally    Ms.Free; 
end;end; var  JSON: TJSonStringType;begin  if
GetJSON('https://api.github.com/repos/OpenBD/openbd-
core/releases/latest', JSON) then    Writeln(JSON);end.

----------------------------------------------------------------------
molly · 2018-04-11 23:36

Quote from: Cyrax on April 11, 2018, 05:59:43 amMaybe setting user
agent variable helps?
Not maybe. Github API demands it.

----------------------------------------------------------------------
Trenatos · 2018-04-12 12:57

AHA! I hadn't seen that molly, thanks!

I haven't had a chance to try out your solution yet GetMem but
hopefully shortly.

----------------------------------------------------------------------
Trenatos · 2018-04-13 00:41

molly, GetMem, I'm adding you to my "owe beer to" list 

Indeed correct, it works with the user agent header, I did follow the
GitHub API info and use the application name

----------------------------------------------------------------------
Trenatos · 2018-04-19 14:56

I spoke too soon.

Adding the user agent solved the problem on Windows, but I'm still
having the same issue on OSX.

GetMem, your code gives the same result on Mac as my original code,
can't connect to GitHub and a response code of 0.

I did add openssl dylib to the same directory as the executable
(Though it's also installed on the system)

----------------------------------------------------------------------
Trenatos · 2018-04-19 15:13

Tested with https://httpbin.org/get and it does return and outputs the
json data, but not with GitHub.


WriteLn( HTTPClient.ResponseHeaders.Text ); outputs nothing for GitHub
WriteLn( HTTPClient.ResponseStatusText ); also outputs nothing for
GitHub

----------------------------------------------------------------------
Phil · 2018-04-19 15:36

Quote from: Trenatos on April 19, 2018, 04:56:47 pmGetMem, your code
gives the same result on Mac as my original code, can't connect to
GitHub and a response code of 0.

I did add openssl dylib to the same directory as the executable
(Though it's also installed on the system)

That code works fine here with macOS 10.13. What version of macOS is
your Mac running?

Note OpenSSL has been deprecated for years on Mac. Mac built-in
libraries work fine. See "Useful" source here for ns_url_request.pas
unit.

https://macpgmr.github.io

----------------------------------------------------------------------
Trenatos · 2018-04-19 15:40

El Capitan - 10.11.5

Bizarre that it works for you but not for me  :/

----------------------------------------------------------------------
Phil · 2018-04-19 15:42

Quote from: Trenatos on April 19, 2018, 05:40:44 pmEl Capitan -
10.11.5

I get the same error as you here on 10.11, so it's probably something
out of date. Try the ns_url_request.pas unit - it works fine.

https://developer.apple.com/library/content/documentation/Security/Con
ceptual/cryptoservices/SecureNetworkCommunicationAPIs/SecureNetworkCom
municationAPIs.html#//apple_ref/doc/uid/TP40011172-CH13-SW3

Here's all you need to do:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i        
HTTPClient := TNSHTTPSendAndReceive.Create;        try         
HTTPClient.Address := AURL;          Headers := TStringList.Create;  
      Headers.Add('User-Agent=Mozilla/5.0 (compatible; fpweb)');     
   Headers.Add('Content-Type=application/json');          if
HTTPClient.SendAndReceive(nil, Ms, Headers) then

----------------------------------------------------------------------
Trenatos · 2018-04-20 03:54

Thanks a bunch Phil, that did the trick, I also used that to stream a
file download, works like a charm.

I really appreciate the help