[SOLVED] Error in most basic TFPHTTPClient usage
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
guest58172 · 2016-07-07 22:33

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
Project1; uses fphttpclient; begin  try   
TFPHTTPClient.SimpleGet('https://api.github.com/repos/octocat/Hello-
World');  except    writeln('oops');  end;end.
but if you go to the url, you'll see that a JSON is well generated:

https://api.github.com/repos/octocat/Hello-World

What's wrong here ?

----------------------------------------------------------------------
Phil · 2016-07-07 22:44

Server is returning error 403.

-Phil

----------------------------------------------------------------------
molly · 2016-07-07 23:03

try this, and you're quit possibly up for the next error ;-)

Code: [Select]program simple;

{$MODE OBJFPC}{$H+}

uses
  fphttpclient, SysUtils;


procedure test;
begin
  try
    TFPHTTPClient.SimpleGet('https://api.github.com:443/repos/octocat/
Hello-World');
  except
    on E: Exception do
      writeLn('An exception was raised: ' + E.Message);
  end;
end;

begin
  Test;
end.

----------------------------------------------------------------------
guest58172 · 2016-07-07 23:05

Yes but for example libcurl succeeds to bring me the answer.

----------------------------------------------------------------------
molly · 2016-07-07 23:10

Quote from: BBasile on July 08, 2016, 01:05:04 amYes but for example
libcurl succeeds to bring me the answer.
Then print out the exception, as shown. I bet you did not even noticed
_all_ the changes in my code

----------------------------------------------------------------------
guest58172 · 2016-07-07 23:20

I didn't see the port but there's always  a 403, with or without a
port in the uri.

----------------------------------------------------------------------
Phil · 2016-07-07 23:24

Try Synapse or Indy or something else and see if that makes a
difference.

I tried on Mac with Cocoa's NSMutableURLRequest class and it returned
the JSON fine.

-Phil

----------------------------------------------------------------------
molly · 2016-07-07 23:25

And.... what does error 403 mean ?

Quote from: BBasile on July 08, 2016, 01:05:04 amYes but for example
libcurl succeeds to bring me the answer.
Then let libcurl be your guide and let it show you what it does to get
that result.

I don't believe SimpleGet() is going to cut it.

----------------------------------------------------------------------
guest58172 · 2016-07-07 23:30

I just tried with libcurl in D and I would prefer to use the FCL
client for a ObjPascal program, it was just as matter of comparison.

By the way it looks like Github API sucks a bit with authorizations,
but I still don't see why libcurl handle them automagically while the
FCL web client not...especially since I use on to get something on
another site that uses the https protocol too.

----------------------------------------------------------------------
d.ioannidis · 2016-07-07 23:32

Hi,

  in GitHub API https://developer.github.com/v3/#user-agent-required
says "requests MUST include a valid User-Agent header".

AFAIK, FPHTTPClient doesn't provide such header by default.

The following snippet works ...

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
Project1; {$MODE OBJFPC}{$H+}{$J-} uses  fphttpclient; var  cli:
TFPHTTPClient; begin  cli := TFPHTTPClient.Create(nil); 
cli.AddHeader('User-Agent','Mozilla/5.0 (compatible; fpweb)');  try  
try      Cli.Get('https://api.github.com/repos/octocat/Hello-World'); 
  except      writeln('oops');      readln;    end;  finally   
cli.free  end;end.
regards,

----------------------------------------------------------------------
guest58172 · 2016-07-07 23:35

Thx much for reading the doc for me.

----------------------------------------------------------------------
molly · 2016-07-07 23:38

Quote from: BBasile on July 08, 2016, 01:30:54 amBy the way it looks
like Github API sucks a bit with authorizations, but I still don't see
why libcurl handle them automagically while the FCL web client not.
And how many code does libcurl need for that... then compare that to
your one-liner.

Do you really expect .simpleget() f.e. to automatically redirect,
handle client certificates, play animations etc. for you ?

That's why i said use curl as your guide and let it do some verbose
logging to see what is required to retrieve that url. Then figure out
if SimpleGet() is able to do that (as i said, i don't believe it is).
so ditch the simpleget() and use TFPHTTPClient exactly in the same way
that curl does. In case that is failing, then there is valid reason to
suspect TFPHTTPClient failure/bug (or missing feature).

edit: ah, ic Dimitrios already figured out the exact reason for you

----------------------------------------------------------------------
d.ioannidis · 2016-07-07 23:46

Quote from: BBasile on July 08, 2016, 01:35:32 amThx much for reading
the doc for me.

It's an API. Shouldn't at least had a quick look how to use it ? 

( Couldn't believe that fphttpclient failed in such basic operation
.... )

----------------------------------------------------------------------
Phil · 2016-07-07 23:48

Quote from: Dimitrios Chr. Ioannidis on July 08, 2016, 01:32:41 amin
GitHub API https://developer.github.com/v3/#user-agent-required says
"requests MUST include a valid User-Agent header".

Good to know, although that strikes me as an unusual requirement. I've
used TFPHTTPClient with dozens of sites and have never run into that
requirement before. Perhaps it has more to do with them logging who's
retrieving what:

"We request that you use your GitHub username, or the name of your
application, for the User-Agent header value. This allows us to
contact you if there are problems."

----------------------------------------------------------------------
Phil · 2016-07-08 00:03

Quote from: Dimitrios Chr. Ioannidis on July 08, 2016, 01:46:19 am(
Couldn't believe that fphttpclient failed in such basic operation ....
)

Yes, BBasile should probably file a bug report. Looking at Synapse, it
appears it use "Mozilla/4.0 (compatible; Synapse)" as default agent.
And Indy appears to use "Mozilla/3.0 (compatible; Indy Library)". So
it seems reasonable to expect fcl-web to pass a default agent header.

I assume Apple's NSMutableURLRequest class does that, perhaps
identifying itself as Safari, since I didn't need to set any headers
for it to work correctly.

-Phil

----------------------------------------------------------------------
d.ioannidis · 2016-07-08 00:16

Quote from: Phil on July 08, 2016, 02:03:54 am So it seems reasonable
to expect fcl-web to pass a default agent header.

( Quoting  Remy Lebeau )

"The User-Agent header is optional in RFC 2616. It SHOULD be used by
clients, but it is not REQUIRED to be used. All of the major third-
party browsers/clients use it, but custom apps/APIs/bots/etc might
not. There are webservers that alter responses, or even reject
requests (which is annoying), based on the User-Agent provided (or
lack of one)."

PLz, fill a feature request ... ( IMO it's not a bug ).

regards,

----------------------------------------------------------------------
guest58172 · 2016-07-08 00:23

Quote from: Dimitrios Chr. Ioannidis on July 08, 2016, 01:46:19
amQuote from: BBasile on July 08, 2016, 01:35:32 amThx much for
reading the doc for me.

It's an API. Shouldn't at least had a quick look how to use it ? 

( Couldn't believe that fphttpclient failed in such basic operation
.... )

I've made the typical error that people do with tech docs and manuals.
They read only the section that they interested by and miss a lot of
things...

----------------------------------------------------------------------
guest58172 · 2016-07-08 16:53

Quote from: Phil on July 08, 2016, 02:03:54 amQuote from: Dimitrios
Chr. Ioannidis on July 08, 2016, 01:46:19 am( Couldn't believe that
fphttpclient failed in such basic operation .... )

Yes, BBasile should probably file a bug report. Looking at Synapse, it
appears it use "Mozilla/4.0 (compatible; Synapse)" as default agent.
And Indy appears to use "Mozilla/3.0 (compatible; Indy Library)". So
it seems reasonable to expect fcl-web to pass a default agent header.

I assume Apple's NSMutableURLRequest class does that, perhaps
identifying itself as Safari, since I didn't need to set any headers
for it to work correctly.

-Phil

http://bugs.freepascal.org/view.php?id=30359

Already closed.

----------------------------------------------------------------------
Phil · 2016-07-08 17:15

Quote from: BBasile on July 08, 2016, 06:53:29
pmhttp://bugs.freepascal.org/view.php?id=30359

Already closed.

Hmm, my source for Synapse does this in THTTPSend.Create:

  FUserAgent := 'Mozilla/4.0 (compatible; Synapse)';

Hence why I said that Synapse "appears" to set the user agent.

Not sure if I understand Michael's comment.

-Phil

----------------------------------------------------------------------
ChrisR · 2017-12-20 13:35

I realize this is an old thread. However, the function below takes
extends the example from Dimitrios Chr. Ioannidis to report the latest
github release

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
latestGitRelease(url: string): string;//Returns string for latest
release (error will return empty string)//example// latestGitRelease('
https://api.github.com/repos/rordenlab/dcm2niix/releases/latest');//wi
ll return// "v1.0.20171204"const     key = '"tag_name":"';var  s, e:
integer;  cli: TFPHTTPClient;begin  result := '';  cli :=
TFPHTTPClient.Create(nil);  cli.AddHeader('User-Agent','Mozilla/5.0
(compatible; fpweb)');  try    try      result := Cli.Get(url);   
except      result := '';    end;  finally    cli.free  end;  if
length(result) < 1 then exit;  s := posex(key, result);  if s < 1 then
begin     result := '';     exit;  end;  s := s+length(key);  e:=
posex('"', result, s);  if e < 1 then begin     result := '';   
 exit;  end;  result := copy(result, s, e-s);end;

----------------------------------------------------------------------
guest58172 · 2018-01-26 11:17

I used fcl-web since 2016. Recently i changed my main OS and realized
that fcl-web is not compatible with most recent OpenSSL versions.

I try to look at other libraries that can contain an HTTPS client and
the situation looks bad. All of them seem old and wont be be
compatible with the default OpenSSL i'll get from the distribution
package manager (FC27 for instance).

----------------------------------------------------------------------
Leledumbo · 2018-01-26 11:33

Quote from: BBasile on January 26, 2018, 12:17:46 pmI used fcl-web
since 2016. Recently i changed my main OS and realized that fcl-web is
not compatible with most recent OpenSSL versions.

I try to look at other libraries that can contain an HTTPS client and
the situation looks bad. All of them seem old and wont be be
compatible with the default OpenSSL i'll get from the distribution
package manager (FC27 for instance).
Manjaro (and perhaps Arch, too) has 1.0.2.n-1 and 1.1.0.g-1
installable in parallel. Yes, 1.1 protocol has some incompatibilities
with 1.0 protocol.

----------------------------------------------------------------------
guest58172 · 2018-01-26 12:00

I think you participated to the issue where i reported the bug. I
couldn't say it there but well, i say it here. That sux a lot. Being
able to do HTTPS request is quite important nowadays.