Downloading from the Internet
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
stansbkg · 2018-03-15 18:05

I have been using elements of Synapse to access web pages. Since about
March 1, it has been failing. Attached is a test program that works
with google.com and bing.com, but not with yahoo.com. I do not
understand.

----------------------------------------------------------------------
Xor-el · 2018-03-15 18:26

in my case, google and yahoo fails, only bing passes.
this is expected since it seems that "TFPCustomHTTPClient.SimpleGet"
does not support "https".

----------------------------------------------------------------------
rvk · 2018-03-15 19:22

Did you put the (correct) openssl dlls in your project directory?
Otherwise you don't have https support.

----------------------------------------------------------------------
stansbkg · 2018-03-16 13:57

The program was functioning for serveral months, but has failed on all
attempts for the past two weeks. I did nothing to the supporting
material. It has behaved the same on two different machines.

----------------------------------------------------------------------
wp · 2018-03-16 14:13

Are you sure that the webservices that you are using are still
existing? In the Internet, you never can be sure that what works today
will still work tomorrow, and most of all, in case of free services.

I once had written a little application to download and display
historic share prices from free Yahoo and Google financial services
(https://sourceforge.net/p/wp-laz/code/HEAD/tree/FinancialChart), and
from one day to the other first Yahoo and then a few months later
Google stopped this free service.

----------------------------------------------------------------------
rvk · 2018-03-16 14:17

Always remember to mention WHAT error-messages you get !!!

First of all, for SimpleGet you don't even need the "with
TFPHttpClient.Create(Nil) do".
SimpleGet already creates a TFPHttpClient for itself.

Second, httpsend is not needed. TFPHttpClient is NOT from Synapse.

Third, SimpleGet just does a HTTPMethod('GET',AURL,Stream,[200]);
It only accepts a 200 response. But yahoo gives an 301 back. (Google
probably too)

So if you want Redirection you need:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
AllowRedirect := true;Page := Get('http://www.yahoo.com');
So the code would be:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i        
with TFPHttpClient.Create(Nil) do        begin          AllowRedirect
:= true;//        Page := Get('http://www.google.com');//        Page
:= Get('http://www.bing.com');          Page :=
Get('http://www.yahoo.com');        end;(In this case you do need the
with TFPHttpClient.Create because you need to set the AllowRedirect
for it)

----------------------------------------------------------------------
stansbkg · 2018-03-16 17:25

Problem solved:  I changed the URL for the site I was accessing from
http:...  to https:... 
I had not noticed that both Google Chrome and Internet Explorer
antomatically make that change. The site I was accessing must have
made the change about the first of March. Before then, my program had
worked fine for several months. Since then, it had always failed.
Thanks.

----------------------------------------------------------------------
Ñuño_Martínez · 2018-03-19 17:02

The browsers "change" it because the web server is sending you a 301
or 307 state with the new URL.  Your program should deal with HTTP
states to work properly.