TfpHTTPClient - website form login lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- fpcoder · 2024-08-31 01:30 I needed to download some files for personal use, but failed to login to the webpage. No matter if I used TFPHttpClient, THttpSend or TIdHTTP. It really doesn't matter what library i choose, but for now lets lets talk about TFPHttpClient since thats FreePascals own lib. There was a discussion earlier, but the OP switched to HTTPSend of Synapse instead and never got fphttpclient going: https://forum.lazaru s.freepascal.org/index.php/topic,35682.msg236310.html SSL I noticed that he had not included units for SSL in his code, but was trying to use HTTPS. Maybe that was the reason. I added opensslsockets, but there is also ssockets, sslsockets, ssl_openssl, ssl_openssl11 and ssl_openssl3. Are any of them needed here? URL Encoding So if the url encoding is wrong it won't work. I use fphttp.FormPost for formdata together with a stringlist for login parameters, not single string. Reason be that the formdata must be urlencoded and https://wiki.freepascal.org/fphttpclient says it is done automatically only if a stringlist is used. I checked the returnvalues of HTTPEncode and EncodeURLElement and the dollar sign "$", "!" and "@" is not encoded but remains as is. Should it be like that? The login string 'ctl00$cph1$lg1$txtEmail' contains the "$". On the other hand, the custom wiki functions found here encode all these signs: https://wiki.freepascal.org/URL_encoding/decoding For the formdata below, I'm also not sure if to use "&" signs in front of the strings after the first one or if that is handled automatically or not needed at all. The result of my efforts is a HTTP 200 code and a not logged in webpage in the Response from fphttpclient. Trial code looks like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  uses  .. openssl, opensslsockets, fphttpclient, httpprotocol; procedure TForm1.btnLoginFpHTTPClick(Sender: TObject);const  URL = 'https://eoddata.com';var  HTTP: TFPHTTPClient;  page: string;  formdata: TStrings;  s:string;  Response: TStringStream;begin  formdata:= TStringList.Create;  formdata.values['ctl00$cph1$lg1$txtEmail'] := 'user@somemail.com';  //Username  formdata.values['ctl00$cph1$ls1$txtPassword'] := 'p@ss=w$rd!';  //Password  formdata.values['ctl00$cph1$ls1$chkRemember'] := '1';  //CheckBox, "Remember me"  formdata.values['ctl00$cph1$ls1$btnLogin'] := 'Login';  //Login Button   InitSSLInterface;  Response:= TStringStream.Create('');   HTTP:= TFPHTTPClient.Create(nil);  try    try      HTTP.AllowRedirect := True;      HTTP.AddHeader('User-Agent', 'Mozilla/5.0 (compatible; fpweb)');   //HTTP.UserAgent:='Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0';      HTTP.AddHeader('Content-Type', 'application/x-www-form-urlencoded');    HTTP.KeepConnection:=True;      //HTTP.Headers.Add('Accept- Encoding: identity');      //HTTP.Protocol:='1.1';      //Call GET before FormPost?      //HTTP.Get(URL);      HTTP.FormPost(URL, formdata, Response);      Response.Position:=0;  //Needed for Loading below.      Memo1.Lines.LoadFromStream(Response);      Memo1.Lines.Add(HTTP.ResponseHeaders.Text);    except      on E: EHttpClient do        ShowMessage(E.Message)      else        raise;  end;  finally    Response.Free;    HTTP.Free;  end;  Also got this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Cache- Control: privateContent-Type: text/html; charset=utf-8Server: Microsoft-IIS/7.5Set-Cookie: ASP.NET_SessionId=qh5hpbbctlwlgngfh5jjvwfu; path=/; HttpOnly; SameSite=LaxX-AspNet-Version: 4.0.30319X-Powered-By: ASP.NETDate: Sat, 31 Aug 2024 01:24:42 GMTContent-Length: 36362  HTML code from the webpage: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i
       
                               
MEMBER LOGIN
       
               
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
Email/Username:
Password:
                                                                                                                                                                                                       
RegisterRese nd ConfirmationForgot
       
---------------------------------------------------------------------- TRon · 2024-08-31 02:29 Quote from: kapibara on August 31, 2024, 03:30:34 amSSL I noticed that he had not included units for SSL in his code, but was trying to use HTTPS. Maybe that was the reason. I added opensslsockets, but there is also ssockets, sslsockets, ssl_openssl, ssl_openssl11 and ssl_openssl3. Are any of them needed here? No, they are not needed. including opensslsockets should be enough. Other then that: javascript. How the data is submitted and what requirements are needed usually depends on how the scripting handles things. And what you get back depends again on javascript. fphttpclient does not handle the scripting. You can do a simple test. In your browser turn off javascript completely and try to login as usual. If that works you got a chance, otherwise you simply don't have any. FWIW: usually if you are a member of such a site they provide an API. Sites uses that to prevent scraping. edit: ... and they do offer an xml web-interface that you use for easy access to 'their' data, for the highest tier ofc ... ---------------------------------------------------------------------- fpcoder · 2024-08-31 03:40 I turned off all javascript in Firefox and logged in without problem. So thats good. Because of the javascript on download buttons I am sceptical to if its possible to download the file I wanted, but that is not so important. This is mostly educational and I will update the wiki documentation when I understand how it works with form logins. ---------------------------------------------------------------------- TRon · 2024-08-31 12:17 Quote from: kapibara on August 31, 2024, 05:40:30 amI turned off all javascript in Firefox and logged in without problem. So thats good. Because of the javascript on download buttons I am sceptical to if its possible to download the file I wanted, but that is not so important. At first sight they try to do everything in their power to prevent you from doing so. Hence the different tiers, for soap/ftp access etc. But, you can always try to see what the browser does and where the files are actually located and what data goes over the connection before the download actually takes place. Always a long-shot but if you are able to do the same in the browser without javascript then you should be able to replicate that process with fphttpclient (or wget,curl, etc) QuoteThis is mostly educational and I will update the wiki documentation when I understand how it works with form logins. In that case take a closer look at the cookies as well. They need to match as well and are usually required to be present somewhere on the website/session. Edit: if the main aim is to automatically download the files that you are allowed f.e. on the free tier then you could always opt to fallback using selenium webdriver (There is a header for pascal). I could have sworn you opted that yourself as well in one of your posts. ---------------------------------------------------------------------- fpcoder · 2024-09-01 02:57 I found a webpage made for testing form logins. Will try fphttpclient there when I get some more time: https://practicetestautomation.com/practice-test-login/ Thanks for the help, Tron. You made me realize that the webpages innocent looking javascript was such a roadblock.