[SOLVED] HTTP Authentication lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- pcurtis · 2020-10-10 13:37 I am using Synapse and would like a simple example of how to authenticate / login to a web server. I know I need headers but I don't know how to prepare them. For example URL http:\\www.abc.def Username jbloggs Password somesecret Thanks in advance. ---------------------------------------------------------------------- Leledumbo · 2020-10-10 14:33 Authentication is not a one answer topic. There are various authentication methods that have been invented and implemented. If you're trying to act as a client to some existing website that's not yours, read first what method they support. It could range from just a basic auth to a 3-legged OAuth v2.0. Otherwise, I'm not using Synapse, but here is a simple client/server example of basic auth. Server: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc} uses  classes,  fphttpapp,  httpdefs,  httproute,  base64; procedure Whatever(ARequest: TRequest; AResponse: TResponse);var  LAuth: String;begin  LAuth := ARequest.Authorization;  if (Copy(LAuth,1,5) <> 'Basic') or (DecodeStringBase64(Copy(LAuth,7,Length(LAuth) - 7 + 1)) <> 'jbloggs:somesecret') then begin    AResponse.Code := 401;    Exit;  end;   AResponse.Content := 'This is my precious content';end; begin  HTTPRouter.RegisterRoute('/accessmethroughbasicauth',@Whatever);  Application.Port := 9000;  Application.Initialize;  Application.Run;end. Client: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  fphttpclient; begin  WriteLn(TFPHTTPClient.SimpleGet('http://jbloggs:s omesecret@localhost:9000/accessmethroughbasicauth'));end. Or you can use any of these cURL commands: Code: [Select]$ curl 'jbloggs:somesecret@localhost:9000/accessmethroughbasicauth' $ curl -u 'jbloggs:somesecret' 'localhost:9000/accessmethroughbasicauth' ---------------------------------------------------------------------- pcurtis · 2020-10-10 15:44 I have tried that method before and it didn't work. For example how would I login to this site? ---------------------------------------------------------------------- Thaddy · 2020-10-10 16:09 There are two things wrong. You should not try http:// But https:// For that you need a server certificate but these are free from e.g. https://letsencrypt.org Authentication for http is not recommended. (Basically: does not exist!!!!) You are a beginner, start from the right place. Then again, provided you use httpS, leledumbo's example should work. ---------------------------------------------------------------------- pcurtis · 2020-10-10 17:05 Is it possible to login to this site using rhat method? ---------------------------------------------------------------------- circular · 2020-10-10 17:18 Quote from: Thaddy on October 10, 2020, 06:09:41 pmYou are a beginner, start from the right place.That was unnecessary. Quote from: pcurtis on October 10, 2020, 07:05:44 pmIs it possible to login to this site using rhat method? Which site are you trying to log on? ---------------------------------------------------------------------- Thaddy · 2020-10-10 17:25 You can always test against https://example.com It it there for that purpose. ---------------------------------------------------------------------- rvk · 2020-10-10 17:26 There are lots of authentication methods. So we indeed need to know the site you are trying to login to. (Or you need to specify exactly what authentication method is used) ---------------------------------------------------------------------- Thaddy · 2020-10-10 17:28 There are rules. Follow my  link Rik. ---------------------------------------------------------------------- pcurtis · 2020-10-10 17:30 Try this site. https://forum.lazarus.freepascal.org How can I discover what method is required? ---------------------------------------------------------------------- rvk · 2020-10-10 17:32 Quote from: Thaddy on October 10, 2020, 07:28:24 pmThere are rules. Follow my  link Rik. You REALLY think following that link will help TS with his question from the opening post? (please read that one again) This wasn't about https versus http but about authentication itself (of which there are many variants). ---------------------------------------------------------------------- rvk · 2020-10-10 17:43 Quote from: pcurtis on October 10, 2020, 07:30:23 pmTry this site. https://forum.lazarus.freepascal.org How can I discover what method is required? This site doesn't use HTTP authentication via headers. You can logout and go back to the login page and look at the source. You see that posting password and username (sometimes along with some session id) will let you login to the site. Cookies are set and the site knows you are logged-in. So if you need to login to this site from code you can follow the same path (posting to the login-form page and saving the cookies). You can follow the examples here. https://forum.lazarus.freepascal.org/index.php?topic=28008.0 ---------------------------------------------------------------------- pcurtis · 2020-10-10 18:37 How do web browsers login to different sites and differentiate between authentication methods? ---------------------------------------------------------------------- rvk · 2020-10-10 18:50 Quote from: pcurtis on October 10, 2020, 08:37:51 pmHow do web browsers login to different sites and differentiate between authentication methods? Webbrowser follow instruction in code (html). So for this site, it is a login form and cookies. For Google sites it is OAuth2. For other sites it could be a authentication via headers. For example see https://en.m.wikipedia.org/wiki/Basic_access_authentication Of that there are again multiple versions. If you pick the wrong one as client, login won't work. So, look in the html of the login site to see if you can spot the method. (Header authentication is seldom used for forum sites.) ---------------------------------------------------------------------- pcurtis · 2020-10-10 19:12 Thanks. Any idea on how to do it for a vBulletin based site? ---------------------------------------------------------------------- rvk · 2020-10-10 19:18 Quote from: pcurtis on October 10, 2020, 09:12:00 pmThanks. Any idea on how to do it for a vBulletin based site? vBulletin also just works with cookies. Look at the login site and see where the login-data is posted (in the form tags). Then look at the link I gave you to a topic here, to login. You can also use Chrome (press f12 on the inlogscreen) to see what is posted and where you are redirected (under network tab). Here is a code-example for vBulletin I once made. https://forum.lazarus.freepascal.org/index.php/topic,49967.msg363705.h tml#msg363705 Ps. Was that in response to you or is that another P. Curtis? ---------------------------------------------------------------------- Remy Lebeau · 2020-10-10 20:33 Quote from: pcurtis on October 10, 2020, 09:12:00 pmThanks. Any idea on how to do it for a vBulletin based site? Again, LOOK AT THE HTML to see how that site handles login authentication. If it uses an HTML-based webform, then you have to POST the authentication credentials in the HTTP message body, typically in "application/x-www-webform-urlencoded" or "multipart/form-data" format.  The HTML will tell you which. If it uses HTTP-based authentication, then I suggest you invest in learning how to use your webbrowser's built-in debugger (if it has one) or an external sniffer like Wireshark or Fiddler, to see what the HTTP headers actually look like.  If the site asks for HTTP authentication, it will send an "WWW-Authenticate" header describing the authentication schemes it supports, which the webbrowser can then choose from and send back the credentials in the appropriate format in a subsequent "Authentication" header. ---------------------------------------------------------------------- zamronypj · 2020-10-10 21:51 I suggest you to look at browser developer tools (for Chrome or Firefox, press F12). While browser developer tool is open , do login manually and inspect what happens behind the scene in developer tools. If it is form-based login, look at URL at action attribute of form tag. Usually it will be POST method. Inspect request headers and body that browser send to URL for POST method and also inspect response that server returns (usually it returns a cookie in response when login is succesful. By observing request and response passed between browser and server, you will be able to replicate login with free pascal built in http client (in fact, browser is really just http client, a very sophisticated one) ---------------------------------------------------------------------- pcurtis · 2020-10-13 06:07 @rvk - Oops, yes it was me. I was probably sidetracked at the time. Anyway I have just tried it and get an error Quote--------------- Result = 500 --------------- POST /auth/ajax-login HTTP/1.0 Host: forum.vbulletin.com Keep-Alive: 300 Connection: keep-alive User-Agent: Mozilla/4.0 (compatible; Synapse) Content-Type: application/x-www-form-urlencoded Content-Length: 72 ---------------------------------------------------------------------- rvk · 2020-10-13 08:21 Quote from: pcurtis on October 13, 2020, 08:07:34 amAnyway I have just tried it and get an error You can't just use my snippet. You have to check what kind of login-page is used. Some use /auth/ajax-login. Others use login.php. And again others use a hashed password instead of passing password itself. I already saw that vBulletin 4.2.2 has a different login-page and procedure from 5.6.4. So each version can have it's own login-page to where you must post. What version of vBulletin are you trying to login to? (usually you can see that at the bottom) ---------------------------------------------------------------------- rvk · 2020-10-13 09:34 PS. I see from your result that you are trying to login to the forum of vbulletin itself. If you have an account there it should work. NOTES: You do need to set the useragent to something else. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     HTTPSend.Useragent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)';You ALSO need to add ssl_openssl to your uses clause and put the openssl dlls into your .exe directory. Make sure the bitness of the openssl matches the bitness of your program (i.e. 32bit=32bit, 64bit-64bit) It should work. I just tested it myself on vBulletin. But with older vBulletin versions you might need other code. Keep that in mind. ---------------------------------------------------------------------- pcurtis · 2020-10-13 11:01 OK. I get a HTTP 200. Getting close. 1. How do I know I am logged in? I see that the site has returned 4 cookies. If I want  to login again do I just save these cookies and then the next time I create the client just reload them? Or are there further steps? I used the vbulletin web site just because it was in the code. The site I want to use is older (v3.8.7) it uses /login.php?do=login Can I be a pest and ask if you know how to prepare the login text? And a BIG thank you for your help so far. Thanks. ---------------------------------------------------------------------- rvk · 2020-10-13 11:07 I think the privacyconsent=1 will make it that you can continue with the login (but I'm not sure). You just have to save the cookies in HTTPSend.Cookies.Text to a string and set it back again before doing a next GET of the page you want. You can see if you are logged in by accessing a page only a logged in person can access (for example the edit profile page). There are some examples available for the login.php but I'm on mobioe now so can't create an example at the moment. I can try later on this afternoon. ---------------------------------------------------------------------- pcurtis · 2020-10-13 12:44 Thank you. ---------------------------------------------------------------------- rvk · 2020-10-13 14:16 Here are the login for both versions. I could only test the last one on v4.2.2. v3.8.7 is really really old (begin 2011) so you might want to warn the administrator about that. There are several exploits for that version and I hope it's completely patched. For the latest version v5.6.4 (change the highlighted lines): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses httpsend, synautil, ssl_openssl; procedure TForm1.Button1Click(Sender: TObject);var  HTTPSend: THTTPSend;  Cookies: string;  ForumUrl, Username, Password: string;  TextFromDocument: string;begin  Memo1.Lines.Clear;   ForumUrl := 'https://forum.vbulletin.com'; // <-- CHANGE THIS  Username := 'username'; // <-- CHANGE THIS  Password := 'password'; // <-- CHANGE THIS   HTTPSend := THTTPSend.Create;  try   HTTPSend.Useragent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)';  HTTPSend.MimeType := 'application/x-www-form-urlencoded; charset=UTF-8';    WriteStrToStream(HTTPSend.Document,      ansistring('username=' + Username + '&password=' + Password +      '&privacyconsent=1&securitytoken=guest'));    HTTPSend.HTTPMethod('POST', ForumUrl + '/auth/ajax-login');    TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    if HTTPSend.ResultCode <> 200 then    begin      Memo1.Lines.Add('---------------');      Memo1.Lines.Add('Loginresult = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);      Memo1.Lines.Add('---------------');    exit;    end;     Cookies := HTTPSend.Cookies.Text; // you can save this for later. It will expire after a while though.    HTTPSend.Clear;    HTTPSend.Cookies.Text := Cookies;     HTTPSend.HTTPMethod('POST', 'https://forum.vbulletin.com/settings/profile'); // <-- CHANGE THIS  TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add('Result = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(HTTPSend.Headers.Text);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(TextFromDocument);    Memo1.Lines.Add('---------------');   finally    HTTPSend.Free;  end;end; And for v4.2.2 (which is almost identical except for the parameters and url): (it seems to have vb_login_md5password etc, but apparently they can be empty) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses httpsend, synautil, ssl_openssl; procedure TForm1.Button2Click(Sender: TObject);var  HTTPSend: THTTPSend;  Cookies: string;  ForumUrl, Username, Password: string;  TextFromDocument: string;begin  Memo1.Lines.Clear;   ForumUrl := 'url_of_main_forum_page/forum.php'; // <-- CHANGE THIS  Username := 'username'; // <-- CHANGE THIS  Password := 'password'; // <-- CHANGE THIS   HTTPSend := THTTPSend.Create;  try     HTTPSend.Useragent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)';    HTTPSend.MimeType := 'application/x-www- form-urlencoded; charset=UTF-8';    WriteStrToStream(HTTPSend.Document,      ansistring('vb_login_username=' + Username + '&vb_login_password=' + Password +      'securitytoken=guest&do=login&vb_login_md5password=&vb _login_md5password_utf=&cookieuser=1s='));    HTTPSend.HTTPMethod('POST', ForumUrl + '?do=login');    TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    if HTTPSend.ResultCode <> 200 then    begin      Memo1.Lines.Add('---------------');      Memo1.Lines.Add('Loginresult = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);      Memo1.Lines.Add('---------------');    exit;    end;     Cookies := HTTPSend.Cookies.Text; // you can save this for later. It will expire after a while though.    HTTPSend.Clear;    HTTPSend.Cookies.Text := Cookies;     HTTPSend.HTTPMethod('GET', 'url_of_control_panel_for_testing.php'); // <-- CHANGE THIS    TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add('Result = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(HTTPSend.Headers.Text);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(TextFromDocument);    Memo1.Lines.Add('---------------');   finally    HTTPSend.Free;  end; end; ---------------------------------------------------------------------- Thaddy · 2020-10-13 14:50 Rik, I appreciate almost of all your contributions, but plz help me to cut of http. Use https. Unless you have full control. ---------------------------------------------------------------------- rvk · 2020-10-13 14:54 Quote from: Thaddy on October 13, 2020, 04:50:29 pmI appreciate almost of all your contributions, but plz help me to cut of http. Use https. Unless you have full control. I did use https in my examples. And this isn't about running your own webserver. This is about logging into an existing web forum. So the choice of http(s) isn't an issue. ---------------------------------------------------------------------- pcurtis · 2020-10-13 15:01 Thanks I'll  take a look. Going back to my previous post - I can't verify that I am actually logged to vbulletin. I yrirf the following Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     ForumUrl := 'https://forum.vbulletin.com';    Username := myusername';    Password := 'mypassword';     HTTPSend := THTTPSend.Create;    try    HTTPSend.Useragent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)';      HTTPSend.MimeType := 'application/x-www-form-urlencoded';       WriteStrToStream(HTTPSend.Document,                       ansistring('username=' + Username + '&password=' + Password +                    '&privacyconsent=1&securitytoken=guest'));       Ok := HTTPSend.HTTPMethod('POST', ForumUrl + '/auth/ajax-login');       HTTPSend.HTTPMethod('GET', 'https://members.vbulletin.com/');       SetLength(sTEMP, HTTPSend.Document.Size);      HTTPSend.Document.Read(sTEMP[1], Length(sTEMP));      HTTPSend.Document.SaveToFile('out.html');      Memo1.Lines.Add(HTTPSend.Headers.Text);    finally      HTTPSend.Free;    end;  The output is Quote --------------- Result = 200 OK --------------- HTTP/1.1 200 OK Date: Tue, 13 Oct 2020 14:54:16 GMT Content-Type: application/json; charset=UTF-8 Content-Length: 48 Connection: keep-alive Set-Cookie: __cfduid=d877fc0d36dd653ee8826206f025d250a1602600856; expires=Thu, 12-Nov-20 14:54:16 GMT; path=/; domain=.vbulletin.com; HttpOnly; SameSite=Lax; Secure Strict-Transport-Security: max-age=31536000 X-Frame-Options: sameorigin Set-Cookie: vblm_sessionhash=fab382f5a2fb7989892f6c0f9137a632; path=/; domain=.vbulletin.com; secure; HttpOnly Set-Cookie: vblm_lastvisit=1602600856; expires=Wed, 13-Oct-2021 14:54:16 GMT; Max-Age=31536000; path=/; domain=.vbulletin.com; secure; HttpOnly Set-Cookie: vblm_lastactivity=1602600856; expires=Wed, 13-Oct-2021 14:54:16 GMT; Max-Age=31536000; path=/; domain=.vbulletin.com; secure; HttpOnly Expires: Sat, 1 Jan 2000 01:00:00 GMT Cache-Control: max-age=0,no-cache,no-store,post-check=0,pre-check=0 Pragma: no-cache X-Frame-Options: sameorigin Content-Security-Policy: frame-ancestors 'self' https://admin.vbulletin.com/ https://www.vbulletin.com/ https://members.vbulletin.com/ https://testsecureacceptance.cybersource.com/ https://secureacceptance.cybersource.com/ https://ssl.kaptcha.com/'; script-src * blob: 'unsafe-inline' 'unsafe-eval' ; object-src * Last-Modified: Tue, 13 Oct 2020 14:54:16 GMT X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Vary: Accept-Encoding CF-Cache-Status: DYNAMIC cf-request-id: 05c40d81bc000027a0b3269200000001 Expect-CT: max-age=604800, report-uri="https://report- uri.cloudflare.com/cdn-cgi/beacon/expect-ct" Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?lkg- colo=31&lkg-time=1602600856"}],"group":"cf-nel","max_age":604800} NEL: {"report_to":"cf-nel","max_age":604800} Server: cloudflare CF-RAY: 5e19e515fe6027a0-PRG --------------- {"errors":[["strikes","\/lostpw"]],"userid":"0"} If you notice the text Quote{"errors":[["strikes","\/lostpw"]],"userid":"0"} at the end. What stupid mistake am I making? ---------------------------------------------------------------------- rvk · 2020-10-13 15:06 For checking if you are logged in you can look at the code I posted. There I try to access the /settings/profile When not logged in you get an errir in html text. The first result should give a newtoken in json. You get an error with strikes. I'm not sure what that means and if you are logged in at that point. What does the /settings/profile show after this call (as shown in my example)? Edit: Ah, you removed that json result from your post. Then you can continue accessing the forum and see if it works. ---------------------------------------------------------------------- rvk · 2020-10-13 15:11 So wait 15 minutes. QuoteUse Login "Strikes" System Setting this to No disables the system that prevents a user (with a specific IP address) from logging in to an account after entering an incorrect password 5 times. After the first failed login attempt, the user receives the following message: You have used 1 out of 5 login attempts. After all 5 have been used, you will be unable to log in for 15 minutes. After the fifth failed login attempt, the user receives the following message: Wrong username or password. You have used up your failed login quota! Please wait 15 minutes before trying again. Don't forget that the password is case sensitive. After the fifth failed attempt, the user is locked out for fifteen minutes and and email will be sent to the email address associated with the account alerting them about the login failure. ---------------------------------------------------------------------- Remy Lebeau · 2020-10-13 16:18 Quote from: pcurtis on October 13, 2020, 01:01:49 pmOK. I get a HTTP 200. Getting close. 1. How do I know I am logged in? For webform-based logins, you have to look at the content that is being sent back, you can't rely on the HTTP response code alone.  The 200 simply means the HTTP request itself was successful, but the webform operation may have failed.  The content could be an error message, or even the login webform again. Quote from: pcurtis on October 13, 2020, 01:01:49 pm ---------------------------------------------------------------------- rvk · 2020-10-13 16:26 Quote from: Remy Lebeau on October 13, 2020, 06:18:32 pmFor webform- based logins, you have to look at the content that is being sent back, you can't rely on the HTTP response code alone.  The 200 simply means the HTTP request itself was successful, but the webform operation may have failed.  The content could be an error message, or even the login webform again. Yes, you're connect. I wanted to change my example to check the json- result of the login, to see if it contained the newtoken (or something) but didn't get to it. For the old forum you would need to check the html itself. ---------------------------------------------------------------------- pcurtis · 2020-10-14 02:55 Bingo! - I've done it. Thank you rvk for your help.