logon to a website using fphttpclient lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- Nitorami · 2025-07-24 14:53 I am an absolute newbie to web programming and don't know if what I am asking for is a trivial or a tricky task. Please forgive me if I am stupid. So, using a very simple example from the wiki, I managed to establish a web connection and to download a http site as string, using TFPHttpClient.SimpleGet and OpenSSL. Yay ! My first internet connection outside a browser, in just ten lines of code ! Now I would like to archive a very long thread from a community I am member of. The thread is split into 30000 individual pages, and clicking through these in a browser would take me ages. So I try to automate this using TFPHttpClient. However the thread is in the members only section and I receive an error 403. Nice! It is all logical. So I need to logon. How can I do this ? EDIT - Hm, I get the error 403 even when trying to access this forum's main page. So there may be other issues that just the logon. SimpleGet() however works on the lazarus forums. ---------------------------------------------------------------------- Thaddy · 2025-07-24 16:14 allowredirects := true. But 403 means you have no Rights. ---------------------------------------------------------------------- n7800 · 2025-07-24 19:18 Quote from: Nitorami on July 24, 2025, 04:53:20 pmNow I would like to archive a very long thread from a community I am member of. The thread is split into 30000 individual pages, and clicking through these in a browser would take me ages. So I try to automate this using TFPHttpClient. Be careful. Multiple requests to the server in a loop can alert administrators or DDoS protection systems. It will be especially unpleasant if you are blocked by IP, and you can no longer log in even through a browser )) ---------------------------------------------------------------------- n7800 · 2025-07-24 19:20 Quote from: Nitorami on July 24, 2025, 04:53:20 pmSo I need to logon. How can I do this ? As far as I know, you need to use "cookie" from your browser where you are already logged in. ---------------------------------------------------------------------- Nitorami · 2025-07-24 19:52 QuoteBe careful. Multiple requests to the server in a loop can alert administrators or DDoS protection systems. It will be especially unpleasant if you are blocked by IP, and you can no longer log in even through a browser ))Understood. I would run this slowly, and in case of problems would be able to contact the admin via personal message. As to the "cookie" - I have no idea how to get this from the browser. But even before thinking about the logon, I wonder why even get() of the forum's main page gives me an error 403. It does not require a logon. ---------------------------------------------------------------------- rvk · 2025-07-24 21:31 Quote from: Nitorami on July 24, 2025, 04:53:20 pmNow I would like to archive a very long thread from a community I am member of. The thread is split into 30000 individual pages, and clicking through these in a browser would take me ages. So I try to automate this using TFPHttpClient.You need to provide the website, or at least the type of forum software which is used there. Every forum software has it's own login procedure and there is no one way to do it. BTW. Often forum software also has XML or feed options but I'm not sure if that also goes for complete threads. ---------------------------------------------------------------------- Nitorami · 2025-07-25 01:17 The website is the old https://forums.winamp.com. It operates with vBulletin written in PHP. ---------------------------------------------------------------------- Thaddy · 2025-07-25 05:09 Spoof the user agent and you get a correct response:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode delphi}{$ifdef windows}{$apptype console}{$endif}{$H+}uses  Sysutils,  classes,  fphttpclient,  opensslsockets;var   Client: TFPHttpClient;  Response : string;begin  Client := TFPHttpClient.Create(nil);  writeln(client.httpversion);  try     // This is Edge     Client.AddHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.199 Safari/537.36 Edg/114.0.1823.67');     Client.AllowRedirect := true;  try      Response := Client.Get('https://forums.winamp.com');      writeln(Response);     except      on E:Exception do        writeln(e.message);    end;  finally    Client.Free;  end;end.403 gone... old fashioned spoofing. Used FPC trunk + OpenSSL 3 + Windows 11 Response: ---------------------------------------------------------------------- Nitorami · 2025-07-25 08:04 Aha ! Thanks Thaddy. I still get the error 403 from here but I see where this is going. As I said, I am a noob on this and had to read about user agent spoofing and its legitimante and fraudulent uses first. Interesting. Altogether, this seems to be more tricky than I thought. I may now adjust the UA string to make it work properly but then I'll still need to do the logon handshake with the server, and in the end run the risk of being mistaken as a bot and banned ? ---------------------------------------------------------------------- rvk · 2025-07-25 08:15 Quote from: Nitorami on July 25, 2025, 10:04:18 amAha ! Thanks Thaddy. I still get the error 403 from here but I see where this is going.I also get the 403 from the exact code from Thaddy. With curl I do see that it uses Cloudflare. And even if you get the 403 back, there is content in the body. But that just says the usual "Just a moment..." from Cloudflare. I haven't look any further yet... but the fact your dealing with Cloudflare should give some pointers. ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2025-07-25 08:19 Hey Nitorami, If you wanna pursue the collection of knowledge that you'll get from doing your own tool, I'm all for it. I do like to re-invent the wheel when I'm trying to master a subject. I also look for some less brain wrecking options, if only, to have them in my back pocket in case plan A through E go flop I'm not sure what operating system you're using, it's not clear from this thread. But if you have access to wget, it would be a lot easier. I can even say it would be a one liner Especially because wget is known for it's archival features. If you use the correct set of switches, it will login, or use login cookies from your browser and it will download those 30K message thread, as slow as you tell it, cuz even that has a switch. With what ever user agent you wanna mask it to. Hope this gives you a last chance plan option, just in case fphttpclient proves to be a bit too challenging. Cheers, Gus ---------------------------------------------------------------------- Thaddy · 2025-07-25 11:00 Well, if there is a correct openssl version... my code works and with proof. Without the spoof I also get 403, btw. I suspect that the server denies access with lower tls versions (maybe even tls1.2, but I have to research that). Which is common, correct and proper. Always make sure your security is in order. Too many people here ignore that and so get punished for it. Which I really satanic like.  Maybe will learn. Security protocols are a fast moving target: what worked yesterday will possibly not work today. And very few on this forum are up-to-date. ---------------------------------------------------------------------- Thaddy · 2025-07-25 11:18 If you can't use openssl3.x try the lts version 1.1.1s, note the s. The handshake for that website is tls1.3. ---------------------------------------------------------------------- Thaddy · 2025-07-25 11:32 Quote from: rvk on July 25, 2025, 10:15:40 amI haven't look any further yet... but the fact your dealing with Cloudflare should give some pointers. Cloudflare is not the issue. It is 2 ways: the user-agent string and tls 1.3 support. See my screenshot. Attached captured page. Full page in zip. Alas, I can do nothing more than that, because it works. Capture code:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode delphi}{$ifdef windows}{$apptype console}{$endif}{$H+}uses  Windows,  Sysutils,  classes,  fphttpclient,  opensslsockets;var   Client: TFPHttpClient;  //Response : string;  Page:TStringlist;begin  Client := TFPHttpClient.Create(nil);  Page := TStringlist.Create;  try     Client.AddHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.199 Safari/537.36 Edg/114.0.1823.67');     Client.AllowRedirect := true;  try      Page.text := Client.Get('https://forums.winamp.com');      //writeln(Page.text);      Page.SaveToFile('response.html');      ShellExecute(0, 'open', 'response.html', nil, nil, SW_SHOWNORMAL);// x-platform: openurl and add interfaces and lcintf to uses. remove windows.     except      on E:Exception do        writeln(e.message);    end;  finally    Page.Free;    Client.Free;  end;end. ---------------------------------------------------------------------- Nitorami · 2025-07-25 11:59 @Gustavo: I am on windows 10, and have read about wget. Maybe I should give it a try. In general, I have always tried to do things on the bottom level, in pascal, which I simply know best. Where others used batch files, I wrote a pascal program.... not the most efficient way but it usually worked. Now with web protocols the complexity may well throw me off. We'll see. It is not easy to get low level help on the Internet on these topics either. Most guidances assume a web browser in the first place - advices such as "clear the cache", "refresh your certificate store" etc. @Thaddy: Complex as these things are, I would find it well possible that your code works in your place, but not in mine - due to different IP, different god-knows-which-machine settings, locale etc. But I admit I used the first SSL version I could get hold of, from https://Indy.fulgan.com/SSL/. I found the site somewhere on the wiki. It seems these are 1.0 versions, so I'll try SSL3. ---------------------------------------------------------------------- Thaddy · 2025-07-25 12:05 If you are on windows I can attach the necessary dll's for openssl if you want. Let me know. Try either 3 or 1.1.1s binaries. We should get it to work for you too. Btw: on linux you may need a different spoof. This spoof is for windows. On Linux, the proper version is likely installed already since it comes with the security patches. I just tested with fpc 3.2.2 and that does NOT work. You need fpc 3.3.1 (or maybe 3.2.3 = fixes or 3.2.4 RC1). That is due to changes in protocol support. Because it is a security issue, I guess it may have been back-ported to 3.2.4 but I am not sure. ---------------------------------------------------------------------- Nitorami · 2025-07-25 12:55 Yes I am on windoze. I found SSL3.0.14 binaries here https://github.com/IndySockets/OpenSSL-Binaries but they seem to have renamed the dlls. Previously it was ssleay.dll and libeay.dll, now I find libcrypto-3-x64.dll and libssl-3-x64.dll. Are these the ones I should use ? And how would fphttpclient find them under this name ? Or have the names been amended in newer versions of fpc / fphttpclient as well ? (I am still at fpc 3.2.2) ---------------------------------------------------------------------- Thaddy · 2025-07-25 12:58 Yes, you can use those and the names are correct too, but read on: https://slproweb.com/products/Win32OpenSSL.html is more current. Btw: I tested 3.2.2 for you and that does NOT work out of the box. (because there is no openssl3 support) I checked the sources from 3.2.3 and these are fine. It works with 3.2.3 (fixes ) and 3.2.4 (RC1) because it is indeed back-ported from trunk so it will be in the next 3.2.4 release. The release candidate is really stable, so install that if you do not want to use trunk. That is really all I can do for you atm. ---------------------------------------------------------------------- Nitorami · 2025-07-25 13:54 Thanks Thaddy. I downloaded 3.3.1 win32 from the ftp site but I guess that was wrong, I will need the 64bit version to work with OpenSSL, correct ? ---------------------------------------------------------------------- rvk · 2025-07-25 14:14 Quote from: Nitorami on July 25, 2025, 03:54:32 pmI downloaded 3.3.1 win32 from the ftp site but I guess that was wrong, I will need the 64bit version to work with OpenSSL, correct ? Depends on the bitness of your compile under Windows. I assume 64 bit for Lazarus. ---------------------------------------------------------------------- Thaddy · 2025-07-25 14:16 No, in that case you just need the 32 bit binaries for openssl. But I would recommend downloading a 64 bit version of fpc. Try the second in this list: https://downloads.freepascal.org/fpc/beta/3.2.4-rc1/x86_64-win64/ I can't try that because I always compile from source. (except the cross compilers, sometimes) ---------------------------------------------------------------------- Nitorami · 2025-07-25 14:22 Alright, I'll try that one. Just attempted to install Lazarus stable with FPC3.2.4 via FPCUpDeluxe but got a fatal error: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i fpcupdeluxe: ERROR: FPC Native Installer (CheckModule: FPC): Repo tag and desired tag do not match.fpcupdeluxe: ERROR: FPC Native Installer (CheckModule: FPC): The desired release_3_2_4 is different from local repository-tag.  ERROR: Fpcupdeluxe fatal error !Sequencer (FPC): Failure running fpcupdeluxe: error executing sequence FPCSequencer (Defaultwin32): Failure running fpcupdeluxe: error executing sequence Defaultwin32 ---------------------------------------------------------------------- rvk · 2025-07-25 14:26 You can also just use OpenSSL 1.1.1w That will work with older FPC. Direct link https://wiki.overbyte.eu/arch/openssl-1.1.1w-win64.zip From https://wiki.overbyte.eu/wiki/index.php/ICS_Download ---------------------------------------------------------------------- Thaddy · 2025-07-25 14:31 I can't test that, but that version looks OK and the names are also correct.[edit] and works. Still I'd recommend openssl 3 ---------------------------------------------------------------------- Nitorami · 2025-07-25 14:33 I installed FPC 3.2.4 from Thaddys link and now it works with my existing OpenSSL installation. Yeah ! Now the next challenge will be to logon somehow. ---------------------------------------------------------------------- Thaddy · 2025-07-25 14:37 ---------------------------------------------------------------------- Nitorami · 2025-07-25 14:38 Yeah, thanks a lot so far. Without assistance, I would have given up already. ---------------------------------------------------------------------- Thaddy · 2025-07-25 14:40 I would ptobably also given up, but I have the time to keep persevering. ---------------------------------------------------------------------- rvk · 2025-07-25 16:19 If you want to go further and login then you need to look at my old post from a long time ago. vBulletin forums use auth/login ajax login. I saw that it is vBulletin 6.1.3 but the 5.6.4 I tested this on a few years ago also used /auth/ajax-login (and the 6.1.3 uses it too if I look at the debugging in Brave). https://forum.lazarus.freepascal.org/index.php/topic,51730.msg380623.h tml#msg380623 Quote from: rvk on October 13, 2020, 04:16:26 pmHere are the login for both versions. I could only test the last one on v4.2.2. v3.8.7 is really really old (begin 2011) so you might want to warn the administrator about that. There are several exploits for that version and I hope it's completely patched. For the latest version v5.6.4 (change the highlighted lines): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses httpsend, synautil, ssl_openssl; procedure TForm1.Button1Click(Sender: TObject);var  HTTPSend: THTTPSend;  Cookies: string;  ForumUrl, Username, Password: string;  TextFromDocument: string;begin  Memo1.Lines.Clear;   ForumUrl := 'https://forum.vbulletin.com'; // <-- CHANGE THIS  Username := 'username'; // <-- CHANGE THIS  Password := 'password'; // <-- CHANGE THIS   HTTPSend := THTTPSend.Create;  try   HTTPSend.Useragent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)';  HTTPSend.MimeType := 'application/x-www-form-urlencoded; charset=UTF-8';    WriteStrToStream(HTTPSend.Document,      ansistring('username=' + Username + '&password=' + Password +      '&privacyconsent=1&securitytoken=guest'));    HTTPSend.HTTPMethod('POST', ForumUrl + '/auth/ajax-login');    TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    if HTTPSend.ResultCode <> 200 then    begin      Memo1.Lines.Add('---------------');      Memo1.Lines.Add('Loginresult = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);      Memo1.Lines.Add('---------------');    exit;    end;     Cookies := HTTPSend.Cookies.Text; // you can save this for later. It will expire after a while though.    HTTPSend.Clear;    HTTPSend.Cookies.Text := Cookies;     HTTPSend.HTTPMethod('POST', 'https://forum.vbulletin.com/settings/profile'); // <-- CHANGE THIS  TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add('Result = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(HTTPSend.Headers.Text);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(TextFromDocument);    Memo1.Lines.Add('---------------');   finally    HTTPSend.Free;  end;end; And for v4.2.2 (which is almost identical except for the parameters and url): (it seems to have vb_login_md5password etc, but apparently they can be empty) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses httpsend, synautil, ssl_openssl; procedure TForm1.Button2Click(Sender: TObject);var  HTTPSend: THTTPSend;  Cookies: string;  ForumUrl, Username, Password: string;  TextFromDocument: string;begin  Memo1.Lines.Clear;   ForumUrl := 'url_of_main_forum_page/forum.php'; // <-- CHANGE THIS  Username := 'username'; // <-- CHANGE THIS  Password := 'password'; // <-- CHANGE THIS   HTTPSend := THTTPSend.Create;  try     HTTPSend.Useragent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)';    HTTPSend.MimeType := 'application/x-www- form-urlencoded; charset=UTF-8';    WriteStrToStream(HTTPSend.Document,      ansistring('vb_login_username=' + Username + '&vb_login_password=' + Password +      'securitytoken=guest&do=login&vb_login_md5password=&vb _login_md5password_utf=&cookieuser=1s='));    HTTPSend.HTTPMethod('POST', ForumUrl + '?do=login');    TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    if HTTPSend.ResultCode <> 200 then    begin      Memo1.Lines.Add('---------------');      Memo1.Lines.Add('Loginresult = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);      Memo1.Lines.Add('---------------');    exit;    end;     Cookies := HTTPSend.Cookies.Text; // you can save this for later. It will expire after a while though.    HTTPSend.Clear;    HTTPSend.Cookies.Text := Cookies;     HTTPSend.HTTPMethod('GET', 'url_of_control_panel_for_testing.php'); // <-- CHANGE THIS    TextFromDocument := ReadStrFromStream(HTTPSend.Document, HTTPSend.Document.Size);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add('Result = ' + HTTPSend.ResultCode.ToString + ' ' + HTTPSend.ResultString);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(HTTPSend.Headers.Text);    Memo1.Lines.Add('---------------');    Memo1.Lines.Add(TextFromDocument);    Memo1.Lines.Add('---------------');   finally    HTTPSend.Free;  end; end; ---------------------------------------------------------------------- rvk · 2025-07-25 16:48 I saw my previous answer used Synapse for HTTPClient. Here... I rewrote the example using fphttpclient. Change the username and password and it should login correctly. After the login, it retrieves a profile page. If it contains the text "Cannot access profile settings as a guest user." there was a problem. Otherwise the login was succesfull. You can use the same instance of Client (which now contains the correct session-cookies) to scrape the private thread. I tested it with a dummy spam account I made there, so it does work. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode delphi}{$ifdef windows}{$apptype console}{$endif}{$H+}uses  Sysutils,  classes,  fphttpclient,  opensslsockets;var  ForumUrl, FormData, Username, Password: string;  Client: TFPHttpClient;  Response : string;  Cookies: string;begin   ForumUrl := 'https://forums.winamp.com'; // <-- CHANGE THIS  Username := 'username'; // <-- CHANGE THIS  Password := 'password'; // <-- CHANGE THIS   FormData := 'username=' + Username + '&password=' + Password + '&privacyconsent=1&securitytoken=guest';   Client := TFPHttpClient.Create(nil);  try    writeln(client.httpversion);    Client.AddHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)');    Client.AllowRedirect := true;    try      Response := Client.FormPost(ForumUrl + '/auth/ajax-login', FormData);      // writeln(Response);       Cookies := Client.Cookies.Text; // you can save this for later. It will expire after a while though.      writeln('Login success ' + Client.ResponseStatusCode.ToString + ' ' + Client.ResponseStatusText);       Response := Client.Get(ForumUrl + '/settings/profile');       if Pos('Cannot access profile', Response) > 0 then         writeln('Get PROBLEM "Cannot access profile" ' + Client.ResponseStatusCode.ToString + ' ' + Client.ResponseStatusText)      else      begin        writeln('Get success ' + Client.ResponseStatusCode.ToString + ' ' + Client.ResponseStatusText);        writeln(Response);      end;       //      // here you can write your loop to retrieve the private thread      //     except      on E:Exception do      begin        writeln(e.message);      end;    end;  finally    Client.Free;    readln;  end;end. ---------------------------------------------------------------------- Nitorami · 2025-07-25 17:23 I'll try this when back home. Two questions: I assume the cookie is valid over the lifetime of the Client, right ? I mean, aside from possible timeouts on the server side. I cannot store and reuse the cookie. And how do browsers know of "ajax" or not "ajax", respectively what exactly each software in its specific version expects ? ---------------------------------------------------------------------- rvk · 2025-07-25 17:31 Quote from: Nitorami on July 25, 2025, 07:23:47 pmI assume the cookie is valid over the lifetime of the Client, right ? I mean, aside from possible timeouts on the server side. I cannot store and reuse the cookie.Yes, you can store the cookies and use them later. But I thought they eventually time out. Not sure if that's after a couple of hours or days. And maybe they don't timeout. I haven't tested it. QuoteAnd how do browsers know of "ajax" or not "ajax", respectively what exactly each software in its specific version expects ? Not sure I understand your question. Ajax (Asynchronous JavaScript and XML) is just a method in javascript to call that login dialog. You see the login is not a seperate page but a popup dialog which calls a back-end page (via Ajax) to login. We just call that page here manually to get the cookies and login. ---------------------------------------------------------------------- Nitorami · 2025-07-25 18:06 Alright, I was under the impression that ajax is just one of many protocols to do the job... when I was a child, Ajax was a washing powder brand  Alright, I'll try your code now. ---------------------------------------------------------------------- Nitorami · 2025-07-25 19:13 Brilliant, I can logon and get a cookie. Let's first try to access a public thread. I set the timeout to 10000 because I know the forum's sometimes sluggish, probably due to AI crawlers. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Client.IOTimeout := 10000;writeln ('Cookie : ',Client.Cookies.Text);Response := Client.Get('https://forums.winamp.com/forum/skinning-and- design/skinning-tips-and-tricks/13367-skinning-software');writeln ('Response on Get: ',Response);readln;  Works ! Now let's try the private thread - error 403 again. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 1.1Login success 200 OKGet success 200 OKNow we are logged in and can do somethingCookie : PHPSESSID=1530fad3cda5274b40859eead8b8590fpath=/secu reHttpOnlySameSite=Strict Unexpected response status code: 403 Hm ! I logged out from the browser just in case the server does not accept multiple logins from the same IP, but same result. I had copied the link to the private thread directly from the browser. Could it by that it is dynamically changed ? But when I start a new browser session and copy the address back from my code into the browser, it works ! ---------------------------------------------------------------------- rvk · 2025-07-25 19:20 I can't test it anymore because they banned me. No idea why. I have no clue if using the code made them ban me. QuoteYou have been permanently banned by NJK for the following reason: spammer. ---------------------------------------------------------------------- Nitorami · 2025-07-25 19:32 LOL, NJK  Yes, they are quite strict these days and sometimes ban new "users" before they could even post. The forum has become a rather deserted place, and if there is a new member, it will be a spammer 99.9%. This is really a lot different from the freepascal forum, where I've never seen this. Not sure what makes the difference. But the mods of the winamp forum have no admin powers, and no automatic detection seems to be in place; they don't even have scripts to delete the spammer's crap, they need to do it all manually. I'll do some research on my own, and if I don't get anywhere, I may ask NJK to reactivate your account. I think I can explain the case to him. ---------------------------------------------------------------------- rvk · 2025-07-25 19:35 I tried it with my old account from 5 years ago from https://forum.vbulletin.com. The login went successful but accessing https://forum.vbulletin.com/forum/general/a-place-to-test-things-out also gave me a 403. Not sure what could trigger it. Need to dive in deeper later on. ---------------------------------------------------------------------- Nitorami · 2025-07-25 19:51 Did not know this place, and my winamp logon does not work there anyway. ---------------------------------------------------------------------- rvk · 2025-07-25 19:59 Comment out the retrieval of the profile. After that it worked for me at vbulletin. It seems that only the first page after logging in works. The second gives 403. See below. Comment out the profile retrieval. The first after that works. The second breaks with 403. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i       Response := Client.FormPost(ForumUrl + '/auth/ajax-login', FormData);      // writeln(Response);       Cookies := Client.Cookies.Text; // you can save this for later. It will expire after a while though.      writeln('Login success ' + Client.ResponseStatusCode.ToString + ' ' + Client.ResponseStatusText);       {      Response := Client.Get(ForumUrl + '/settings/profile');      if Pos('Cannot access profile', Response) > 0 then         writeln('Get PROBLEM "Cannot access profile" ' + Client.ResponseStatusCode.ToString + ' ' + Client.ResponseStatusText)      else      begin        writeln('Get success ' + Client.ResponseStatusCode.ToString + ' ' + Client.ResponseStatusText);        // writeln(Response);      end;    }       writeln(Cookies);       // https://forum.vbulletin.com/forum/general/a-place-to-test-things-out       Response := Client.Get(ForumUrl + '/forum/general/a-place-to-test- things-out');      writeln('ok 1');     // this works ok       Response := Client.Get(ForumUrl + '/forum/general/a-place-to-test- things-out');      writeln('ok 2');     //    above crashes with 403       writeln(Response);       //      // here you can write your loop to retrieve the private thread      // ---------------------------------------------------------------------- Nitorami · 2025-07-25 20:04 How odd. I commented out the call to the profile, and indeed don't get an error 403 anymore, but instead an "Error reading data from socket". Oh wait, after logging out from the browser it seems to work. Will write a loop and see what happens when I access a few pages in succession. Hm, "Error reading data from socket" again, on the first try already. Seeing how fragile and random this all is, I really wonder how the Internet works in the first place... On the other hand, access via browser currently also takes ages. ---------------------------------------------------------------------- rvk · 2025-07-25 21:20 Ok, here is a possible solution. DON'T reuse an existing TFPHTTPClient once you have retrieved a page. It messes up the cookies. I'm not sure why but a lot of cookies are gone after the first page retrieval. Here is an example. I split the TFPHTTPClient Get and Post into separate functions so they are not reused. For me, on vbulletin it works. (Change the 4 variables at the beginning for your need) (bedtime for me ) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode delphi}{$ifdef windows}{$apptype console}{$endif}{$H+}uses  Sysutils,  classes,  fphttpclient,  opensslsockets; procedure DoHTTPPost(URL, DataString: String; var Cookies: String);var  myClient: TFPHttpClient;begin  myClient := TFPHttpClient.Create(nil);  try    myClient.AddHeader('user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)');    myClient.Cookies.Text := Cookies;    myClient.FormPost(Url, DataString);    Cookies := myClient.Cookies.Text;    writeln('Post ok ' + myClient.ResponseStatusCode.ToString + ' ' + myClient.ResponseStatusText);  finally    myClient.Free;  end;end; function DoHTTPGet(URL: String; var Cookies: String): String;var  myClient: TFPHttpClient;begin  myClient := TFPHttpClient.Create(nil);  try    myClient.AddHeader('user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)');    myClient.Cookies.Text := Cookies; // ONLY FILL THEM    Result := myClient.Get(URL);    // Cookies := myClient.Cookies.Text;    DONT DO THIS !!!!!!!!!    writeln('Get ok ' + myClient.ResponseStatusCode.ToString + ' ' + myClient.ResponseStatusText);  finally    myClient.Free;  end;end; var  ForumUrl, GetThreadYouWant, FormData, Username, Password: string;  Response : string;  Cookies: string;begin   ForumUrl := 'https://forums.winamp.com'; // <-- CHANGE THIS  GetThreadYouWant := '/forum/general/a-place-to-test-things-out'; // <-- CHANGE THIS  Username := 'username'; // <-- CHANGE THIS  Password := 'password'; // <-- CHANGE THIS   FormData := 'username=' + Username + '&password=' + Password + '&privacyconsent=1&securitytoken=guest';   Cookies := '';  DoHTTPPost(ForumUrl + '/auth/ajax-login', FormData, Cookies);  Response := DoHTTPGet(ForumUrl + GetThreadYouWant, Cookies);  Response := DoHTTPGet(ForumUrl + GetThreadYouWant, Cookies);  Response := DoHTTPGet(ForumUrl + GetThreadYouWant, Cookies);   writeln('Press enter to quit');  readln; end. QuotePost ok 200 OK Get ok 200 OK Get ok 200 OK Get ok 200 OK Press enter to quit ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2025-07-26 02:34 Hey Nitorami, Quote from: Nitorami on July 25, 2025, 01:59:17 pm@Gustavo: I am on windows 10, and have read about wget. Maybe I should give it a try. In general, I have always tried to do things on the bottom level, in pascal, which I simply know best. Where others used batch files, I wrote a pascal program.... not the most efficient way but it usually worked. Now with web protocols the complexity may well throw me off. We'll see. Like I said, if it's the pursuit of knowledge, then hack away at it with full gusto!!! Quote from: Nitorami on July 25, 2025, 01:59:17 pmIt is not easy to get low level help on the Internet on these topics either. Most guidances assume a web browser in the first place - advices such as "clear the cache", "refresh your certificate store" etc. Indeed... Not very programmer friendly Cheers, Gus ---------------------------------------------------------------------- Thaddy · 2025-07-26 03:22 The cookie issue with TFpHttpClient is known and recently fixed in trunk but exists in 3.2.2. I have to check if it is fixed in 3.2.4 ---------------------------------------------------------------------- rvk · 2025-07-30 10:55 Quote from: Thaddy on July 26, 2025, 05:22:28 amThe cookie issue with TFpHttpClient is known and recently fixed in trunk but exists in 3.2.2. I have to check if it is fixed in 3.2.4 They completely broke cookie handling in trunk at the moment. FCookies is deprecated but they neglected to adjust the source where cookies are still put in the old FCookies: TStringList in DoCookies with FCookies.Add(). So when there are cookies, trunk will just give an exception that FCookies is nil. Because of the swap of FCookies to FSentCookies and FCookies := nil in SendRequest. And after that the header is read in ReadResponse and the server cookies are added to FCookies, which is nil. Don't use trunk at the moment when dealing with cookies ---------------------------------------------------------------------- Thaddy · 2025-07-30 14:28 That is not what I experience, Fred. ---------------------------------------------------------------------- Nitorami · 2025-07-30 16:54 I'm back, been offline for a few days. Now with FPC 3.2.4 and rvk's latest code example, it works. I can batch download and store forum pages into local html files. Brillant ! What remains to do for me - a single html page is around 200k in size, 99% of which is page layout, formatting, links and stuff I don't need. I want to get rid of that and extract the relevant information as plain text. Probably will write a simple filter program to do this. Or is there a html processing unit to assist ? Anyway, thanks a lot for the support ! Frankly I had not hoped to get that far. ---------------------------------------------------------------------- rvk · 2025-07-30 18:12 Quote from: Nitorami on July 30, 2025, 06:54:10 pmWhat remains to do for me - a single html page is around 200k in size, 99% of which is page layout, formatting, links and stuff I don't need. I want to get rid of that and extract the relevant information as plain text. Probably will write a simple filter program to do this. Or is there a html processing unit to assist ?For new posts on the forum (and subforums) you can use /external?type=xml or rss/rss2. For example: https://forum.vbulletin.com/external?type=xml https://forum.vbulletin.com/external?type=xml&nodeid=40 Unfortunately you can't do similar with topics itself. So yes, you need to scrape them yourself (including next pages etc). Sometimes there is a plugin for "clean print" (it was default installed in vBulletin upto version 3 but not standard in later versions). You can parse the HTML file the usual way in FPC. (search for html dom here on the forum). ---------------------------------------------------------------------- Nitorami · 2025-08-02 08:49 Now the following happened: I scraped a few pages - not a lot really, I am very careful not to trigger any alarms. When I then tried to visit the winamp forum via firefox, I got a cloudflare "confirm you are human" captcha for the first time ever. Well, I had not changed the useragent ID in my program, so I thought cloudflare may have detected two different IDs from the same IP. So I asked google for my browser's ID, and copied that into the code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0' But surprisingly, that gives me error 403 consistently. But it works with rvk's ID. What could be the reason for that ? EDIT: I just noted that now I am getting this captcha each time I try to login to the forum from my browser... it seems that my browser's ID is on cloudflare's list ---------------------------------------------------------------------- rvk · 2025-08-02 11:14 Quote from: Nitorami on August 02, 2025, 10:49:10 amEDIT: I just noted that now I am getting this captcha each time I try to login to the forum from my browser... it seems that my browser's ID is on cloudflare's list It could be that in time that captcha goes away for you in the browser. But as I stated in the beginning... protection through Cloudflare is really difficult to beat. It could be that the 403 you're getting is the captcha page. In that case the program should switch to 'human' mode and tell Cloudflare it's human (which of course is a fantasy) 😂 https://www.zenrows.com/blog/bypass-cloudflare#fortified-headless- browsers You could try to find out the original ip of forum.winamp.com. (not the cloadflare ip) (See method #3 in above article) ---------------------------------------------------------------------- Nitorami · 2025-08-02 12:41 The funny thing is, I am not trying to do anything illegal and am neither planning a DDoS campaign or anything that would have any noticeable impact on the server infrastructure. The forum these days attracts around 5 legit members, old farts like me, while the user stats page usually shows 200 ... 20000 "guests". According to NJK these are mainly AI scrapers, google and amazon crawlers. And around 1000 new spammers per month, but, while a nuisance, these hardly contribute to the active user stats. What is the point of cloudflare if they just manage to annoy harmless tinkerers like me, while letting through the large-style professional scrapers ? ---------------------------------------------------------------------- rvk · 2025-08-02 13:05 Quote from: Nitorami on August 02, 2025, 02:41:22 pmThe funny thing is, I am not trying to do anything illegal and am neither planning a DDoS campaign or anything that would have any noticeable impact on the server infrastructure. forums.winamp.com is hosted by vbulletin.net (as are many other forums). You can see that because winamp.vbulletin.net exists as hostname (also pointing to cloudflare). There are hundreds of these subdomains (one per hosted forum). So the whole cloudflare thing is handled (or ordered) by vbulletin.net. You say you're not doing anything against the rules when scaping the forum. But the chances are this does violate their terms of service (at least the ones from vbulletin.net). So winamp might be a small forum... it falls under the whole umbrella of vbulletin and they have implemented strict access (via cloudflare). BTW. The professional scrapers also have problems... or they are (individual) allowed because of searchability (by cloudflare). And yes, with millions of scrapers... sometimes a thousand get through... a small percentage. ---------------------------------------------------------------------- Nitorami · 2025-08-02 13:39 Ok. But oddly, they have my browser under suspicion - on almost every page I try to access, I now get a captcha - while my "scraping" program still works (with your user agent ID). So this seems not related to winamp.com, and maybe this is just coincidence. The cloudflare action might have been triggered by something entirely different outside my reach.