Using Tor to download a file from the internet lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- phoenix27 · 2023-11-05 21:40 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function DownloadFile(URL: String; dest: String): Boolean;var  client: TFPHTTPClient;begin    client := TFpHttpClient.Create(nil);  try    try      client.AllowRedirect := true;      client.Get(URL, dest);    client.Proxy.Host:='127.0.0.1';      client.proxy.port := 9050;      Result := true;    except      on E:EHTTPClient do begin        Result := false;        if FileExists(dest) then           DeleteFile(dest);      end;    end;  finally    Client.Free;  end;end;  I tried to use the Proxy property and set it to Tor address but it doesn't work. I suppose it's not for socks proxy. Anyway is there a way to use Tor together with TFPHTTPClient(or another class) to download a file? ---------------------------------------------------------------------- KodeZwerg · 2023-11-06 01:25 Quote from: phoenix27 on November 05, 2023, 10:40:56 pmAnyway is there a way to use Tor together with TFPHTTPClient(or another class) to download a file?I would say no. Its urgent to have a Socks5 proxy connection running IIRC. With Indy it could be possible I guess but I've read about ssl library problems (outdated) ... Remy here on this Forum maintain it, he can give better advise than I ever could. In hope he's reading and having a solution for you. Best regards, Kode. ---------------------------------------------------------------------- delphius · 2023-11-06 10:39 Quote from: phoenix27 on November 05, 2023, 10:40:56 pmAnyway is there a way to use Tor together with TFPHTTPClient(or another class) to download a file? The easiest way on Linux is to use an external proxifier like proxychains ---------------------------------------------------------------------- Remy Lebeau · 2023-11-06 18:49 Quote from: phoenix27 on November 05, 2023, 10:40:56 pmAnyway is there a way to use Tor together with TFPHTTPClient(or another class) to download a file? Tor operates through a SOCKS proxy.  But AFAIK, TFPHTTPClient only supports HTTP proxies, not SOCKS proxies.  So, you will have to use a different HTTP client that supports SOCKS. ---------------------------------------------------------------------- Remy Lebeau · 2023-11-06 18:55 Quote from: KodeZwerg on November 06, 2023, 02:25:09 amWith Indy it could be possible I guess Indy's TIdHTTP component can use a SOCKS proxy, by connecting a TIdSocksInfo component to the TIdHTTP.Socket.TransparentProxy property. Quote from: KodeZwerg on November 06, 2023, 02:25:09 ambut I've read about ssl library problems (outdated) ... Remy here on this Forum maintain it, he can give better advise than I ever could. In hope he's reading and having a solution for you. Indy uses OpenSSL by default, and yes, its OpenSSL support is a bit outdated (it supports only up to OpenSSL 1.0.2u right now).  That's fine up to TLS 1.2, but if you need TLS 1.3 then there is a WIP SSLIOHandler for OpenSSL 1.1+ available as a PR on Indy's GitHub repo.  Or, you can always write your own SSLIOHandler using whatever TLS library you want. ---------------------------------------------------------------------- phoenix27 · 2023-11-07 18:54 Tank you all  for your answers. This is what i did:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function DownloadFile(URL: String; dest: String; out AErrMsg: String): Boolean;var  cl: TIdHTTP;  si: TIdSocksInfo;  OutStream: TFileStream;  SSL: TIdSSLIOHandlerSocketOpenSSL;begin  AErrMsg := '';  cl := TIdHTTP.Create;  try    si := TIdSocksInfo.Create(cl);    try        si.Host:='localhost';        si.Port:=9050;        si.Authentication := saNoAuthentication;        si.Version := svSocks5;        SSL := TIdSSLIOHandlerSocketOpenSSL.Create(cl);        try            cl.HandleRedirects:=true;            cl.RedirectMaximum:=10;          SSL.TransparentProxy:=si;            cl.IOHandler := SSL;            OutStream:= TFileStream.Create(dest,fmCreate);            try            try                 cl.Get(URL,OutStream);                Result := true;              except                on E:Exception do begin                AErrMsg := E.Message;                  Result := false;                  if FileExists(dest) then                     DeleteFile(dest);                end;              end;            finally               OutStream.free;            end;        finally          SSL.free;        end;    finally       si.free;    end;  finally     cl.free;  end;end;  Remy Lebeau, i searched more and on stackoverflow i saw how to use your suggestion. And it works but sometimes it gives me a 403 forbidden error whereas in javascript using the same exit node address it works. SO i wanted to ask if the code above is the right one or if i should correct it or add something else ---------------------------------------------------------------------- Remy Lebeau · 2023-11-07 19:58 Quote from: phoenix27 on November 07, 2023, 07:54:46 pmsometimes it gives me a 403 forbidden error whereas in javascript using the same exit node address it works. That is an HTTP error, not related to SOCKS at all. Quote from: phoenix27 on November 07, 2023, 07:54:46 pmSO i wanted to ask if the code above is the right one or if i should correct it or add something else Looks fine for me, as far as setting up the SOCKS proxy is concerned.  Setting up HTTP authentication is completely separate.  Clearly the HTTP server is asking for authentication, so you need to provide whatever credentials it is asking for.  What is in the WWW- Authenticate response header when the error occurs?  If it is asking for BASIC authentication, you can just set the TIdHTTP.Request.BasicAuthentication property to true and fill in the TIdHTTP.Request.Username and TIdHTTP.Request.Password properties.  But, if it wants a different type of authentication (ie, NTLM, Bearer, etc) then you will have to utilize the appropriate IdAuthentication... units and maybe the TIdHTTP.On(Select)Authorization event(s), too. ---------------------------------------------------------------------- phoenix27 · 2023-11-07 21:24 Remy thank you for your answer. It's not related to authentication. As i said i tried at the same time with pascal and then with Javascript(Electron). First, by downloading the http://icanhazip.com page which shows your address i checked that both(javascript and pascal) were the same Tor exit node, and they were, then i downloaded the file i was interested in and while javascript succeeded pascal gave me a 403 forbidden error. Maybe it has to do with SSL. In your previous answer you talked about different versions of OpenSSL and  I didn't understand well.... ---------------------------------------------------------------------- KodeZwerg · 2023-11-07 23:54 The error you encounter can have many possible reasons, one of them could be the (incompatible since outdated) ssl interface handshake. If you really would want to find out what is going on, I suggest you take advantage of WireShark and analyze your connection with it. It can show up very fast very detailed when there are errors and telling you exact the "whats wrong" part. Since you mention that you download somehow with JavaScript over the TOR network, that would mean to me you are actually downloading via WebBrowser, so that could also be your solution to integrate WebBrowser that runs your script, maybe that site saves cookies or does do many redirects or or or... many possible situations. Can you share your JavaScript and the address you try to download? ---------------------------------------------------------------------- phoenix27 · 2023-11-08 02:38 @KodeZwerg No i' m not using the webbrowser, i' m using a little library called bent which under the hood calls the 'https' library of nodejs.  Yes, i think the problem is the ssl part. Infact i'm discovering other urls which give a more clear message. This is one of the many: "https://www.hdwallpaperspulse.com/wp- content/uploads/2013/11/05/1.jpg". As for the Javascript code here it is Code: Javascript  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    function downloadImage(imgUrl, destPath){                return new Promise(async function(resolve, reject){                        function tryFollowingRedirect(err){                                let codes = [300, 301, 302, 303, 304, 305, 306, 307, 308];                              if(codes.indexOf(err.statusCode) > -1){ //redirection                                      console.log('Redirecting...');                                      return err.headers.location;                              } else {                                      console.log('Download Error: Status Code ' + err.statusCode);                              }                                return null;                        }                        async function f(iUrl){                                try{                                      let agent = new ProxyAgent('socks://127.0.0.1:9050');                                      const getBentBuffer = bent('buffer');                                        let buffer = await getBentBuffer(iUrl, null, null, agent);                                      fs.writeFile(destPath, buffer, function(){                                              resolve();                                      });                                } catch(err){                                      let url = tryFollowingRedirect(err)                                      if(url){                                              if(!url.startsWith('http')){                                                      let urlObj = URI.parse(iUrl)                                                      if(url.startsWith('/')){                                                              f(urlObj.protocol + '/' + urlObj.hostname + url);                                                      } else{                                                              f(urlObj.protocol + '/' + urlObj.hostname + urlObj.pathname + url);                                                      }                                              } else {                                                        f(url);                                                }                                      } else {                                              console.log(err);                                              reject(err);                                        }                              }                        }                        f(imgUrl);                });        } Bent and ProxyAgent are external libraries that you can find on github while URI and  fs are nodejs modules ---------------------------------------------------------------------- Remy Lebeau · 2023-11-08 18:38 Quote from: phoenix27 on November 07, 2023, 10:24:15 pmIt's not related to authentication. You are getting an HTTP authentication error, so how is that not related to authentication? Quote from: phoenix27 on November 07, 2023, 10:24:15 pmMaybe it has to do with SSL. In your previous answer you talked about different versions of OpenSSL and  I didn't understand well.... The current version of OpenSSL is 3.1.4, which supports TLS 1.3.  Indy officially supports only up to OpenSSL 1.0.2u, which supports TLS 1.2 (and unofficially supports OpenSSL 3.x via a pending PullRequest in the GitHub repo). Quote from: KodeZwerg on November 08, 2023, 12:54:02 amThe error you encounter can have many possible reasons, one of them could be the (incompatible since outdated) ssl interface handshake. That would not cause an HTTP error, though.  If the SSL/TLS handshake failed, the server would have no choice but to simply close the TCP connection, and maybe send a TLS alert beforehand.  But certainly not send an HTTP error - it couldn't, as there is no HTTP session yet. Quote from: phoenix27 on November 08, 2023, 03:38:29 amCode: Javascript  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i let codes = [300, 301, 302, 303, 304, 305, 306, 307, 308];if(codes.indexOf(err.statusCode) > -1){ //redirection  FYI, 304 is not a redirection. ---------------------------------------------------------------------- phoenix27 · 2023-11-08 21:41 QuoteYou are getting an HTTP authentication error, so how is that not related to authentication?What is authentication? It's when the server requires you to give an username and a password, right? Well, it's not an authentication issue then!! Take this url for example 'https://cdn. pixabay.com/photo/2016/08/11/23/55/trees-1587301_960_720.jpg'. With the javascript routine i posted above i can download it using Tor without any problem whatsoever, whereas with pascal, using the same tor address it gives me a 'HTTP/1.1 403 forbidden'. How can it be authentication! It's not! It has to do with SSL or something else. Other urls give me a SSL connection error. Maybe the error depends on the server, how it was configured. I would like to know how can i update indy to the more recent version of SSL. I went to the github page you gave me but i cannot figure out the steps. ---------------------------------------------------------------------- Remy Lebeau · 2023-11-09 01:13 Quote from: phoenix27 on November 08, 2023, 10:41:04 pmWhat is authentication? It's when the server requires you to give an username and a password, right? Well, it's not an authentication issue then!! Nevermind, I was confusing 403 Forbidden with 401 Unauthorized. Quote from: phoenix27 on November 08, 2023, 10:41:04 pmI would like to know how can i update indy to the more recent version of SSL. I already answered that earlier.  But, you haven't even verified yet if SSL/TLS is actually the real problem or not. In any case, I just re-reviewed the code you showed earlier, and I notice you are not configuring the TIdSSLIOHandlerSocketOpenSSL.SSLOptions property at all.  For instance, the SSLOptions.SSLVersions property defaults to TLS 1.0, so try enabling TLS 1.1 and 1.2 as well, eg: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SSL.SSLOptions.SSLVersions := [sslvTLSv1, sslvTLSv1_1, sslvTLSv1_2]; Quote from: phoenix27 on November 08, 2023, 10:41:04 pmI went to the github page you gave me but i cannot figure out the steps. I didn't write the new SSLIOHandler, so I can't give you exact steps.  All I know is that the PR code can be dropped on top of a default Indy installation, and then it provides new SSLIOHandler classes that can be used in client and server code. ---------------------------------------------------------------------- phoenix27 · 2023-11-09 01:37 Quoten any case, I just re-reviewed the code you showed earlier, and I notice you are not configuring the TIdSSLIOHandlerSocketOpenSSL.SSLOptions property at all.  For instance, the SSLOptions.SSLVersions property defaults to TLS 1.0, so try enabling TLS 1.1 and 1.2 as well, eg:I did configure it later because i kept searching and i came across a post on stackoverflow and so i did: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ssl.SSLOpti ons.SSLVersions:=[sslvSSLv3,sslvSSLv2,sslvTLSv1,sslvTLSv1_1,sslvTLSv1_ 2];  But i still got the same error, although i noticed an improvement in the sense that some sites that before gave me a SSL connection error didn't anymore. But the forbidden error still remains. ---------------------------------------------------------------------- Remy Lebeau · 2023-11-09 16:56 Quote from: phoenix27 on November 09, 2023, 02:37:51 amI did configure it later because i kept searching and i came across a post on stackoverflow and so i did: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ssl.SSLOpti ons.SSLVersions:=[sslvSSLv3,sslvSSLv2,sslvTLSv1,sslvTLSv1_1,sslvTLSv1_ 2]; DO NOT enable sslvSSLv2 and sslvSSLv3!  SSL v2.0 and v3.0 are no longer secure, and nobody uses them anymore.  TLS v1.0 should be the absolute bare minimum, though most servers nowadays require TLS v1.1/1.2, if not TLS v1.3. Quote from: phoenix27 on November 09, 2023, 02:37:51 amBut i still got the same error, although i noticed an improvement in the sense that some sites that before gave me a SSL connection error didn't anymore. But the forbidden error still remains. Then you probably need to reach out to that site owner and ask them to look into the problem to find out why their site is rejecting you.