Indy TIdHTTP & Private Proxies Issue.... lazarus / 21-General ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-15 18:05 I'm doing a small test project that does a simple google search query through Indy and a private proxy that requires username & password. I keep getting the following error message while trying to get the HTML data back from the Google search: "Connection closed gracefully". Here's the code I've tried: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses    Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,    IdHTTP, IdSSLOpenSSL, IdAuthentication, IdIOHandler, IdIOHandlerSocket, IdIOHandlerStack, IdServerIOHandlerStack;                       procedure TForm1.Button1Click(Sender: TObject);var  IdHTTP: TIdHTTP;  SSLHandler: TIdSSLIOHandlerSocketOpenSSL;  Response: TStringStream;  SearchQuery, URL: string;begin  IdHTTP := TIdHTTP.Create(nil);  SSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);  Response := TStringStream.Create;  try    // Configure SSL handler    SSLHandler.SSLOptions.Method := sslvTLSv1_2;    IdHTTP.IOHandler := SSLHandler;     // Set proxy parameters    IdHTTP.ProxyParams.ProxyServer := edit1.text; //Proxy Host address (e.g. 168.1.9.169)    IdHTTP.ProxyParams.ProxyPort := 50101; // Replace with your proxy port    IdHTTP.ProxyParams.ProxyUsername := edit3.text;    IdHTTP.ProxyParams.ProxyPassword := edit4.text;     // Set user agent to mimic a browser    IdHTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3';     // Prepare the search query    SearchQuery := 'lazarus pascal indy proxy example';  URL := 'https://www.google.com/search?q=dog';     // Perform the GET request    IdHTTP.Get(URL, Response);     // Display the response in Memo1    Memo1.Lines.Text := Response.DataString;  except    on E: Exception do      Memo1.Lines.Text := 'Error: ' + E.Message;  end;  // Clean up  Response.Free;  SSLHandler.Free;  IdHTTP.Free; end; ---------------------------------------------------------------------- Remy Lebeau · 2024-12-15 23:18 You are seeing a DEBUGGER message.  The debugger sees all exceptions before your app's code does. Just resume execution and TIdHTTP should catch that exception internally. It should not be escaping into your normal code at runtime.  If you don't want to see the exception in the debugger, you can configure the debugger to ignore it. ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-16 15:09 Right, but how do I get the HTML from the IdHTTP into memo1? Why would there be any "connection closed gracefully" in the first place if my code was written correctly? Quote from: Remy Lebeau on December 16, 2024, 12:18:21 amYou are seeing a DEBUGGER message.  The debugger sees all exceptions before your app's code does.  TIdHTTP should catch that exception internally, it should not be escaping into your normal code at runtime.  If you don't want to see the exception in the debugger, you can configure the debugger to ignore it. ---------------------------------------------------------------------- CCRDude · 2024-12-16 15:15 Your issue is a bit vague. Most simply answer would be: ignore the exception, and it probably should appear in Memo1.Lines . ---------------------------------------------------------------------- Remy Lebeau · 2024-12-16 15:52 Quote from: AaronCatolico1 on December 16, 2024, 04:09:35 pmRight, but how do I get the HTML from the IdHTTP into memo1? What you already have is fine for that. However, TIdHTTP.Get() has overloads which return a String, you don't need the Response stream, eg: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Memo1.Text := IdHTTP.Get(URL); Quote from: AaronCatolico1 on December 16, 2024, 04:09:35 pm Why would there be any "connection closed gracefully" in the first place if my code was written correctly? Because HTTP allows for that. EIdConnClosedGracefully happens when Indy tries to read more data from the socket after the peer has closed the connection. For HTTP, that means the server has either closed the connection prematurely before it has sent the end of the response, or the server is using a socket closure as the end-of-response indicator rather than using the Content-Length or Transfer-Encoding response headers. You would have to analyze the traffic to see what is really happening. ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-16 15:59 Well, I ignored the exception & still nothing comes back while trying to tunnel through a private proxy. As soon as I get rid of the the following proxy lines, the html data actually appears in the memo1: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     IdHTTP.ProxyParams.ProxyServer := edit1.text;    IdHTTP.ProxyParams.ProxyPort := 50101; // Replace with your proxy port    IdHTTP.ProxyParams.ProxyUsername := edit3.text;    IdHTTP.ProxyParams.ProxyPassword := edit4.text; It seems to only work when I turn off the proxy configurations. I'd like to keep this as an option while connecting to any websites. Any idea why I'm not able to connect through this private proxy? BTW, I know that this private proxy is working because I've made the configurations on my pc which shows that I am indeed able to connect to Google, YouTube, this forum, etc.. The problem has got to be with Indy or the way that I've written the code, just not sure exactly how it's supposed to be implemented. Quote from: CCRDude on December 16, 2024, 04:15:19 pmYour issue is a bit vague. Most simply answer would be: ignore the exception, and it probably should appear in Memo1.Lines . ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-16 16:17 I still keep getting the "connection closed gracefully" while trying to use the private proxy. I turn it off & then nothing comes back at all. Here's the updated code I'm using now: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var  IdHTTP: TIdHTTP;  SSLHandler: TIdSSLIOHandlerSocketOpenSSL;  URL: string;begin  IdHTTP := TIdHTTP.Create(nil);  SSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);  try    // Configure SSL handler    SSLHandler.SSLOptions.Method := sslvTLSv1_2;    IdHTTP.IOHandler := SSLHandler;     // Set proxy parameters    IdHTTP.ProxyParams.ProxyServer := edit1.text;    IdHTTP.ProxyParams.ProxyPort := strToInt(edit2.text); // Replace with your proxy port    IdHTTP.ProxyParams.ProxyUsername := edit3.text;    IdHTTP.ProxyParams.ProxyPassword := edit4.text;     // Set user agent to mimic a browser    IdHTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3';     // Prepare the search query  URL := 'https://www.google.com/search?q=dog';     //Display the html data from google    Memo1.Text := IdHTTP.Get(URL);   except    on E: Exception do      showmessage('Error: ' + E.Message);  end;  // Clean up   SSLHandler.Free;  IdHTTP.Free; end; How would I "analyze the traffic to see what is really happening"? You mean using a program like wireshark or fiddler? If so, what exactly am I looking for? Even when I find it, what changes would most likely need to be made to the code I'm using above? I've use plenty of programming languages in the past such as Python, PHP, Visual Basic, Javascript, etc., & have never had such difficulty trying to use any library with proxies. I'm not sure why it's so difficult to do in Pascal. It's usually just plug n play in all other languages & libraries. I'm not trying to be rude, I'm just being honest that I've never had this much difficulty to do something that is usually very easy to accomplish in other languages. Quote from: Remy Lebeau on December 16, 2024, 04:52:17 pmQuote from: AaronCatolico1 on December 16, 2024, 04:09:35 pmRight, but how do I get the HTML from the IdHTTP into memo1? What you already have is fine for that. However, TIdHTTP.Get() has overloads which return a String, you don't need the Response stream, eg: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Memo1.Text := IdHTTP.Get(URL); Because HTTP allows for that. EIdConnClosedGracefully happens when Indy tries to read more data from the socket after the peer has closed the connection. For HTTP, that means the server has either closed the connection prematurely before it has sent the end of the response, or the server is using a socket closure as the end-of-response indicator rather than using the Content-Length or Transfer-Encoding response headers. You would have to analyze the traffic to see what is really happening. ---------------------------------------------------------------------- Remy Lebeau · 2024-12-16 17:05 Quote from: AaronCatolico1 on December 16, 2024, 04:59:08 pmWell, I ignored the exception & still nothing comes back while trying to tunnel through a private proxy. What kind of proxy are you using, exactly?  Is it an HTTP proxy, a SOCKS proxy, something else, etc?  The ProxyParams feature only works with HTTP proxies.  If you need to support other kinds of proxies, you have to use the IOHandler's TransparentProxy feature instead. Quote from: AaronCatolico1 on December 16, 2024, 04:59:08 pmIt seems to only work when I turn off the proxy configurations. I'd like to keep this as an option while connecting to any websites. Any idea why I'm not able to connect through this private proxy? Not without more detailed information.  The ProxyParams feature has been available for a VERY LONG time, so I know it works under normal conditions. Can you provide a capture of the raw socket traffic and the HTTP traffic? Quote from: AaronCatolico1 on December 16, 2024, 04:59:08 pmBTW, I know that this private proxy is working because I've made the configurations on my pc which shows that I am indeed able to connect to Google, YouTube, this forum, etc.. That doesn't negate the possibility that the proxy may be something that is unsupported by the ProxyParams feature.  There are many different kinds of proxies.  I need more details to diagnose the issue. Quote from: AaronCatolico1 on December 16, 2024, 04:59:08 pmThe problem has got to be with Indy or the way that I've written the code, just not sure exactly how it's supposed to be implemented. The code shown is fine, provided the proxy is a standard HTTP proxy. Quote from: AaronCatolico1 on December 16, 2024, 05:17:51 pmI still keep getting the "connection closed gracefully" while trying to use the private proxy. Without seeing the HTTP/socket traffic, there is no way to diagnose that. Quote from: AaronCatolico1 on December 16, 2024, 05:17:51 pmI turn it off & then nothing comes back at all. Then that is a completely different issue. Quote from: AaronCatolico1 on December 16, 2024, 05:17:51 pmHow would I "analyze the traffic to see what is really happening"? You mean using a program like wireshark or fiddler? Yes, if the problem turns out to be at the socket/proxy level.  If the problem is at the HTTP level, you can capture the HTTP requests/responses by assigning one of Indy's TIdLog... components to the TIdHTTP.Intercept property. Quote from: AaronCatolico1 on December 16, 2024, 05:17:51 pmIf so, what exactly am I looking for? At what point exactly the socket closure is actually occurring in relation to the rest of the traffic.  For instance, is the socket closure intentional at the HTTP level (ie, there is a Connection: close response header and no Connect-Length or Transfer-Encoding header)? Or, is it happening without any HTTP response being sent at all? etc.  Maybe the proxy doesn't like the request and is forcing the connection closed.  OR maybe there is a firewall blocking the connection.  There is just not enough information yet to know what is really going on. Quote from: AaronCatolico1 on December 16, 2024, 05:17:51 pmEven when I find it, what changes would most likely need to be made to the code I'm using above? I can't answer that without first identifying what the root cause actually is. Quote from: AaronCatolico1 on December 16, 2024, 05:17:51 pmI've use plenty of programming languages in the past such as Python, PHP, Visual Basic, Javascript, etc., & have never had such difficulty trying to use any library with proxies. I'm not sure why it's so difficult to do in Pascal. It's usually just plug n play in all other languages & libraries. I'm not trying to be rude, I'm just being honest that I've never had this much difficulty to do something that is usually very easy to accomplish in other languages. Are you really complaining about Pascal in general, or just about Indy specifically? Those other languages you mention are more higher-level, whereas Pascal in general is more lower-level.  High-level languages are designed for simplicity, so they tend to provide more wrappers and less complexity.  Pascal tends to be more flexible, and sometimes that means you need to piece multiple isolated components together to get the same tasks done. ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-16 18:14 QuoteWhat kind of proxy are you using, exactly?  Is it an HTTP proxy, a SOCKS proxy, something else, etc?  The ProxyParams feature only works with HTTP proxies.  If you need to support other kinds of proxies, you have to use the IOHandler's TransparentProxy feature instead. - Gambit The private proxy I ordered is in IPV4 format which requires username & password authentication. It supports both SOCKS5 and HTTP protocols depending on the port at which I try to access it. I've tried both ports to no avail. If you'd like (and I personally don't mind because I know that you're a trustworthy guy), I could send you the login credentials for you to do a quick test with the code I used above to see if you can quickly find where the issue is on the network. I could send you a quick PM with the login credentials, if you don't mind testing it. It shouldn't take long at all. I was given permission by proxy-seller . com to access this proxy for up to 3 devices. PM Sent! ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-16 19:22 @Gambit Here's what I've come up with from WireShark as far as the TCP Stream goes while trying to connect through the private proxy: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i CONNECT www.google.com:443 HTTP/1.1Pragma: no-cacheProxy-Connection: keep- aliveHost: www.google.comAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8User- Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3  HTTP/1.1 407 Proxy Authentication RequiredDate: Mon, 16 Dec 2024 19:15:52 GMTProxy- Authenticate: Basic realm="proxy"Connection: closeContent-Length: 0  GET /search?q=dog HTTP/1.1Host: www.google.comAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8User- Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3Proxy- Authorization: Basic *HIDDEN== ---------------------------------------------------------------------- Remy Lebeau · 2024-12-17 17:09 Quote from: AaronCatolico1 on December 16, 2024, 07:14:46 pmThe private proxy I ordered is in IPV4 format which requires username & password authentication. It supports both SOCKS5 and HTTP protocols depending on the port at which I try to access it. I've tried both ports to no avail. The TIdHTTP.ProxyParams does not support SOCKS.  For that, you must use a separate TIdSocksInfo component and assign it to the IOHandler's TransparentProxy property when you want the connection to go through a SOCKS proxy, eg: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);var  IdHTTP: TIdHTTP;  SSLHandler: TIdSSLIOHandlerSocketOpenSSL;  IdSocks: TIdSocksInfo;  URL: string;begin  try    IdHTTP := TIdHTTP.Create(nil);    try      // Configure SSL handler      SSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP);      SSLHandler.SSLOptions.Method := sslvTLSv1_2;      IdHTTP.IOHandler := SSLHandler;           // Set proxy parameters      if UseSOCKSProxy then // <-- add this option to your UI!      begin        IdSocks := TIdSocksInfo.Create(SSLHandler);        IdSocks.Authentication := saUsernamePassword;        IdSocks.Host := Edit1.Text;        IdSocks.Port := StrToInt(Edit2.Text);        IdSocks.Username := Edit3.Text;        IdSocks.Password := Edit4.Text;        IdSocks.Version := svSocks5;        SSLHandler.TransparentProxy := IdSocks;      end      else if UseHTTPProxy then // <-- add this option to your UI!      begin        IdHTTP.ProxyParams.ProxyServer := Edit1.Text;        IdHTTP.ProxyParams.ProxyPort := StrToInt(Edit2.Text);        IdHTTP.ProxyParams.ProxyUsername := Edit3.Text;        IdHTTP.ProxyParams.ProxyPassword := Edit4.Text;    end;           // Set user agent to mimic a browser      IdHTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3';           // Prepare the search query      URL := 'https://www.google.com/search?q=dog';           //Display the html data from google      Memo1.Text := IdHTTP.Get(URL);         finally    // Clean up           IdHTTP.Free;    end;  except    on E: Exception do      ShowMessage('Error: ' + E.Message);  end;end; Quote from: AaronCatolico1 on December 16, 2024, 08:22:06 pmHere's what I've come up with from WireShark as far as the TCP Stream goes while trying to connect through the private proxy: That does not look right.  After the 407 reply, TIdHTTP should have send another CONNECT request with the Proxy-Authorization header added to it. It should not be sending the GET request (without the Proxy- Authorization header) until after CONNECT has returned a 200 reply first.  I will have to dig into the code to see what's happening. ---------------------------------------------------------------------- Remy Lebeau · 2024-12-17 17:51 Quote from: Remy Lebeau on December 17, 2024, 06:09:46 pmQuote from: AaronCatolico1 on December 16, 2024, 08:22:06 pmHere's what I've come up with from WireShark as far as the TCP Stream goes while trying to connect through the private proxy: That does not look right.  After the 407 reply, TIdHTTP should have send another CONNECT request with the Proxy-Authorization header added to it. It should not be sending the GET request (without the Proxy- Authorization header) until after CONNECT has returned a 200 reply first.  I will have to dig into the code to see what's happening. Nevermind, I see the problem.  You don't have the hoInProcessAuth flag enabled in the TIdHTTP.HTTPOptions property, so when connecting to the HTTP proxy TIdHTTP ignores the 407 reply and moves on with sending the GET request.  If you enable the flag, then TIdHTTP will process the 407 to login to the proxy before sending the GET. ---------------------------------------------------------------------- AaronCatolico1 · 2024-12-28 19:59 @Remy Lebeau Thanks again Remy! My apologies for such a delay in response time. I was out of state, then came back with the flu. I've been out for a while now. It seems like you've solved the issues here. Just wanted to thank you again for your usual awesome support. Happy holidays to you as well!