idHTTP and Accept-Encoding
lazarus / 38-Networking and Web Programming

----------------------------------------------------------------------
RDL · 2018-03-11 06:48

Hi

Why if I specify Accept-Encoding: gzip
idHTTP sends Accept-Encoding: deflate, gzip, identity?

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Headers:='Host: '+Domain+#13+                 'User-Agent:
'+UserAgent+#13+                 'Accept: '+Accept+#13+              
 'Accept-Encoding: gzip'+#13+                 'Cookie: '+Cookie+#13+ 
              'Connection: keep-alive'+#13+                 'Content-
Type: application/x-www-form-urlencoded'; HTTP:=TIdHTTP.Create();HTTP.
HTTPOptions:=[hoForceEncodeParams,hoKeepOrigProtocol];HTTP.ProtocolVer
sion:=pv1_1;HTTP.IOHandler:=SSL;HTTP.Compressor:=CompressorZLib;HTTP.H
andleRedirects:=True;HTTP.Request.CustomHeaders.AddText(Headers);Respo
nse.ResponseText:=HTTP.Post(URL,PostStream,IndyTextEncoding_UTF8);
In debag proxy showing:

Accept-Encoding: deflate, gzip, identity

Why?

Lazarus 1.8.0, indy 5444 OPM, Linux platform

----------------------------------------------------------------------
Remy Lebeau · 2018-03-12 21:37

Quote from: RDL on March 11, 2018, 07:48:20 amWhy if I specify Accept-
Encoding: gzip
idHTTP sends Accept-Encoding: deflate, gzip, identity?

Because those are the encodings that Indy natively supports when
receiving a response from the server.  When you assign a Compressor to
TIdHTTP, 'deflate' and 'gzip' are added to the Request.AcceptEncoding
if they are not already present, and 'identity' is added to the
Request.AcceptEncoding if not blank and 'identity' is not already
present.  This behavior is by design.

If you don't want this behavior, you will have to unassign the
Compressor from TIdHTTP, and then use the Request.CustomHeaders
property instead of the Request.AcceptEncodings property to specify
your own 'Accept-Encoding' value.  But then you will have to manually
decompress the response data yourself if it is sent compressed, which
means you can't use the overloaded version of TIdHTTP.Post() that
returns a String, you will have to use the overloaded version that
populates an output TStream instead, and then you can decompress it
and decode the resulting bytes to a String.

Quote from: RDL on March 11, 2018, 07:48:20 amCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
HTTP.Request.CustomHeaders.AddText(Headers);
That is not the correct way to set the headers you are assigning
custom values to.  Use the appropriate TIdHTTP.Request properties
instead:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
//HTTP.Request.Host := Domain; // done automatically by
TIdHTTPHTTP.Request.UserAgent := UserAgent;HTTP.Request.Accept :=
Accept;//HTTP.Request.AcceptEncoding := 'gzip'; // let
TIdHTTP.Compressor handle thisHTTP.Request.Connection := 'keep-
alive';HTTP.Request.ContentType := 'application/x-www-form-
urlencoded';HTTP.Request.CustomHeaders.Values['Cookie'] := Cookie; //
consider letting TIdCookieManager handle this instead
Also, since you are posting an 'application/x-www-form-urlencoded'
request, your PostStream should be a TStringList, let TIdHTTP.Post()
format its own internal stream, and set the Request.ContentType for
you.

----------------------------------------------------------------------
RDL · 2018-03-13 04:08

[Remy Lebeau]
Thank you. At me all has turned out.

I've got one more question.
Is it possible to somehow put Content-Length last line. For example:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i POST
http://URL/ HTTP/1.1Content-Type: application/x-www-form-
urlencoded1Host: URLAccept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8Accept-
Encoding: gzip, identityUser-Agent: Mozilla/3.0 (compatible; Indy
Library)Content-Length: 47

----------------------------------------------------------------------
Remy Lebeau · 2018-03-13 17:51

Quote from: RDL on March 13, 2018, 05:08:32 amIs it possible to
somehow put Content-Length last line.

TIdHTTP calculates the Content-Length header for you.  You have no
control over the order in which TIdHTTP sends its HTTP headers, nor
should it usually matter since HTTP itself doesn't generally care
about the order of headers.