TFPHTTPClient get ERROR 401 (Unauthorized) lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- DidaJI · 2020-08-24 08:54 Hi, I have a problem logging in to the device API via TFPHTTPClient. This works for me from the web browser: http://admin:admin123@192.168.0.39/cgi- bin/record.cgi?action=find&name=Access. This works for me from python script: import requests from requests.auth import HTTPDigestAuth response = requests.get("http://192.168.0.39/cgi- bin/record.cgi?action=find&name=Access", auth=HTTPDigestAuth("admin", "admin123")) print (response) But if I use TFPHTTPClient, it always returns Error 401 For example, I tried this: url := 'http://admin:admin123@192.168.0.39/cgi- bin/recordFinder.cgi?action=find&name=Access'; htmlResponse := TFPCustomHTTPClient.SimpleGet(url); Or this:  httpClient := TFPHTTPClient.Create(nil);   try    httpClient.AllowRedirect := true;    httpClient.Password := 'admin123';    httpClient.UserName := 'admin';    httpClient.AddHeader('User-Agent','Mozilla/5.0 (compatible; fpweb)');    FileContents := httpClient.Get('http://192.168.0.39/cgi- bin/recordFinder.cgi?action=find&name=Access');    theStatusCode := httpClient.ResponseStatusCode;   finally    httpClient.Free;   end; Or this:  httpClient := TFPHTTPClient.Create(nil);  FormData := tstringlist.Create;   try    FormData.Values['auth'] := 'HTTPDigestAuth';    FormData.Values['username'] := 'admin';    FormData.Values['password'] := 'admin123';    FileContents := httpClient.FormPost('http://192.168.0.39/cgi- bin/recordFinder.cgi?action=find&name=Access', FormData);    theStatusCode := httpClient.ResponseStatusCode;   finally     FormData.Free;     httpClient.Free;   end; And other variants. Unfortunately, everything leads to the result "Error 401". Would anyone please know how to solve this? Thank's: Zdenek. ---------------------------------------------------------------------- paweld · 2020-08-24 09:12 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  uses  base64; var  login, pass: String;begin  login := 'admin';  pass := 'admin123';  httpClient := TFPHTTPClient.Create(nil);  try   httpClient.AllowRedirect := true;   httpClient.RequestHeaders.Add('Authorization: Basic ' + EncodeStringBase64(login + ':' + pass));   FileContents := httpClient.Get('http://192.168.0.39/cgi- bin/recordFinder.cgi?action=find&name=Access');   theStatusCode := httpClient.ResponseStatusCode;  finally   httpClient.Free;  end;end; ---------------------------------------------------------------------- DidaJI · 2020-08-24 09:35 To paweld: Thanks, I tried this too and it's still the same. ---------------------------------------------------------------------- DidaJI · 2020-08-24 11:06 So sorry, this: http://admin:admin123@192.168.0.39/cgi- bin/recordFinder.cgi?action=find&name=Access it also doesn't work from a web browser. I was only able to do this because the browser remembered the login details. If I call it from a browser, then I still have to enter the name and password in the form that appears. The only thing that works well is this python script: import requests from requests.auth import HTTPDigestAuth response = requests.get("http://192.168.0.39/cgi- bin/record.cgi?action=find&name=Access", auth=HTTPDigestAuth("admin", "admin123")) print (response) Would anyone know how to replace this script with TFPHTTPClient? Thank's: Zdenek. ---------------------------------------------------------------------- rvk · 2020-08-24 11:27 HTTPDigestAuth should result in the HTTP Basic Authentication that paweld already gave you. What does the following python3 script give you (adding debug options)? Code: Python  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i import requestsfrom requests.auth import HTTPDigestAuth import loggingtry:  import http.client as http_clientexcept ImportError:    # Python 2    import httplib as http_clienthttp_client.HTTPConnection.debuglevel = 1 logging.basicConfig()logging.getLogger().setLevel(logging.DEBUG)reques ts_log = logging.getLogger("requests.packages.urllib3")requests_log.se tLevel(logging.DEBUG)requests_log.propagate = True response = requests.get("http://192.168.0.39/cgi- bin/record.cgi?action=find&name=Access", auth=HTTPDigestAuth("admin", "admin123"   ))print (response) ---------------------------------------------------------------------- trev · 2020-08-24 11:36 Are you sure the device web server implements basic authentication? ---------------------------------------------------------------------- paweld · 2020-08-24 11:38 https://forum.lazarus.freepascal.org/index.php/topic,15747.msg85028.ht ml#msg85028 ---------------------------------------------------------------------- rvk · 2020-08-24 11:44 Quote from: paweld on August 24, 2020, 01:38:49 pmhttps://forum.lazaru s.freepascal.org/index.php/topic,15747.msg85028.html#msg85028 Woops. Yes, HTTPBasicAuth is something different from HTTPDigestAuth. I missed that it was the Digest auth. Logging would have confirmed that. https://en.wikipedia.org/wiki/Digest_access_authentication (Bit more complicated but doable) ---------------------------------------------------------------------- DidaJI · 2020-08-24 12:04 To rvk: the modified script returns the following: DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 192.168.0.39:80 send: 'GET /cgi-bin/record.cgi?action=find&name=Access HTTP/1.1\r\nHost: 192.168.0.39\r\nConnection: keep-alive\r\nAccept- Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python- requests/2.21.0\r\n\r\n' reply: 'HTTP/1.1 401 Unauthorized\r\n' header: WWW-Authenticate: Digest realm="Login to 6a285c4b6cf2c6685d0eee0bbca565d6", qop="auth", nonce="1339093192", opaque="12a495b4a51a1ad37368ffdbf2174e850e759965" header: Connection: close header: Set-Cookie:secure; HttpOnly header: CONTENT-LENGTH: 0 DEBUG:urllib3.connectionpool:http://192.168.0.39:80 "GET /cgi- bin/record.cgi?action=find&name=Access HTTP/1.1" 401 0 DEBUG:urllib3.connectionpool:Resetting dropped connection: 192.168.0.39 send: 'GET /cgi-bin/record.cgi?action=find&name=Access HTTP/1.1\r\nHost: 192.168.0.39\r\nConnection: keep-alive\r\nAccept- Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python- requests/2.21.0\r\nCookie: secure\r\nAuthorization: Digest username="admin", realm="Login to 6a285c4b6cf2c6685d0eee0bbca565d6", nonce="1339093192", uri="/cgi-bin/record.cgi?action=find&name=Access", response="ff0257016ad4988646fe323e7ac0ce9d", opaque="12a495b4a51a1ad37368ffdbf2174e850e759965", qop="auth", nc=00000001, cnonce="4b5442b851a4e032"\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: X-XSS-Protection: 1;mode=block header: X-Frame-Options: SAMEORIGIN header: Content-Security-Policy: script-src 'self' 'unsafe-inline' 'unsafe-eval' header: Strict-Transport-Security: max-age=604800; includeSubDomains header: Content-type: text/plain;charset=utf-8 header: CONNECTION: close header: Set-Cookie:secure; HttpOnly header: CONTENT-LENGTH: 37131 DEBUG:urllib3.connectionpool:http://192.168.0.39:80 "GET /cgi- bin/record.cgi?action=find&name=Access HTTP/1.1" 200 37131 ---------------------------------------------------------------------- rvk · 2020-08-24 12:23 Quote from: DidaJI on August 24, 2020, 02:04:50 pmthe modified script returns the following: You see that there are to requests done (GET). One is without authentication. You'll get a header back with WWW-Authenticate: Digest and some extra information (realm, qop, nonce and opaque). You'll need that extra information to create a second request with correct authentication in response. As seen here https://en.wikipedia.org/wiki/Digest_access_authentication If I can make some time today I can create an example. ---------------------------------------------------------------------- rvk · 2020-08-24 15:37 Had a few minutes to spare and made a small example for fphttpclient and Digest authentication. It uses the sample server at https://jigsaw.w3.org/HTTP/ https://jigsaw.w3.org/HTTP/Digest Drop a TButton and a TMemo on a new project. Double click the button and paste the following code in the editor. (You'll need the openssl libraries in the .exe directory if they are not available systemwide) If it works you can change the login, pass and URL for your server and see if it works there too. (Last text in sample should be "Your browser made it!") Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses fphttpclient, URIParser, md5, opensslsockets, dateutils; // sample server with Digest Authentication = https://jigsaw.w3.org/HTTP/Digest/ procedure TForm1.Button1Click(Sender: TObject);var  login, pass: string;  HTTPClient: TFPHTTPClient;  Stream: TStringStream;  I, J: integer;  URL: string;  aStr: string;  aList: TStringList;  realm, nonce, qop, opaque, cnonce: string;  h1, h2, h3, h4: string;  URI: TURI;  UriStr: string;begin   login := 'guest';  pass := 'guest';  URL := 'https://jigsaw.w3.org/HTTP/Digest';  HTTPClient := TFPHTTPClient.Create(nil);  Stream := TStringStream.Create;  aList := TStringList.Create;  try     URI := ParseURI(URL, False);    UriStr := URI.Path;    begin      if (Length(URI.Path) > 0) and ((Length(UriStr) = 0) or (UriStr[Length(UriStr)] <> '/')) then        UriStr := UriStr + '/';      UriStr := UriStr + URI.Document;    end;    if Length(URI.Params) > 0 then      UriStr := UriStr + '?' + URI.Params;    Memo1.Lines.Add(UriStr);     HTTPClient.KeepConnection := true;    HTTPClient.AllowRedirect := true;    HTTPClient.HTTPMethod('GET', URL, Stream, [200, 401]);    if HTTPClient.ResponseStatusCode = 401 then  begin       for I := 0 to HTTPClient.ResponseHeaders.Count - 1 do    begin        if LeftStr(uppercase(HTTPClient.ResponseHeaders.Strings[I]), 24) = 'WWW- AUTHENTICATE: DIGEST' then        begin          realm := '';        nonce := '';          qop := '';          opaque := '';          cnonce := md5Print(md5String(IntToStr(DateTimeToUnix(Now()))));         aList.Clear;          aList.StrictDelimiter := True;          aList.Delimiter := ',';          aList.DelimitedText := trim(Copy(HTTPClient.ResponseHeaders.Strings[I], 25));          for J := 0 to pred(aList.Count) do          begin            aStr := trim(aList.Strings[J]);            if LeftStr(aStr, 5) = 'realm' then realm := Copy(aStr, 7, Length(aStr)).DeQuotedString(#34);            if LeftStr(aStr, 5) = 'nonce' then nonce := Copy(aStr, 7, Length(aStr)).DeQuotedString(#34);            if LeftStr(aStr, 3) = 'qop' then qop := Copy(aStr, 5, Length(aStr)).DeQuotedString(#34);          if LeftStr(aStr, 6) = 'opaque' then opaque := Copy(aStr, 8, Length(aStr)).DeQuotedString(#34);          end;           h1 := md5Print(md5String(login + ':' + realm + ':' + pass));          h2 := md5Print(md5String('GET' + ':' + UriStr));          if (qop = 'auth') or (qop = 'auth-int') then            h3 := md5Print(md5String(h1 + ':' + nonce + ':00000001:' + cnonce + ':' + qop + ':' + h2))          else            h3 := md5Print(md5String(h1 + ':' + nonce + ':' + h2));           h4 := 'username=' + AnsiQuotedStr(login, #34);        h4 := h4 + ', realm=' + AnsiQuotedStr(realm, #34);          h4 := h4 + ', nonce=' + AnsiQuotedStr(nonce, #34);          h4 := h4 + ', uri=' + AnsiQuotedStr(UriStr, #34);          if (qop = 'auth') or (qop = 'auth-int') then          begin            h4 := h4 + ', qop=' + qop;            h4 := h4 + ', nc=00000001';          end;          h4 := h4 + ', cnonce=' + AnsiQuotedStr(cnonce, #34);          h4 := h4 + ', response=' + AnsiQuotedStr(h3, #34);          if opaque <> '' then          h4 := h4 + ', opaque=' + AnsiQuotedStr(opaque, #34);           HTTPClient.RequestHeaders.Add('Authorization: Digest ' + h4);         Stream.Clear; // clear the previous request in the stream          HTTPClient.HTTPMethod('GET', URL, Stream, [200]);           Memo1.Clear;          Memo1.Lines.Add('-----');          Memo1.Lines.Add(HTTPClient.ResponseHeaders.Text);          Memo1.Lines.Add('-----');          Memo1.Lines.Add(Stream.DataString);         end;      end;    end;  finally    HTTPClient.Free;    Stream.Free;    aList.Free;  end;end; ---------------------------------------------------------------------- DidaJI · 2020-08-25 05:30 Quote from: rvk on August 24, 2020, 05:37:18 pm It works, thank you very very much !!! Zdenek.