HTTP request over Unix Domain Socket lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- dubst3pp4 · 2017-12-02 08:55 Hello, I'm wondering if it is possible to send HTTP requests over the Unix Domain Socket? I want to query a service, which exposes an API via HTTP and Unix Domain Sockets, but I could not find out how to do this with TFPHTTPClient class... Any ideas? Best regards, Marc ---------------------------------------------------------------------- Leledumbo · 2017-12-02 16:49 It requires additional support to connect over unix domain socket, which I don't think TFPHTTPClient has. ---------------------------------------------------------------------- KemBill · 2017-12-02 19:16 I know "simple sockets", but if the only difference between UNIX and IP Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i server_sockaddr.sun_family = AF_UNIX; instead Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i server_sockaddr.sun_family = AF_INET; you can try to overload the method ---------------------------------------------------------------------- dubst3pp4 · 2017-12-03 16:47 It seems that the sockets unit is all what I need to connect to a Unix socket. Maybe subclassing the TFPHTTPClient class to use a Unix socket instead of an Internet socket should work... All the base HTTP functionality is already in the class, so it would be great if I could use it... ---------------------------------------------------------------------- KemBill · 2017-12-03 17:16 Are you sur you have nothing listening using netstat 127.0.0.1:80 ? It seems you can inject a TSocketHandler class into TFPHTTPClient, so you just need to write a wrapper using TUnixSocket ---------------------------------------------------------------------- dubst3pp4 · 2017-12-04 10:56 Quote from: KemBill on December 03, 2017, 06:16:59 pmAre you sur you have nothing listening using netstat 127.0.0.1:80 ? It seems you can inject a TSocketHandler class into TFPHTTPClient, so you just need to write a wrapper using TUnixSocket I'm trying to query the Docker daemon, which could be exposed to a real network host, but I want to avoid that. I've played around for some time and connecting to the daemon is as easy as: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i UnixSocket := TUnixSocket.Create('/var/run/docker.sock'); After that you can send the HTTP-Request with the WriteBuffer method and read data with the ReadBuffer method. Now the interesting part: the TFPHTTPClient class creates a TInetSocket instance for the private field FSocket, which is also of type TInetSocket, in the protected method ConnectToServer. There are no dependencies to the functionality of TInetSocket in any other methods, everywhere else just methods of the parent class TSocketStream of TInetSocket and TUnixSocket are used. So if I could change FSocket from a private to a protected field and override the ConnectToServer method in a subclass, all should be fine. Unfortunately to change the private field in TFPHTTPClient, I would have to create a copy of the fphttpclient unit, wouldn't I? Any chance to take another route to solve this problem? I also think it would be a good idea to contact the author of the unit (Michael Van Canneyt?) to propose such changes, so that the class is usable with Unix domain sockets, too (as more and more Linux services use HTTP over UDS, for example Docker and Node.js). ---------------------------------------------------------------------- dubst3pp4 · 2017-12-04 14:54 I got it now working by changing the type of the FSocket field to FSocketStream. In my subclass I'm extending TFPHttpCustomClient by reimplementing the ConnectToServer method, which in turn creates an instance of TUnixSocket, while the original ConnectToServer creates a TInetSocket instance. Maybe this is not the best solution, but it seems to work without problems. I would like to propose this change but don't know how or to whom... ---------------------------------------------------------------------- Leledumbo · 2017-12-04 19:53 Quote from: dubst3pp4 on December 04, 2017, 03:54:30 pmMaybe this is not the best solution, but it seems to work without problems. I would like to propose this change but don't know how or to whom...  Ask Michael, you already knew his name before. ---------------------------------------------------------------------- dubst3pp4 · 2017-12-05 04:57 Okay, I will send him an email ---------------------------------------------------------------------- istoica · 2022-01-25 10:10 Have you managed to complete this ? ---------------------------------------------------------------------- MarkMLl · 2022-01-25 10:15 Quote from: istoica on January 25, 2022, 11:10:24 amHave you managed to complete this ? Necroposting will get you in trouble these days. What are you trying to do: specifically use HTTP or use unix-domain sockets in general? If the latter then see this recent thread https:// forum.lazarus.freepascal.org/index.php/topic,57706.msg429320.html#msg4 29320 MarkMLl ---------------------------------------------------------------------- istoica · 2022-01-25 11:04 Thank you MarkMLI, I try to do both actually, exactly like the author of this thread. Make an HTTP request over unix domain sockets, there are APIs such as docker api or podman api that due to security reasons, they don't expose http servers over TCP https://docs.podman.io/en/latest/_static/api.html curl --unix-socket /run/podman/podman.sock http://d/v3.0.0/libpod/info - The curl client has built-in support for it these days - A famous JS http client library axios, has `socketPath` as argument to support it - https://axios-http.com/docs/req_config - Unix sockets are very useful for these scenarios too, even Windows supports it from 2017 - https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows So back to original, I simply want to use the simplicity of `TFPHTTPClient` using unix domain sockets. Tried to follow discoveries here, but arrived nowhere, segfault Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit2; {$mode ObjFPC}{$H+} interface uses  Classes, SysUtils, fphttpclient, ssockets, httpdefs, uriparser, strutils; type   TFPExHTTPClient = Class(TFPCustomHTTPClient)  private    FUnixSocketPath: string;    FSocket : TSocketStream;    FIOTimeout: Integer;    FConnectTimeout: Integer;  Public    Property UnixSocketPath: string Read FUnixSocketPath Write FUnixSocketPath;    Procedure ConnectToServer(const AHost: String; APort: Integer; UseSSL : Boolean=False); override;  end; implementation    procedure TFPExHTTPClient.ConnectToServer(const AHost: String;  APort: Integer; UseSSL : Boolean = False); Var  G : TSocketHandler;  begin  If IsConnected Then    DisconnectFromServer; // avoid memory leaks  if (Aport=0) then    if UseSSL then      Aport:=443    else      Aport:=80;  G:=GetSocketHandler(UseSSL);  if FUnixSocketPath = '' then    FSocket:=TInetSocket.Create(AHost,APort,G)  else    WriteLn(FUnixSocketPath);    FSocket:=TUnixSocket.Create(FUnixSocketPath);   try    if FIOTimeout<>0 then      FSocket.IOTimeout:=FIOTimeout;    if FConnectTimeout<>0 then      FSocket.ConnectTimeout:=FConnectTimeout;   //     if TypeOf(FSocket) is TInetSocket then    //  FSocket.Connect;  except    FreeAndNil(FSocket);    Raise;  end;end; end. ---------------------------------------------------------------------- MarkMLl · 2022-01-25 11:21 I can't help with the HTTP side of it... I'm not even going to start thinking about it since it's outside my field and I've got a lot of other stuff on my plate. I suggest stepping into TUnixSocket.Create() and seeing what's going wrong, and also looking at the earlier discussion. It might be something silly like a malformed path or access rights to (the file- like named entity implementing) the path, or it might be that that object isn't fully implemented. In extremis read through the earlier thread I cited and try opening the socket as a simple text file. Most of my experience with UD sockets has involved using them for datagrams, i.e. UDP-like rather than TCP-like. The problems were almost always on the server side (i.e. access rights etc. which necessitated my getting involved with POSIX Capabilities) but in the case of streams (TCP-like) there are also hazards when a connection has been shut down and not properly restarted. MarkMLl ---------------------------------------------------------------------- istoica · 2022-01-25 11:57 I have copy-pasted the entire fphttpclient unit and modified the following Modified private `FSocket : TInetSocket;` to `FSocket : TUnixSocket;`Added  private `FUnixSocketPath: String` to `TFPCustomHTTPClient`Added  public `Property UnixSocketPath : String Read FUnixSocketPath Write FUnixSocketPath;`Modified  public `TFPCustomHTTPClient.ConnectToServer` and commented out `FSocket.Connect;` as there is no Connect for TUnixSocket - one could add dummy Connect to TUnixSocket Now this beauty works properly, I get expected response Quotevar   S : String; begin   With TFPCustomHTTPClient.Create(nil) do     try        UnixSocketPath := '/tmp/podman.sock';       S := Get('http://d/v3.0.0/libpod/info');       WriteLn(S);     finally       Free;     end; It does the job, but it is so ... unpleasant to copy-paste the entire unit, it would be so cool if this is built-in ---------------------------------------------------------------------- MarkMLl · 2022-01-25 12:13 In that case you'll have to submit a patch or at least make a feature request providing your suggested implementation via https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues There was something I found I had to do to get basic sockets working... ah yes, Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   (* This is a reimplementation of the standard Connect() since it doesn't    support a unix-domain address and truncates it if told to cast it to an    inet domain address structure.     Test using e.g.  $ nc -lkU ~/dsocket  noting the -k to prevent the listener    terminating when the writer (this program) closes the connection.  *)  Function Connect(Sock:longint;const addr: TUnixSockAddr;var SockIn,SockOut:text):Boolean;      Function DoConnect(Sock:longint;const addr: TUnixSockAddr): Boolean;     var    res: longint;    begin      repeat        res:=fpconnect(Sock,@Addr,SizeOF(TUnixSockAddr)); (* NOTA BENE *)    until (res<>-1) or (SocketError <> EsockEINTR);      DoConnect:= res = 0;    end;    begin    Connect:=DoConnect(Sock,addr);    If Connect then       Sock2Text(Sock,SockIn,SockOut);  end { Connect } ;  noting the different address type, which is of course much the same thing that you tackled in your earlier posting. MarkMLl ---------------------------------------------------------------------- istoica · 2022-01-25 16:47 Thank you, I've did my best, I am not that big expert in FPC design / idioms https://gitlab.com/freepascal.org/fpc/source/-/merge_requests/151 ---------------------------------------------------------------------- czd · 2022-08-08 21:19 Hi, Sorry to open this post again, but I cant convert the fphttpclient for UnixSocket, can someone has the modified versions of this fphttpclient unit. Thanks. ---------------------------------------------------------------------- MarkMLl · 2022-08-09 07:16 Why? What problem are you having... error messages or run-time behaviour? In what way is the existing discussion (with attachments and links) unhelpful? Where's your example code? MarkMLl ---------------------------------------------------------------------- czd · 2022-08-09 10:59 Hi MarkMLI, Soon, I realized that link https://gitlab.com/freepascal.org/fpc/sourc e/-/merge_requests/151/diffs#diff- content-e2642c4dbb5c1d3691ddf9a8201235e1a41e1e0e. I'll try from that source. Thanks. ---------------------------------------------------------------------- MarkMLl · 2022-08-09 11:02 Otherwise see some of the snippets I've posted in this thread. I went through the various possibilities and got all working (not necessarily using FPC library code) for piping (term used loosely) text between two different programs a few months ago. But /please/ bear in mind that this is OS-specific etc., and that you really do need to include information on what you're using in the body of your message even if it's non-obvious why. MarkMLl