Tprocess - How to time out while sending data to stdin? lazarus / 62-General ---------------------------------------------------------------------- fedkad · 2021-08-04 14:41 I am currently trying to implement calling a process and communicating with it using pipes. The algorithm is something like this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var  P : TProcess;  outstr : String = '';  errstr : String = '';  buf : Array [0..BUFSIZ-1] of char;  oc, ec, maxoc, maxec,  i  : Integer;  exitcode : Integer = 0;  stime : QWord;  begin (*...*)  P := TProcess.Create(Nil);    try    P.Options := P.Options+[poUsePipes,poNoConsole];    P.Executable:= CmdFileName;    for i:=low(pars) to high(pars) do      P.Parameters.add(pars[i]);    P.Execute;    if instr <> '' then      P.Input.Write(instr[1], Length(instr));    P.CloseInput;    P.WaitOnExit(1000);     oc:=0;  maxoc := 500000;    ec:=0;  maxec := 200;    stime := GetTickCount64;    while p.Running or (p.output.NumBytesAvailable>0) or (p.stderr.NumBytesAvailable>0) do    begin      if p.output.NumBytesAvailable>0 then      begin        i := min(BUFSIZ,p.output.NumBytesAvailable);        p.output.read(buf,i);      if oc0 then      begin        i := min(BUFSIZ,p.stderr.NumBytesAvailable);        p.stderr.read(buf,i);        if ec4000 then break;    end;    (* *)    P.Terminate(exitcode);    if exitcode=0 then      exitcode := p.exitcode;  except    on E: Exception do    begin      errstr := E.Message + eol + CmdFileName;      exitcode := -1;    end;  end;  P.Free;  (* ... *)         As you see, I am trying to read stdout and stderr in a loop and if the calling process does not finish writing these outputs in a specific time period (4 secs) I am timing out and exiting the "read loop." However, I cannot do something similar while sending large amount of data to stdin of the called process. For example, I "P.Input.Write" (@line 20) a large amount of data and the called process does not even bother to read this data, my program hangs indefinitely, until I kill the called process. How can I gracefully time out in such a case? ---------------------------------------------------------------------- marcov · 2021-08-04 15:05 Look at the body of runcommandloop and its applications, e.g. https://forum.lazarus.freepascal.org/index.php/topic,50525.msg368880.h tml#msg368880 ---------------------------------------------------------------------- fedkad · 2021-08-04 15:27 @marcov In my case, the problem is that the call P.Input.Write(...); (@line 20) hangs. I need a method to time out this call or something similar to NumBytesAvailable in output and stderr. ---------------------------------------------------------------------- rsz · 2021-08-04 18:44 I'd outsource the process interaction into a separate thread (and class) and periodically check the current processing status in the main thread and if it's stuck too long on the same status then just kill the child process from the main thread and let the pipe break (SIGPIPE exception is raised on Linux, can't say for Windows). Here's a program that reproduces the hang when writing more than 256 bytes to it: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Hang; uses Classes, sysutils, iostream; var  Stream: TIOStream;  I: Integer;  B: Byte;begin  Stream := TIOStream.Create(iosInput);  try  // read first 256 bytes form stdin    for I := 1 to 256 do    begin    B := Stream.ReadByte;      WriteLn(Chr(B));    end;    // hang indefinitely    while True do    begin      Sleep(1000);    end;  finally    Stream.Free;  end;end. and here's a program that uses a thread and kills the child process and gracefully waits for the thread and frees it. Change the line marked CHANGEME to the above program that hangs. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; uses  {$ifdef UNIX}  cthreads, cmem, BaseUnix,  {$endif}  {$ifdef WINDOWS}  // uses for WINAPI  {$endif}  Classes, sysutils, Process, syncobjs  ; type  TMyThread = class(TThread)  private    FCriticalSection: TCriticalSection;    FProcessId: Integer;  public  constructor Create;    destructor Destroy; override;    procedure Execute; override;    function GetProcessId: Integer;    procedure SetProcessId(Value: Integer);    property ProcessId: Integer read GetProcessId write SetProcessId;  end; constructor TMyThread.Create;begin  inherited Create(True);   // start suspended, we want to manually start it  FreeOnTerminate := False; // we want to manually free it  FProcessId := -1;  FCriticalSection := TCriticalSection.Create;end; destructor TMyThread.Destroy;begin  FCriticalSection.Free;  inherited;end; function TMyThread.GetProcessId: Integer;begin  // use critical section to prevent race conditions  FCriticalSection.Acquire;  try    Result := FProcessId;  finally    FCriticalSection.Release;  end;end; procedure TMyThread.SetProcessId(Value: Integer);begin  // use critical section to prevent race conditions  FCriticalSection.Acquire;  try    FProcessId := Value;  finally    FCriticalSection.Release;  end;end; procedure TMyThread.Execute;var  P : TProcess;  InStr: String;  I: Integer;begin  P := TProcess.Create(Nil);  try    InStr := '';    SetLength(InStr, 1024000);    for I := Low(InStr) to High(InStr) do    begin      InStr[I] := 'A';    end;    P.Options := P.Options+[poUsePipes,poNoConsole];    P.Executable:= '/path/to/program/hang'; // CHANGE ME    P.Execute;    ProcessId := P.ProcessId;    P.Input.Write(instr[1], Length(instr));    P.CloseInput;    P.WaitOnExit;  finally    P.Free;  end;end; procedure KillProcessById(PID: Integer);begin  if PID = -1 then    raise Exception.Create('Invalid PID...');  {$ifdef WINDOWS}  // implement for windows with WINAPI function to kill process  {$endif}  {$ifdef UNIX}  FpKill(PID, SIGKILL);  {$endif}end; var  T: TMyThread;begin  T := TMyThread.Create;  try    T.Start;    Sleep(3000);    KillProcessById(T.ProcessId);    T.WaitFor;  finally    T.Free;  end;end. As is with threads, care should be taken when accessing shared memory from multiple threads and care should also be taken if you decide to kill them. Implementing this in a non blocking read/write way without threading may also be possible, but could require changing the internals of TProcess and the streams it uses. ---------------------------------------------------------------------- Edson · 2021-08-04 20:01 I would recommend https://github.com/t-edson/UnTerminal It's a wrapper for a wrapper for TProcess with prompt detection and event driven. ---------------------------------------------------------------------- marcov · 2021-08-04 20:05 Quote from: fedkad on August 04, 2021, 05:27:34 pm@marcov In my case, the problem is that the call P.Input.Write(...); (@line 20) hangs. I need a method to time out this call or something similar to NumBytesAvailable in output and stderr. Yeah. Probably you need to switch to async I/O (writefile with overlapped record as param and then waitmultiple() on the result). On *nix iirc async I/O is done using the aio_write etc functions, but I don't know the details. Would be a pretty big rewrite of the pipes unit. ---------------------------------------------------------------------- fedkad · 2021-08-06 10:57 Quote from: marcov on August 04, 2021, 10:05:49 pmWould be a pretty big rewrite of the pipes unit. Yes. But, I am not going to do such a thing. It seems that the Tprocess.Input "output pipe stream" is missing something like, I would say, a "symmetrical" property that the Tprocess.Output or Tprocess.StdErr "input pipe streams" do have: Similar to the NumBytesAvailable property (the number of bytes available for reading from the input pipe stream), we should have something like a NumBytesAvailable property (the number of bytes available for writing to the output pipe stream without the Write function blocking). This should be 0 when the pipe is full and the size of pipe when the called program has completely read everything from the pipe and waiting for more input. Then, the calling program will be able to write as much as (but, not more than) the amount indicated in the NumBytesAvailable property and will wait until some space is freed in the pipe before writing more. In such a scenario, it would be very easy to time out such an operation, if the called program does not read data from its input pipe for a certain amount of time. ---------------------------------------------------------------------- PascalDragon · 2021-08-06 13:19 Quote from: fedkad on August 06, 2021, 12:57:31 pmIt seems that the Tprocess.Input "output pipe stream" is missing something like, I would say, a "symmetrical" property that the Tprocess.Output or Tprocess.StdErr "input pipe streams" do have: Similar to the NumBytesAvailable property (the number of bytes available for reading from the input pipe stream), we should have something like a NumBytesAvailable property (the number of bytes available for writing to the output pipe stream without the Write function blocking). This should be 0 when the pipe is full and the size of pipe when the called program has completely read everything from the pipe and waiting for more input. At least on Windows the pipes used for inter process communication do not provide this functionality, thus FPC's TOutputPipeStream can't provide it either. ---------------------------------------------------------------------- fedkad · 2021-08-06 13:29 Quote from: PascalDragon on August 06, 2021, 03:19:27 pmAt least on Windows the pipes used for inter process communication do not provide this functionality, thus FPC's TOutputPipeStream can't provide it either.   That was something I was afraid of. As another workaround, is there any possibility to automatically kill the called process after a certain amount of time passes (say 4 seconds of timeout in my case), so that any blocked Write function to the "output pipe stream" gets released? (Without resorting to multi- threaded programming.)