Why TProcess gives deadlock, whereas the same thing can be run in the
terminal ?
lazarus / 66-Unix

----------------------------------------------------------------------
comingnine · 2017-11-13 14:49

I would like to use TProcess to run a simple series of shell commands
and collect the tiny amount of output.

As can be shown from ~/test.sh or ~/test_oneline.sh, this series of
commands start script1 and then look for script2, and reports
script2's PID and stops when script2 is found.

Both script file runs well in the terminal. However, when using with
TProcess as shown below, the executable hangs for ever, indicating
certain deadlock. However, the output is really small and the output
pipe should not be full. I am really lost about the reason why the
deadlock occurs here... Can you help me to solve this problem ? Many
thanks !

Code: Bash  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
[vagrant@localhost ~]$ ps aux | grep [^]]script[vagrant@localhost ~]$
ps aux | grep [^]]sleep[vagrant@localhost ~]$ cat ~/script1#!/bin/bash
-xsleep 20~/script2 [vagrant@localhost ~]$ cat ~/script2#!/bin/bash
-xsleep 4320[vagrant@localhost ~]$ cat ~/test.sh#!/bin/bash -x nohup
~/script1 & bg_id=$!disown -awhile true ; do   if ! ps -p $bg_id >
/dev/null 2>&1 ; then     echo "script1 terminated"    break  fi 
str=$(ps aux | grep [^]]script2) 
BashRegex='^[[:alnum:]_]+\s+([[:alnum:]_]+)'  if [[ $str =~ $BashRegex
]] ; then     for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do      
echo ${BASH_REMATCH[$i]}    done    break  else     sleep 5 
fidone[vagrant@localhost ~]$ cat ~/test_oneline.sh#!/bin/bash -xnohup
~/script1 & bg_id=$! ; disown -a ; while true ; do if ! ps -p $bg_id >
/dev/null 2>&1 ; then echo "script1 terminated" ; break; fi ; str=$(ps
aux | grep [^]]script2) ; BashRegex='^[[:alnum:]_]+\s+([[:alnum:]_]+)'
; if [[ $str =~ $BashRegex ]] ; then for (( i=1 ;
i<${#BASH_REMATCH[@]} ; i++)) ; do echo ${BASH_REMATCH[$i]} ; done ;
break ; else sleep 5 ; fi ; done[vagrant@localhost ~]$ 

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
test_freepascal;{$IFDEF FPC} {$MODE Delphi} {$ELSE} {$APPTYPE CONSOLE}
{$ENDIF}uses SysUtils, Classes, Process; procedure
Test_TProcess_Executable_parameters; // http://wiki.freepascal.org/Exe
cuting_External_Programs#Reading_large_outputconst BUF_SIZE = 2048; //
Buffer size for reading the output in chunksvar  Proc: TProcess; 
OutStream, ErrStream: TStream;  OutLines, ErrLines: TStrings; 
BytesRead: LongInt;  Buffer: array[1..BUF_SIZE] of Byte;begin  Proc :=
TProcess.Create(nil);  try    Proc.Executable := '/bin/bash';   
Proc.Parameters.Add('-c');    Proc.Parameters.Add('nohup ~/script1 &
bg_id=$! ; disown -a ; while true ; do if ! ps -p $bg_id > /dev/null
2>&1 ; then echo "script1 terminated" ; break; fi ; str=$(ps aux |
grep [^]]script2) ; BashRegex=''^[[:alnum:]_]+\s+([[:alnum:]_]+)'' ;
if [[ $str =~ $BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]}
; i++)) ; do echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ;
fi ; done');    //Proc.Options := [poUsePipes, poWaitOnExit]; // Does
not work...    Proc.Options := [poUsePipes]; // Does not work...   
Proc.Execute;     OutStream := nil; ErrStream := nil; OutLines := nil;
ErrLines := nil;    try      OutStream := TMemoryStream.Create;
ErrStream := TMemoryStream.Create; OutLines := TStringList.Create;
ErrLines := TStringList.Create;       repeat        BytesRead :=
Proc.Output.Read(Buffer, BUF_SIZE);        OutStream.Write(Buffer,
BytesRead)      until BytesRead = 0;      OutStream.Position := 0;   
 OutLines.LoadFromStream(OutStream);       repeat        BytesRead :=
Proc.StdErr.Read(Buffer, BUF_SIZE);        ErrStream.Write(Buffer,
BytesRead)      until BytesRead = 0;      ErrStream.Position := 0;   
 ErrLines.LoadFromStream(ErrStream);       Writeln('##### Out #####' +
sLineBreak + OutLines.Text + sLineBreak + '##### Err #####' +
sLineBreak + ErrLines.Text);    finally      OutStream.Free;
ErrStream.Free; OutLines.Free; ErrLines.Free;    end;   finally   
Proc.Free;  end;end; begin  Test_TProcess_Executable_parameters;end.

----------------------------------------------------------------------
tudi_x · 2017-11-13 15:06

for mere mortals would you mind uploading the two script files and the
app folder.
i think i have a mint box.

----------------------------------------------------------------------
comingnine · 2017-11-13 15:24

Many thanks for your efforts !
The binary file is rather large and could not be uploaded as
attachment.
The two script and the .lpr/.lpi files are attached.

----------------------------------------------------------------------
tudi_x · 2017-11-13 15:38

i would place the code in the scripts instead of TProcess parameters.
cat ~/test.sh in your initial post gives a lot of content whereas the
scripts you loaded are very few lines.

if you have several commands maybe loop them in a TProcess execution
for each.

----------------------------------------------------------------------
comingnine · 2017-11-13 15:42

As shown in the .lpr content below, the RunCommand version works as
expected.  
However, it seems that RunCommand can only provide merged StdOut &
StdErr, and I would like to retrieve StdErr content separately to see
more clearly if something goes wrong.

There must be some root difference between RunCommand internal and the
wiki usage of TProcess. Your insights will be appreciated very much 

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
test_freepascal_runcommand;{$IFDEF FPC} {$MODE Delphi} {$ELSE}
{$APPTYPE CONSOLE} {$ENDIF}uses SysUtils, Classes, Process; //
http://wiki.freepascal.org/Executing_External_Programs// http://forum.
lazarus.freepascal.org/index.php/topic,38398.0.htmlprocedure
Test_RunCommand;var  Output: string;begin  RunCommand(   
'/bin/bash',    ['-c',     'nohup ~/script1 & bg_id=$! ; disown -a ;
while true ; do if ! ps -p $bg_id > /dev/null 2>&1 ; then echo
"script1 terminated" ; break; fi ; str=$(ps aux | grep [^]]script2) ;
BashRegex=''^[[:alnum:]_]+\s+([[:alnum:]_]+)'' ; if [[ $str =~
$BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do
echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ; fi ; done'  
],    Output,    []  );  Writeln(Output);end; begin 
Test_RunCommand;end.

----------------------------------------------------------------------
comingnine · 2017-11-13 15:48

Quote from: tudi_x on November 13, 2017, 04:38:39 pmi would place the
code in the scripts instead of TProcess parameters.
cat ~/test.sh in your initial post gives a lot of content whereas the
scripts you loaded are very few lines.

Both ~/test.sh and ~/test_oneline.sh works as expected.
Furthermore, calling directly /home/vagrant/test.sh as
TProcess.Executable does not work either. I am confused.

Quote from: tudi_x on November 13, 2017, 04:38:39 pmif you have
several commands maybe loop them in a TProcess execution for each.

Multiple TProcess.... Two TProcess instances in the wiki page already
look complicated enough.

----------------------------------------------------------------------
comingnine · 2017-11-14 08:17

A working version which mimics the Process.internalRuncommand
procedure...
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i procedure
MoreTest_TProcess_Executable_Parameters_ReadOnlyIfAvailable; // Try to
mimic Process.internalRuncommandconst BUF_SIZE = 65536; // Read at
most BUF_SIZE bytes at one timevar  Proc: TProcess;                
OutMemStream, ErrMemStream: TMemoryStream;  OutNumBytes, ErrNumBytes:
LongInt;  OutLines, ErrLines: TStrings;  Available, BytesRead:
LongInt;begin  Proc := TProcess.Create(nil);  try    //Proc.Executable
:= '/home/vagrant/test.sh';    //(*    Proc.Executable :=
'/bin/bash';    Proc.Parameters.Add('-c');   
Proc.Parameters.Add('nohup ~/script1 & bg_id=$! ; disown -a ; while
true ; do if ! ps -p $bg_id > /dev/null 2>&1 ; then echo "script1
terminated" ; break; fi ; str=$(ps aux | grep [^]]script2) ;
BashRegex=''^[[:alnum:]_]+\s+([[:alnum:]_]+)'' ; if [[ $str =~
$BashRegex ]] ; then for (( i=1 ; i<${#BASH_REMATCH[@]} ; i++)) ; do
echo ${BASH_REMATCH[$i]} ; done ; break ; else sleep 5 ; fi ; done'); 
  //*)    //Proc.Options := [poUsePipes, poWaitOnExit]; // Also
works...    Proc.Options := [poUsePipes];    Proc.Execute;    
OutMemStream := nil;    ErrMemStream := nil;    OutLines := nil;   
ErrLines := nil;    try      OutMemStream := TMemoryStream.Create;   
 ErrMemStream := TMemoryStream.Create;      OutLines :=
TStringList.Create;      ErrLines := TStringList.Create;      
OutNumBytes := 0;      ErrNumBytes := 0;      while Proc.Active or
(Proc.Output.NumBytesAvailable > 0) or (Assigned(Proc.Stderr) and
(Proc.Stderr.NumBytesAvailable > 0)) do      begin        Available :=
Proc.Output.NumBytesAvailable;        while Available > 0 do // Only
read if data from corresponding stream is available. Otherwise, read
blocks on linux        begin          if Available > BUF_SIZE then   
     begin            Available := BUF_SIZE; // Read at most BUF_SIZE
bytes at one time          end;          OutMemStream.Size :=
OutNumBytes + Available; // Allocate storage in advance         
BytesRead := Proc.Output.Read((OutMemStream.Memory + OutNumBytes)^,
Available); // Read data into stream storage          Assert(BytesRead
= Available);          if BytesRead > 0 then          begin          
Inc(OutNumBytes, BytesRead);          end;          Available :=
Proc.Output.NumBytesAvailable;        end;         if
Assigned(Proc.Stderr) then // Check for assigned(P.stderr) to prevent
AV if poStderrToOutput in p.Options.        begin                    
Available := Proc.Stderr.NumBytesAvailable;          while Available >
0 do          begin                   if Available > BUF_SIZE then   
       begin              Available := BUF_SIZE; // Read at most
BUF_SIZE bytes at one time            end;           
ErrMemStream.Size := ErrNumBytes + Available; // Allocate storage in
advance            BytesRead := Proc.StdErr.Read((ErrMemStream.Memory
+ ErrNumBytes)^, Available); // Read data into stream storage        
  Assert(BytesRead = Available);            if BytesRead > 0 then    
      begin              Inc(ErrNumBytes, BytesRead);            end;
              Available := Proc.Stderr.NumBytesAvailable;         
end;        end;      end;                 OutMemStream.Size :=
OutNumBytes;      ErrMemStream.Size := ErrNumBytes;      
OutMemStream.Position := 0;     
OutLines.LoadFromStream(OutMemStream);       ErrMemStream.Position :=
0;      ErrLines.LoadFromStream(ErrMemStream);       Writeln('#####
Out #####' + sLineBreak + OutLines.Text + sLineBreak + '##### Err
#####' + sLineBreak + ErrLines.Text);    finally     
OutMemStream.Free;      ErrMemStream.Free;      OutLines.Free;     
ErrLines.Free;    end;  finally    Proc.Free;  end;end;