[SOLVED] Output from TProcess breaks lines over 80 characters lazarus / 31-LCL ---------------------------------------------------------------------- guest62822 · 2018-07-19 18:51 I have managed to use TProcess to run a command-line tool and output all messages in a component in my application. The issue is, if a line exceeds 80 characters, it breaks into two lines. Is there any way to fix this? This is not all the code, but shows how I run the process and get the output: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TThreadRunProcess.UpdateTerminal;begin   UseForm.MemoTerminal.Lines.Add(ReadBuffer);  UseForm.MemoTerminal.Update; end; procedure TThreadRunProcess.Execute;var  RunProcess  : TProcess;begin   FreeOnTerminate:= true;   { Start process }  RunProcess:= TProcess.Create(nil);   try     RunProcess.Executable:= ProcessExecutable;    if ProcessFlags <> '' then     RunProcess.Parameters.Add(ProcessFlags);    if ProcessFileName <> '' then     RunProcess.Parameters.Add(ProcessFileName);     RunProcess.Options := [poUsePipes];     RunProcess.Execute;     { Get messages while the process is running }    while ProcessIsRunning do begin       if (not Runprocess.Running) and (RunProcess.Output.NumBytesAvailable = 0)       then break;       if RunProcess.Output.NumBytesAvailable > 0 then begin         SetLength(ReadBuffer, RunProcess.Output.NumBytesAvailable);         RunProcess.Output.Read(ReadBuffer[1],         RunProcess.Output.NumBytesAvailable);         Synchronize(@UpdateTerminal);       end;     end;   finally     RunProcess.Free;   end;   { Complete thread }  Synchronize(@Complete); end; ReadBuffer is a string in this case. ---------------------------------------------------------------------- Leledumbo · 2018-07-20 09:20 A little less than enough information, like who makes it exceed 80 characters? It can be both the executing and executed program. Personally I would do: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i UseForm.MemoTerminal.Lines.Text := UseForm.MemoTerminal.Lines.Text + ReadBuffer;ReadBuffer := '';UseForm.MemoTerminal.Update; instead in UpdateTerminal. There's a little dangerous point in the output reading code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SetLength(ReadBuffer, RunProcess.Output.NumBytesAvailable); // (1)       RunProcess.Output.Read(ReadBuffer[1],         RunProcess.Output.NumBytesAvailable); // (2) (2) may be larger than (1) and that can cause buffer overflow. Save it to a local variable before SetLength and use the same variable for Read, that's the safe way. ---------------------------------------------------------------------- marcov · 2018-07-20 09:40 I'm not sure, but can't you simply enable scrollbars for your memo ? Then it won't break lines. ---------------------------------------------------------------------- guest62822 · 2018-07-20 10:06 Quote from: Leledumbo on July 20, 2018, 11:20:44 amA little less than enough information, like who makes it exceed 80 characters? It's the command-line tool that output lines over 80 characters. It seems like the TProcess assumes the maximum length per line is 80 and automatically add a #13 + #10 (CR) to break the lines. I tried to filter this out, but this will also filter out the CRs that's supposed to be there and multiple lines will now be on one line. I can't guarantee other command-line tools. but I guess a solution would be to change my own comand-line tool so it splits the messages into multiple lines. Quote from: marcov on July 20, 2018, 11:40:31 amI'm not sure, but can't you simply enable scrollbars for your memo ? Then it won't break lines. It's not related to the memo component, I can type longer lines manually. ---------------------------------------------------------------------- marcov · 2018-07-20 10:19 Tprocess is binary, but doesn't return "whole" lines. So it might read in 80 bytes increments, but that is windows. Probably the adding to a memo adds the linefeeds. So you need to buffer what you read in the tprocess loop till you have a line  with a (cr)lf, and only then pass to memo. ---------------------------------------------------------------------- guest62822 · 2018-07-20 11:13 Quote from: marcov on July 20, 2018, 12:19:27 pmTprocess is binary, but doesn't return "whole" lines. So it might read in 80 bytes increments, but that is windows. Probably the adding to a memo adds the linefeeds. So you need to buffer what you read in the tprocess loop till you have a line  with a (cr)lf, and only then pass to memo. I did a test and noticed that the procedure UpdateTerminal was only called twice and the output is like 12 lines in total. So it grabs multiple lines at once and it seem like the output already contains all the (cr)lf. The problem is, I can't really guess which ones to remove. I could assume if I read 80 characters and find a (cr)lf, then the line is broken into two lines, but what IF the line is exactly 80 characters long? edit: However, I had a quick look on the Internet and it's seems to be a good practice to keep lines under 80 characters, so I think the best solution would be to modify my command-line tool and make sure that no lines exceed 80 characters. ---------------------------------------------------------------------- sash · 2018-07-23 18:13 C's printf("string\r\n"); or Pascal's WriteLn('string'); already have (CR|LF) in the end. MemoTerminal.Lines.Add(ReadBuffer); - And this code adds additional CR for each call. Just echo everything as is: Leledumbo gave you a code that should work. Quote from: gnu_trisquel on July 20, 2018, 01:13:34 pmHowever, I had a quick look on the Internet and it's seems to be a good practice to keep lines under 80 characters A good practice is to to respect actual console session's setting (MODE command for dos/windows), but I don't see how it matters, if you're piping output. ---------------------------------------------------------------------- guest62822 · 2018-07-23 19:27 QuoteC's printf("string\r\n"); or Pascal's WriteLn('string'); already have (CR|LF) in the end. MemoTerminal.Lines.Add(ReadBuffer); - And this code adds additional CR for each call. Just echo everything as is: Leledumbo gave you a code that should work. Yes, I have changed the code according to Leledumbo's suggestions. It reduces a few extra linefeeds, but it doesn't solve my main issue - lines that's longer than 80 characters is still split into multiple lines - ReadBuffer already contains a (LF), even before adding them to the Memo component. ReadBuffer = #1: The first 80 characters(LF) <-- Added by TProcess #2: The rest of the line(LF) So, my question is: Is it possible to change the behavior of TProcess? ---------------------------------------------------------------------- engkin · 2018-07-23 20:08 Quote from: gnu_trisquel on July 23, 2018, 09:27:56 pmReadBuffer = #1: The first 80 characters(LF) <-- Added by TProcess #2: The rest of the line(LF) So, my question is: Is it possible to change the behavior of TProcess? AFAIR, TProcess does *not* change the stream. It does not add LF. In fact it uses THandleStream. ---------------------------------------------------------------------- guest62822 · 2018-07-23 21:16 Quote from: engkin on July 23, 2018, 10:08:38 pmQuote from: gnu_trisquel on July 23, 2018, 09:27:56 pmReadBuffer = #1: The first 80 characters(LF) <-- Added by TProcess #2: The rest of the line(LF) So, my question is: Is it possible to change the behavior of TProcess? AFAIR, TProcess does *not* change the stream. It does not add LF. In fact it uses THandleStream. Ah, I see. Well, the only solution I have for now is to keep the output lines (from my command-line tool) shorter than 80 characters. ---------------------------------------------------------------------- sash · 2018-07-23 21:25 Quote from: gnu_trisquel on July 23, 2018, 09:27:56 pm#1: The first 80 characters(LF) <-- Added by TProcess Normally it should not, unless you set WindowColumns or similar properties (are you on Windows?) Are you sure your child process (isn't it chained like with more?) does not chop lines by itself? To check, run it from console with output redirection like Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i childproc -args > output.txt ---------------------------------------------------------------------- guest62822 · 2018-07-23 21:47 Quote from: sash on July 23, 2018, 11:25:15 pmQuote from: gnu_trisquel on July 23, 2018, 09:27:56 pm#1: The first 80 characters(LF) <-- Added by TProcess Normally it should not, unless you set WindowColumns or similar properties (are you on Windows?) Are you sure your child process (isn't it chained like with more?) does not chop lines by itself? To check, run it from console with output redirection like Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i childproc -args > output.txt The command-line tool is written in Pascal (compiled with FPC) and use Writeln to output the messages. I'm a Linux user, so no clue if it behave differently on Windows. I guess I can give it a try. ---------------------------------------------------------------------- 440bx · 2018-07-23 22:49 Quote from: gnu_trisquel on July 23, 2018, 09:27:56 pmIt reduces a few extra linefeeds, but it doesn't solve my main issue - lines that's longer than 80 characters is still split into multiple lines - ReadBuffer already contains a (LF), even before adding them to the Memo component. I only know Windows and only the API (I don't use components), I don't know if what I am going to mention is applicable in the case you've described but... just in case.  When you output to a console in Windows, the default width of the console is 80 characters.  Whenever a program writes 80 characters or more onto the console, the line automatically gets split into two (since it does not fit on one line.) In Windows, you can programmatically control the width and height of the console.  In Windows, the solution would simply be to make the console wider, that would solve the problem you are experiencing.  I presume that this is possible in the OS you are using but, I have no idea how it's done but, I figured you might.  If you can and know how, simply make the console window wider. I hope that comment might be useful to you. ---------------------------------------------------------------------- Leledumbo · 2018-07-24 00:09 Quote from: gnu_trisquel on July 23, 2018, 09:27:56 pm#1: The first 80 characters(LF) <-- Added by TProcess Before outputting to a TMemo, try outputting to a console. I'm still not clear who wraps that 80 characters, it could be the TMemo itself. But I assure you TProcess doesn't mess with it at all. Quote from: engkin on July 23, 2018, 10:08:38 pmAFAIR, TProcess does *not* change the stream. It does not add LF. In fact it uses THandleStream. Correct. ---------------------------------------------------------------------- guest62822 · 2018-07-24 01:07 Quote from: Leledumbo on July 24, 2018, 02:09:10 amQuote from: gnu_trisquel on July 23, 2018, 09:27:56 pm#1: The first 80 characters(LF) <-- Added by TProcess Before outputting to a TMemo, try outputting to a console. I'm still not clear who wraps that 80 characters, it could be the TMemo itself. But I assure you TProcess doesn't mess with it at all. Quote from: engkin on July 23, 2018, 10:08:38 pmAFAIR, TProcess does *not* change the stream. It does not add LF. In fact it uses THandleStream. Correct. I did check the whole string (the content of ReadBuffer) by outputting character by character - I wrote '(cr)' for character code 13 and '(lf)' for character code 10. So, I can confirm that a LF is already added at this point (before the string is added to TMemo). ---------------------------------------------------------------------- Jurassic Pork · 2018-07-24 04:41 hello, Quote from: gnu_trisquel on July 19, 2018, 08:51:38 pmI have managed to use TProcess to run a command-line tool and output all messages in a component in my application. you have better than TProcess to do that :  TProcessEx in the unit ProcessUtils of fpcUpDeluxe project. TprocessEx includes Real time output management. Example under linux to have the output of my biglines executable in realtime in a Tmemo : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, FileUtil, Forms, Controls, Graphics,  Dialogs, StdCtrls, LazUtf8, Processutils; type   { TForm1 }   TForm1 = class(TForm)    BtGo: TButton;    MemConsole: TMemo;    procedure BtGoClick(Sender: TObject);  private   public    procedure ProcessOutput(Sender:TProcessEx; output:string);    procedure ProcessError(Sender:TProcessEx; {%H-}IsException:boolean);  end; var  Form1: TForm1; implementation {$R *.lfm} { TForm1 } procedure TForm1.BtGoClick(Sender: TObject);var Proc: TProcessEx;begin   try   Proc := TProcessEx.Create(nil);   Proc.Executable := '/home/jurassic/Lazarus/projets/biglines_output/biglines';   Proc.OnErrorM:=@(ProcessError);   Proc.OnOutputM:=@(ProcessOutput);   Proc.Execute();   finally     Proc.Free;   end;end; procedure TForm1.ProcessError(Sender: TProcessEx; IsException: boolean);begin   MemConsole.Lines.Append('Erreur ! ' + Sender.ExceptionInfo);end; procedure TForm1.ProcessOutput(Sender: TProcessEx; output : String);begin MemConsole.Lines.Text :=  MemConsole.Lines.Text + output;  // si vous avez des problème d'accent  //MemConsole.Lines.Text := MemConsole.Lines.Text + ConsoleToUtf8(output);  // pour scroll automatique  MemConsole.SelStart := Length(MemConsole.Lines.Text)-1;  MemConsole.SelLength:=0;end; end.the biglines executable writes two lines of 120 characters : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program biglines;var myline : String;beginmyline := StringofChar('a',120);writeln(myline);myline := StringofChar('b',120);writeln(myline);end. The result  under Lubuntu 16.04  Lazarus 1.8.2 is in attachment. The lines with 120 characters are correctly displayed. The project is in attachment. Processutils.pas coming from fpcUpDeluxe project has a MIT or LGPL licence. Friendly, J.P ---------------------------------------------------------------------- Leledumbo · 2018-07-24 05:45 Quote from: gnu_trisquel on July 24, 2018, 03:07:59 amI did check the whole string (the content of ReadBuffer) by outputting character by character - I wrote '(cr)' for character code 13 and '(lf)' for character code 10. So, I can confirm that a LF is already added at this point (before the string is added to TMemo). Right, so it's the called program that does the line break. Gotta modify that one. ---------------------------------------------------------------------- guest62822 · 2018-07-24 09:59 Quoteyou have better than TProcess to do that :  TProcessEx in the unit ProcessUtils of fpcUpDeluxe project. TprocessEx includes Real time output management. Example under linux to have the output of my biglines executable in realtime in a Tmemo : Great, I will have a look into that. QuoteRight, so it's the called program that does the line break. Gotta modify that one. The command-line program is a program written in Pascal (compiled with FPC) and I use Writeln to output messages. I haven't added anything on my own to break lines over 80 characters. As 440bx said, it's almost like the program was run in a 80 character wide console (or terminal), and that's why the lines gets cut off. Many terminals in Linux has a character width of 80 by default as well. But my current solution is, I have made a few changes and try to make sure that lines doesn't exceed 80 characters. ---------------------------------------------------------------------- 440bx · 2018-07-24 10:16 Quote from: gnu_trisquel on July 24, 2018, 11:59:29 amAs 440bx said, it's almost like the program was run in a 80 character wide console (or terminal), and that's why the lines gets cut off. Many terminals in Linux has a character width of 80 by default as well. But my current solution is, I have made a few changes and try to make sure that lines doesn't exceed 80 characters. I've often run into the problem of having lines longer than 80 characters wrap around.  As I mentioned previously, one solution is to programmatically control the width of the terminal/console.  The disadvantage of that solution is that your program has to know beforehand what the longest line is going to be (or make a guess).  The other solution, which IMO is a better one is, to always redirect the output to a file, that way there won't be any wrapping around no matter how long the longest line is.    Once the output is in a file, you can have your own way of displaying what is in it with full control over how each line is formatted.  Basically, you implement a simple file viewer that does horizontal scrolling if necessary. Another advantage of the second solution is that it is a LOT faster to write to a file than to the console/terminal and there is no guessing about the length of the longest line. Of course, changing the program to ensure its lines are less than 80 characters is nice. ---------------------------------------------------------------------- sash · 2018-07-24 10:22 It would be better if you upload a minimal test project, that reproduces a possible bug, so we can test it. You said you are linux user, but I cannot reproduce your case on linux. ---------------------------------------------------------------------- guest62822 · 2018-07-24 10:30 QuoteAs I mentioned previously, one solution is to programmatically control the width of the terminal/console. TProcess has the property WindowColumns, but it's for Windows only. So, I'm not sure if there is other ways to control this. QuoteOf course, changing the program to ensure its lines are less than 80 characters is nice. Yes, it works quite well. ---------------------------------------------------------------------- guest62822 · 2018-07-26 13:59 Just an update: I tried to run my application in Windows 7. The command-line tool run as expected when running it directly from the DOS prompt. However, when starting it from the main application (using TProcess), the DOS prompt shows up and it's completely empty - which also means that no output has been generated (no ReadBuffer --> empty TMemo). It's the same code, so no clue why it doesn't work in Windows, or if it's a Windows thing to not show output in console in some cases. ---------------------------------------------------------------------- M+AUDIO · 2018-07-26 14:16 Quote from: gnu_trisquel on July 26, 2018, 03:59:26 pmwhen starting it from the main application (using TProcess), the DOS prompt shows up and it's completely emptyTry and see if this helps: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     RunProcess.Options := [poUsePipes,poNoConsole]; ---------------------------------------------------------------------- guest62822 · 2018-07-26 14:32 Quote from: M+AUDIO on July 26, 2018, 04:16:43 pmQuote from: gnu_trisquel on July 26, 2018, 03:59:26 pmwhen starting it from the main application (using TProcess), the DOS prompt shows up and it's completely emptyTry and see if this helps: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     RunProcess.Options := [poUsePipes,poNoConsole];  I tried it - it stops the console to appear, but still no output. ---------------------------------------------------------------------- Jurassic Pork · 2018-07-26 23:40 hello, with TprocessEx it still works under windows Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.BtGoClick(Sender: TObject);var Proc: TProcessEx;begin   try   Proc := TProcessEx.Create(nil);   Proc.Executable :='biglines.exe';     Proc.OnErrorM:=@(ProcessError);   Proc.OnOutputM:=@(ProcessOutput);   Proc.Execute();   finally     Proc.Free;   end;end;         Friendly, J.P ---------------------------------------------------------------------- guest62822 · 2018-07-27 12:04 Quote from: Jurassic Pork on July 27, 2018, 01:40:37 amhello, with TprocessEx it still works under windows Friendly, J.P I tried ProcessEx - I had to rewrite some code, but sadly, I got exactly the same results. I will see if I can find a fix or maybe go for a different solution (maybe skip processes and integrate the command-line tool into the main application and run it as a thread). ---------------------------------------------------------------------- guest62822 · 2018-07-28 09:22 I finally solved it! It was the command-line tool (or child process) that didn't work as expected in Windows. Like I said earlier, the program worked fine when running it directly in the DOS prompt, but for some reason didn't generate any output (in Windows only) when running it from the main application using TProcess(Ex). The changes I made to the command-line tool was: 1) I added a missing Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}{$H+} 2) I replaced "Crt" with "Classes". (no clue what I used "Crt" for in the past) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  Classes, ... I have no clue how this fixed the output, but now it works. In Windows, I don't get any lines cut off after 80 characters, so I assume that's a Linux thing. ---------------------------------------------------------------------- tatamata · 2021-03-15 19:16 Quote from: Jurassic Pork on July 24, 2018, 06:41:06 amhello, Quote from: gnu_trisquel on July 19, 2018, 08:51:38 pmI have managed to use TProcess to run a command-line tool and output all messages in a component in my application. you have better than TProcess to do that :  TProcessEx in the unit ProcessUtils of fpcUpDeluxe project. TprocessEx includes Real time output management. Example under linux to have the output of my biglines executable in realtime in a Tmemo : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit Unit1; {$mode objfpc}{$H+} interface uses  Classes, SysUtils, FileUtil, Forms, Controls, Graphics,  Dialogs, StdCtrls, LazUtf8, Processutils; type   { TForm1 }   TForm1 = class(TForm)    BtGo: TButton;    MemConsole: TMemo;    procedure BtGoClick(Sender: TObject);  private   public    procedure ProcessOutput(Sender:TProcessEx; output:string);    procedure ProcessError(Sender:TProcessEx; {%H-}IsException:boolean);  end; var  Form1: TForm1; implementation {$R *.lfm} { TForm1 } procedure TForm1.BtGoClick(Sender: TObject);var Proc: TProcessEx;begin   try   Proc := TProcessEx.Create(nil);   Proc.Executable := '/home/jurassic/Lazarus/projets/biglines_output/biglines';   Proc.OnErrorM:=@(ProcessError);   Proc.OnOutputM:=@(ProcessOutput);   Proc.Execute();   finally     Proc.Free;   end;end; procedure TForm1.ProcessError(Sender: TProcessEx; IsException: boolean);begin   MemConsole.Lines.Append('Erreur ! ' + Sender.ExceptionInfo);end; procedure TForm1.ProcessOutput(Sender: TProcessEx; output : String);begin MemConsole.Lines.Text :=  MemConsole.Lines.Text + output;  // si vous avez des problème d'accent  //MemConsole.Lines.Text := MemConsole.Lines.Text + ConsoleToUtf8(output);  // pour scroll automatique  MemConsole.SelStart := Length(MemConsole.Lines.Text)-1;  MemConsole.SelLength:=0;end; end.the biglines executable writes two lines of 120 characters : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program biglines;var myline : String;beginmyline := StringofChar('a',120);writeln(myline);myline := StringofChar('b',120);writeln(myline);end. The result  under Lubuntu 16.04  Lazarus 1.8.2 is in attachment. The lines with 120 characters are correctly displayed. The project is in attachment. Processutils.pas coming from fpcUpDeluxe project has a MIT or LGPL licence. Friendly, J.P Hello, Processutils.pas coming from fpcUpDeluxe does not contain TProcessEx class? Did I miss something or it is renamed in meantime? Where can it be downloaded? ---------------------------------------------------------------------- Roland57 · 2021-03-15 20:25 Quote from: tatamata on March 15, 2021, 08:16:54 pmProcessutils.pas coming from fpcUpDeluxe does not contain TProcessEx class? Did I miss something or it is renamed in meantime? Where can it be downloaded? Indeed, it seems that the unit has been remade. The old version of the unit is included in the project attached above. Regards. Roland