commandline behaviour in TProcess (different from shell) lazarus / 1-Linux ---------------------------------------------------------------------- new2linux · 2014-10-15 05:01 I am trying to capture the results of the lsof command from Free Pascal. In the terminal things work (although it takes several seconds to complete). The command is simply:    lsof -n When I execute the command using TProcess (and pipe the results back to Free Pascal), the command does NOT return (I awaited say 10 minutes)! How to debug (or solve) this? Note that I did set the current working directory explicitly before using TProcess. Without this even the " ls"  command would not work. My code: function ShellExec(acommandline: string): TStringList; begin   result := TStringList.Create;   with TProcess.Create(nil) do try     try       // see: http://wiki.lazarus.freepascal.org/Executing_External_Programs       CommandLine := acommandline;  // deprecated but ok       // wait until command done, record output       Options := Options + [poWaitOnExit, poUsePipes];       Execute;       result.LoadFromStream(Output);     except       FreeAndNil(result);     end   finally     Free;   end;   // doesnt work? consider setting current working folder! end; SetCurrentDir('/home/new2linux/Desktop'); ShellExec('lsof -n') ---------------------------------------------------------------------- new2linux · 2014-10-15 05:15 Continueing my efforts I piped the output to a file:   lsof -n > "/home/new2linux/Desktop/serial.xdc" Within the shell this command works and creates a file of 3.2MB, that's right: MEGABYTE. Within Free Pascal this same command fails: 'lsof: WARNING: can''t stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs'#10'      Output information may be incomplete.'#10'lsof: status error on >: No such file or directory'#10 Incomplete? The file given wasn't even touch'd. ---------------------------------------------------------------------- Leledumbo · 2014-10-15 08:32 Two things you skipped from the wiki: http://wiki.lazarus.freepascal.org/Executing_External_Programs#Reading _large_output This covers your "does not return" problem due to I/O deadlockhttp://w iki.lazarus.freepascal.org/Executing_External_Programs#Replacing_shell _operators_like_.22.7C_.3C_.3E.22 This covers your second trial to redirect from within CommandLine, which won't work ---------------------------------------------------------------------- new2linux · 2014-10-15 13:16 Quote from: Leledumbo on October 15, 2014, 10:32:32 amTwo things you skipped from the wiki: This covers your "does not return" problem due to I/O deadlock This covers your second trial to redirect from within CommandLine, which won't work Thanks Leledumbo for taking your time. This resolved my problems. Know what? The docs now include " This example is kept simple so you can learn from it. Please don't use this example in production code, but use the code in #Reading large output". I don't remember reading that a couple of months ago --opinion-- The problem I am solving is basic stuff, I suggest that Free Pascal include a couple of functions instead of merely listing (unfinished) code. This functionality is crucial on linux, and *should* work out of the box. ---------------------------------------------------------------------- BigChimp · 2014-10-15 13:21 Please feel free to submit a bug tracker item with a patch for FreePascal... (and note that at least fpc trunk has convenience functions in the TProcess unit for running commands) Note: what you call unfinished code is actually an attempt to progress from an easy example to a difficult example so people can understand the concepts and adapt it to their needs. That said, if you feel like improving the wiki, you're more than welcome! PS: Why is this functionality critical on Linux and not on e.g. Windows or OSX? ---------------------------------------------------------------------- marcov · 2014-10-15 13:39 Quote from: new2linux on October 15, 2014, 03:16:27 pm--opinion-- The problem I am solving is basic stuff, I suggest that Free Pascal include a couple of functions instead of merely listing (unfinished) code. Read the docs again: http://wiki.lazarus.freepascal.org/Executing_Exte rnal_Programs#.28Process..29RunCommand ---------------------------------------------------------------------- Leledumbo · 2014-10-15 13:43 Quote from: new2linux on October 15, 2014, 03:16:27 pmThe problem I am solving is basic stuff, I suggest that Free Pascal include a couple of functions instead of merely listing (unfinished) code. This functionality is crucial on linux, and *should* work out of the box. Which functionality are you referring to? Reading large output or shell operators? For reading large output, FYI, FPC trunk provides wrapper functions for reading large output. TProcess is available for those who want to have full control, not simple one. The reason is because there are (perhaps) unlimited ways to handle large output. Not everyone can use in-memory buffer that gets larger as more and more output comes. This will break in a very low RAM system (e.g. embedded). OTOH, systems with big RAM might want that to achieve speed. For the shell operators, however, I disagree. It will complicate TProcess for things it shouldn't handle. Also, different shells provide different features and syntax, which one should be followed? With that fact, I think delegating to real shell is the best. ---------------------------------------------------------------------- marcov · 2014-10-15 13:48 The shell issues and variation is the reason why commandline got deprecated in the first place. ---------------------------------------------------------------------- new2linux · 2014-10-15 14:50 Quote from: marcov on October 15, 2014, 03:39:34 pmRead the docs again: http://wiki.lazarus.freepascal.org/Executing_External_Programs# .28Process..29RunCommand Quote from: BigChimp on October 15, 2014, 03:21:20 pmfpc trunk has convenience functions in the TProcess unit for running commands PS: Why is this functionality critical on Linux and not on e.g. Windows or OSX? I looked at this page about 5 months ago. Knowing about the RunCommandXxx functions would indeed have saved me much grief. The docs now strike me as excellent, but note this single page is the size of a wiki-book: I measured 16 printed pages. Much too long when all one is seeking is the correct command (i.e. a ShellExecute replacement). Ideally when I type ShellExecute in Google the manual page would come up top, and it would have a pointer to the Linux or multi-platform alternative: RunCommand(). All one gets is tuts and confusion, NOT the information needed. Google doesn't even list a page from www.freepascal.org/docs-html/, apparently ShellExecute isn't there. I do find Shell, FpExecve and fpSystem, which are useless to get info back. This is critical on Linux? The Linux community sends people to the commandline (hell) for solutions, while Windows sends people to the API. If you could point me to an API on Linux for running say lsof or strace functionality, I would be much pleased, but of course there isn't any: for utilities the commandline *is* the API. Summary: there's excellent info, but also much confusion surrounding this topic that needs to be wiped (at least from Google). How to do that, I'm not sure, but I think it means a page listing use-cases and their typical solutions, and some refactoring from there. ---------------------------------------------------------------------- new2linux · 2014-10-15 14:53 And once again: thanks everybody! For I leave convinced the Quality is there. I just did't find it. ---------------------------------------------------------------------- Leledumbo · 2014-10-16 03:05 Quote from: new2linux on October 15, 2014, 04:50:52 pmIdeally when I type ShellExecute in Google the manual page would come up top, and it would have a pointer to the Linux or multi-platform alternative: RunCommand(). All one gets is tuts and confusion, NOT the information needed.Unfortunately, google results are not the same from person to person. It's personalized based on your query habit. The wiki should already have SEO enabled, but google personalized search could mess everything. Quote from: new2linux on October 15, 2014, 04:50:52 pmGoogle doesn't even list a page from www.freepascal.org/docs-html/, apparently ShellExecute isn't there. I do find Shell, FpExecve and fpSystem, which are useless to get info back.Because it's simply not there. Windows unit is not and is unlikely will ever be covered in FPC docs. MSDN already covers that and it's a HUGE library. We even haven't documented all of our units (those jewels that many people don't know and make them reinvent the wheel), how could we document vendor specific unit? ---------------------------------------------------------------------- BigChimp · 2014-10-16 06:06 Quote from: new2linux on October 15, 2014, 04:50:52 pmThis is critical on Linux? The Linux community sends people to the commandline (hell) for solutions, while Windows sends people to the API. If you could point me to an API on Linux for running say lsof or strace functionality, I would be much pleased, but of course there isn't any: for utilities the commandline *is* the API. lsof or strace obviously communicate through  an  API unless  they implement  all  functionality by themselves.... and they don't. They use kernel-provided facilities to list processes, trace through exececution etc. AFAIU, the API is even nicely documented,  see  e.g. the man pages (there's a separate section for the API, and a separate section for programs that use the API). Not a Linux expert but you can look at how the Unix/Linux  units in fpc communicate with the Linux APIs to get usernames etc Alternatively, you can read/write to the /proc filesystem which offers a lot of functionality. ---------------------------------------------------------------------- marcov · 2014-10-16 07:30 Quote from: new2linux on October 15, 2014, 04:50:52 pm Google doesn't even list a page from www.freepascal.org/docs-html/, apparently ShellExecute isn't there. Shellexecute is not a Free Pascal function. It is windows API, and we don't duplicate the whole of MSDN. (though some people use an old Borland helpfile with pascalized win32 help, but that is only for an old core set) Quote I do find Shell, FpExecve and fpSystem, which are useless to get info back. So afaik is shellexecute? Anyway, (fp)system() is the shellexecute equivalent (sans the document opening), and would work with your redirect version. QuoteThis is critical on Linux? The Linux community sends people to the commandline (hell) for solutions, while Windows sends people to the API. If you could point me to an API on Linux for running say lsof or strace functionality, I would be much pleased, but of course there isn't any: for utilities the commandline *is* the API. Depends. Sometimes the actual part that does the work of such utils is very small (compared to the part that does the cmdline and help). QuoteSummary: there's excellent info, but also much confusion surrounding this topic that needs to be wiped (at least from Google). How to do that, I'm not sure, but I think it means a page listing use- cases and their typical solutions, and some refactoring from there. While I don't agree entirely that it is really flawed, I do think the wikipage could be better sorted, going from easy to complicated instead of the other way around, and maybe emphasize the stock solutions more. When the run* paragraph was added, many linux distros came with FPC's that didn't have it. That should now be fixed hopefully for all but the oldest and most conservative distributions, so maybe it is time to update that, and accept it as "always there" functionality.