Pascal version of the UCI Filter program lazarus / 67-Windows ---------------------------------------------------------------------- Roland57 · 2024-10-11 08:33 Hello everybody. I translated to Pascal the program UCI Filter. It's a program designed to be used as a filter between a UCI chess engine and a chess GUI. The Pascal version doesn't work correctly. Would someone help me to find the error? Here is how to compile and test the original program: (You need to have make and fpc available on the command line.) D:\c\echecs\ucifilter-pascal-241011>make gcc -Wall ucifilter.c -o ucifilter.exe D:\c\echecs\ucifilter-pascal-241011>make test ucifilter .\CT800_V1.34_x32.exe CT800 V1.34 32 bit UCI version 2016-2019 by Rasmus Althoff Free software under GPLv3+ quit Here is how to compile and test the Pascal version: D:\c\echecs\ucifilter-pascal-241011>make ucifilterpas fpc -Mobjfpc -Sh -dDEBUG ucifilter.pas -oucifilterpas.exe Compiling Debug Version Free Pascal Compiler version 3.2.0 [2021/02/21] for i386 Copyright (c) 1993-2020 by Florian Klaempfl and others Target OS: Win32 for i386 Compiling ucifilter.pas Compiling log.pas Linking ucifilterpas.exe 524 lines compiled, 0.3 sec, 170256 bytes code, 6724 bytes data D:\c\echecs\ucifilter-pascal-241011>make testpas ucifilterpas .\CT800_V1.34_x32.exe | a♠CT800 V1.34 32☻  | a♠►' £ a♠2016-2019 by R☻  | a♠╣ a♠Free software ☻  | aquit You can see that the output of the UCI engine is corrupted. Which error did I make? ---------------------------------------------------------------------- bobby100 · 2024-10-11 13:07 Are you sure about using ansichar and ansistring? To me, it looks like you are getting multibyte text, and the garbage comes from reading it as singlebyte (ansi) chars. ---------------------------------------------------------------------- af0815 · 2024-10-11 13:17 Best way, write the original raw stringdata to a file and analyse this file. I have see this, if i read output from a Delphi programm (2 Byte Chars) and misinterpret this in my first try. ---------------------------------------------------------------------- Roland57 · 2024-10-16 13:18 Quote from: bobby100 on October 11, 2024, 03:07:54 pmAre you sure about using ansichar and ansistring? I think ansichar is the equivalent of the char type used in the C program. The ansistring type is used only for the executable path, and it seems to work (since the executable starts). Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function EngineStarten(EnginePath: AnsiString): THandle; forward; ---------------------------------------------------------------------- Fred vS · 2024-10-16 20:59 Quote from: Roland57 on October 16, 2024, 03:18:52 pmI think ansichar is the equivalent of the char type used in the C program. ; There is also cchar from ctypes.pp. ---------------------------------------------------------------------- Thaddy · 2024-10-17 04:47 The equivalent of a C char is usually a byte, not a Pascal char. ---------------------------------------------------------------------- TRon · 2024-10-17 05:19 A byte is equivalent to a unsigned char. ---------------------------------------------------------------------- Thaddy · 2024-10-17 05:54 That is compiler dependant. You can only assume that the size is sizeof(byte). See remark in ctypes. ---------------------------------------------------------------------- Roland57 · 2024-10-17 09:02 Thank you all for your answers. I wonder if my buffers are properly initialized, here : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure GuiNachEngine(pFilter: PFilterDaten);// ...begin  // ...    lp.hDatei := pFilter^.hVonGui;  lp.Groesse := 10000;  lp.pPuffer := StrAlloc(lp.Groesse + 1);    //FillChar(lp.pPuffer, lp.Groesse * SizeOf(ansichar), #0);  StrPCopy(lp.pPuffer, StringOfChar(#0, lp.Groesse)); And here : Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function EngineNachGui(pParam: Pointer): DWORD; stdcall;// ...begin  // ...    pFilter := PFilterDaten(pParam);    lp.hDatei := pFilter^.hVonEngine;  lp.Groesse := 10000;  lp.pPuffer := StrAlloc(lp.Groesse + 1);    //FillChar(lp.pPuffer, lp.Groesse * SizeOf(ansichar), #0);  StrPCopy(lp.pPuffer, StringOfChar(#0, lp.Groesse)); ---------------------------------------------------------------------- TRon · 2024-10-17 10:07 Is the text output from the server still garbled ? or does the engine not respond ? What platform is the engine running on, and what platform the client ? (the UCI documentation mentions some specifics) ---------------------------------------------------------------------- Thaddy · 2024-10-17 10:24 Sigh, use AllocMem when translatinng C code. ---------------------------------------------------------------------- Roland57 · 2024-10-17 13:11 Quote from: TRon on October 17, 2024, 12:07:27 pmIs the text output from the server still garbled ? or does the engine not respond ? What platform is the engine running on, and what platform the client ? The engine responds but the output of the filter is corrupted. The filter uses Windows API. Please see my first post. Quote from: Thaddy on October 17, 2024, 12:24:50 pmSigh, use AllocMem when translating C code. Thanks, I will try. ---------------------------------------------------------------------- TRon · 2024-10-17 13:16 Quote from: Roland57 on October 17, 2024, 03:11:24 pmThe engine responds but the output of the filter is corrupted. The filter uses Windows API. Please see my first post. "\n" means #13#10 on/for windows (as expressed in the UCI specifications) ---------------------------------------------------------------------- Roland57 · 2024-10-17 13:33 AllocMem works, but doesn't solve the issue. @TRon Thanks, but I don't think that this be the issue. ---------------------------------------------------------------------- TRon · 2024-10-17 13:41 @Roland: Then don't be surprised if the output isn't as expected. The c-code seems to follow the convention, the translation doesn't. There is very little specifications written down, so I would personally stick to what actually is available. ---------------------------------------------------------------------- Roland57 · 2024-10-17 13:43 Quote from: TRon on October 17, 2024, 03:41:48 pmThen don't be surprised if the output isn't as expected. The c-code seems to follow the convention, the translation doesn't. OK, I will take a closer look. ---------------------------------------------------------------------- Roland57 · 2024-10-17 14:19 @TRon It is better now. I have three lines, as with the C program. (Previously I had only one.) So, you were right. But there are still unwanted characters. Maybe I did something wrong. I attach the latest version of ucifilter.pas. ---------------------------------------------------------------------- Fred vS · 2024-10-17 14:22 Hello Roland! I try your project here on Linux via wine (dont have a Win machine disposable at the moment). Running this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ~$ wine CT800_V1.34_x32.exegives that: QuoteCT800 V1.34 32 bit UCI version 2016-2019 by Rasmus Althoff Free software under GPLv3+ And running that: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ~$ wine ucifilter.exe CT800_V1.34_x32.exegives this: Quote$_CT800 V1.34 32 î$_'D_2016-2019 by Raî$_a_Free software uî$_ So yes there is a problem. I would try to do it isolating the thing: create a simple test prog that only runs CT800_V1.34_x32.exe via a TProcess + pipe output, taking inspiration from the wiki example: https://wiki.lazarus.freepascal.org/Executing_External_Programs#TProce ss And maybe, do a try with TProcessUTF8 (I think it is possible to use it for console app) QuoteTProcessUTF8 is part of Lazarus (introduced in the development version which will end up as Lazarus 1.4 in future) in the UTF8Process unit; it can be used from command-line programs by setting a requirement to LCLBase [EDIT] With your last change there is indeed 3 lines: Quote╕_CT800 V1.34 32 î╕_'╫_ 2016-2019 by Rî╕_⌠_ Free software î╕_ ---------------------------------------------------------------------- TRon · 2024-10-17 15:13 Quote from: Roland57 on October 17, 2024, 04:19:55 pmIt is better now. I have three lines, as with the C program. (Previously I had only one.) So, you were right. Thank you for reporting back and that is at least some progress  QuoteBut there are still unwanted characters. Maybe I did something wrong. Don't forget options writings when command "uciok" is received. There might be other subtle differences but then I would have to take a closer look (which might take some time, that is if I'm even able to detect it at all) ---------------------------------------------------------------------- Roland57 · 2024-10-17 15:33 Quote from: Fred vS on October 17, 2024, 04:22:38 pmI would try to do it isolating the thing: create a simple test prog that only runs CT800_V1.34_x32.exe via a TProcess + pipe output, taking inspiration from the wiki example: https://wiki.lazarus.freepascal.org/Executing_External_Programs#TProce ss Hello Fred! Thank you for testing. Yes, I could use TProcess. I know that it will work, because I already did it. But my project here is to translate the C program, as it is (using Windows API). But, following you idea, maybe I could add more details in log files, to see where things go wrong. For now the log files of the C program and of the Pascal program have no differences. I would have hoped that someone would see immediately which error I made. Quote from: TRon on October 17, 2024, 05:13:33 pmDon't forget options writings when command "uciok" is received. OK, I will review the whole thing tomorrow. ---------------------------------------------------------------------- Fred vS · 2024-10-17 15:59 Hello Roland. I did try too with CreateProcessW()  (with the "W" at end for widestring) but same result. Indeed in your project eschecs, using TProcess works perfectly.  I never used CreateProcess() that is Windows only. ---------------------------------------------------------------------- TRon · 2024-10-17 19:42 @Roland57: Feel free to correct me when wrong but Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   str[len] = '\n'; Literally means Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   str_ := str_ + #13#10But because str being a pchar Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     str_[len] := #13;    str_[len+1] := #10; I am aware of the len+1 in the functions that are invoked after the addition in str so it might very well be enough to only use Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    str_[len] := #13; Perhaps worth a bit of testing/experimenting ? ---------------------------------------------------------------------- Roland57 · 2024-10-18 13:45 @TRon Thank you for the code. I made the modifications that you suggested. But the unwanted characters are always here. ---------------------------------------------------------------------- Fred vS · 2024-10-18 14:32 Hello Roland! I did try to use, before CreateProcess(): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SetConsoleOutputCP(CP_UTF16);or Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SetConsoleOutputCP(CP_UTF8);or Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SetConsoleOutputCP(437);or Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i SetConsoleOutputCP(CP_NONE); Each time there are other chars but not the wanted... ---------------------------------------------------------------------- tetrastes · 2024-10-18 20:48 The problem is in WriteFile. You must dereference str_: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i WriteFile(pFilter^.hNachGui, str_^, len + 1, AnzGeschrieben, nil) Then the first variant works like C code, the second has some regression. ---------------------------------------------------------------------- TRon · 2024-10-19 00:03 Quote from: tetrastes on October 18, 2024, 10:48:39 pmThe problem is in WriteFile. You must dereference str_: Not according to the windows defined API. If it is (for which I have no reason to doubt) then someone messed up the header translation (again) and reason I did not flagged it. The jedi units are far more superior in that regards. ---------------------------------------------------------------------- Fred vS · 2024-10-19 00:41 Quote from: TRon on October 19, 2024, 02:03:19 amQuote from: tetrastes on October 18, 2024, 10:48:39 pmThe problem is in WriteFile. You must dereference str_: Not according to the windows defined API. If it is (for which I have no reason to doubt) then someone messed up the header translation (again) and reason I did not flagged it. The jedi units are far more superior in that regards. Yes, it is possible, but I may be wrong (I hope it), it seems to me that fpc usage of the Windows CreateProcess() API does not handle Stdoutput as it should. PS: ---------------------------------------------------------------------- Thaddy · 2024-10-19 04:51 It is much easier to rip out all that windows specific stuff and simply use rtl equivalents for file handling and pipes. ---------------------------------------------------------------------- tetrastes · 2024-10-19 05:59 Quote from: TRon on October 19, 2024, 02:03:19 amQuote from: tetrastes on October 18, 2024, 10:48:39 pmThe problem is in WriteFile. You must dereference str_: Not according to the windows defined API. If it is (for which I have no reason to doubt) then someone messed up the header translation (again) and reason I did not flagged it. The jedi units are far more superior in that regards. There were holiwars about it, f.e. https://forum.lazarus.freepascal.or g/index.php/topic,59388.msg442698.html#msg442698 ---------------------------------------------------------------------- TRon · 2024-10-19 06:50 Quote from: tetrastes on October 19, 2024, 07:59:17 amThere were holiwars about it, f.e. https://forum.lazarus.freepascal.org/index.php /topic,59388.msg442698.html#msg442698 Thank you for that link tetrastes as I seem to have missed that discussion. Shame that I missed it because for sure I would have strongly advocated my opinion on that topic. Every excuse to deviate from an official API is simply wrong and I am positive that someone who believes otherwise has never had to work with these API's in practice (other than an occasional call here and there. For someone disagreeing with that try translating code with hundreds of API calls and try keep that point of view). I am already fed up with all these API parameters being var'ed while many API's explicitly state that the "values" of these (pointer) parameters can be nil (and actually makes a whole lot of difference in behaviour of such a function). Lesson learned to never ever use the windows unit (not that I did since my switch to linux but even there this kind of crap is creeping up more and more). And for the literate: yes, I am aware that the original official declaration also exist .... but whaddayouknow sometimes a function declaration follows the official API except for a parameter here and there being var'ed so you never know unless explicitly looking at the available translation(s) of such declaration(s). Complete randomness with a sauce of Delphi compatibility (and no excuse makes it smell less)  And now it makes also perfect sense why TS has these issues so thank you for that hint tetrastes :thumbsup: ---------------------------------------------------------------------- Roland57 · 2024-10-19 09:20 @tetrastes Great! Many thanks. I can't wait to test by myself. (Unfortunately I am at home today, where I have only a Linux installation.) ---------------------------------------------------------------------- TRon · 2024-10-19 09:27 Quote from: Roland57 on October 19, 2024, 11:20:55 am(Unfortunately I am at home today, where I have only a Linux installation.) Now, if you only had ....  BTW: in the case that was indeed the (only) issue then you might be able to get away with just using #10 but be advised that in case of issues the specification explicitly mentions to use #13#10 for windows. In practice most code simply ignores the #13 and 'chops' on the #10 (but that could differ per engine). ---------------------------------------------------------------------- Roland57 · 2024-10-19 10:04 Quote from: TRon on October 19, 2024, 11:27:33 amNow, if you only had ....  I don't understand. Sorry, english is not my native language.  Quote from: TRon on October 19, 2024, 11:27:33 amBTW: in the case that was indeed the (only) issue then you might be able to get away with just using #10 but be advised that in case of issues the specification explicitly mentions to use #13#10 for windows. In practice most code simply ignores the #13 and 'chops' on the #10 (but that could differ per engine). Thanks for the clarification. ---------------------------------------------------------------------- TRon · 2024-10-19 10:14 Quote from: Roland57 on October 19, 2024, 12:04:24 pmI don't understand. Sorry, english is not my native language.  I ruins the (bad) joke but... The full sentence would have been: "Now, if you only had used Free Pascal's RTL functions for the file- and pipe- handling, then you would have been able to compile/run it on/for Linux" (and yes I understand that defeats the purpose of your translation of this program, which was the second layer of the joke). ---------------------------------------------------------------------- Roland57 · 2024-10-19 10:38 I see.  Maybe when I have time, I will try to make a multiplatform version. By the way, I already made something similar, using TProcess. When you say RTL, do you mean the Process unit, or something else? ---------------------------------------------------------------------- TRon · 2024-10-19 10:45 Quote from: Roland57 on October 19, 2024, 12:38:01 pmMaybe when I have time, I will try to make a multiplatform version. That was not the intention of the "joke" but could perhaps be a nice exercise. QuoteBy the way, I already made something similar, using TProcess. Ah yes, I see. Not the exact same functionality as the filter program but the base functionality seems to be in place. QuoteWhen you say RTL, do you mean the Process unit, or something else? Indeed TProcess but perhaps also in combination with the pipes unit. ---------------------------------------------------------------------- Thaddy · 2024-10-19 12:35 The last sentence is exactly what I meant. ---------------------------------------------------------------------- Fred vS · 2024-10-19 14:54 Hello. The boring one is back!  Doing this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program launchprogram; uses   Process; var   AProcess: TProcess; begin   AProcess := TProcess.Create(nil);   AProcess.Executable:= 'CT800_V1.34_x32.exe';   AProcess.Options := AProcess.Options + [poWaitOnExit];   AProcess.Execute;   AProcess.Free;   end. Gives this: Quote$ wine launchprogram.exe CT800 V1.34 32 bit UCI version 2016-2019 by Rasmus Althoff Free software under GPLv3+ As you can see, using TProcess gives a good result in the standard output. So I still have doubts about how fpc handles the standard output while using the Windows API CreateProcess() and related methods. ---------------------------------------------------------------------- Thaddy · 2024-10-19 16:28 Fred, your only and biggest problem is that in a Lazarus GUI app, stdin/out/err are not available by default on Windows. That is all. This is also true for Delphi. The easiest way to get around it is specifying {$apptype console} and hide the console. I have shown quite a few times how to do that. On almost all other platforms, there is always a working set of stdin/out/err by default. ---------------------------------------------------------------------- Fred vS · 2024-10-19 16:45 Quote from: Thaddy on October 19, 2024, 06:28:27 pmFred, your only and biggest problem is that in a Lazarus GUI app, stdin/out/err are not available by default on Windows. That is all. Hello Thaddy. I hope this is my only problem! But it shouldn't be this time, in Roland's demo I added right at the beginning of testing: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$APPTYPE CONSOLE} Also added this to be sure (but it is done by default): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  StartInfo.hStdInput := GetStdHandle(STD_INPUT_HANDLE);  StartInfo.hStdoutput := GetStdHandle(STD_OutPUT_HANDLE); And the StartInfo.hStdoutput has not same result than the STD_OutPUT from TProcess. Sadly, I dont have Delphi installed to test the demo of Roland compiled with Delphi vs with fpc. ---------------------------------------------------------------------- tetrastes · 2024-10-19 19:42 Quote from: Fred vS on October 19, 2024, 04:54:41 pmAs you can see, using TProcess gives a good result in the standard output. So I still have doubts about how fpc handles the standard output while using the Windows API CreateProcess() and related methods. What do you mean? At windows TProcess uses CreateProcess (CreateProcessW to be precise). So "how fpc handles ... etc" you can see in sources of TProcess (packages\fcl-process\src\win\process.inc). How the particular programmer handles this, is the matter of this programmer. ---------------------------------------------------------------------- Fred vS · 2024-10-19 23:39 Quote from: tetrastes on October 19, 2024, 09:42:28 pmQuote from: Fred vS on October 19, 2024, 04:54:41 pmAs you can see, using TProcess gives a good result in the standard output. So I still have doubts about how fpc handles the standard output while using the Windows API CreateProcess() and related methods. What do you mean? At windows TProcess uses CreateProcess (CreateProcessW to be precise). So "how fpc handles ... etc" you can see in sources of TProcess (packages\fcl-process\src\win\process.inc). How the particular programmer handles this, is the matter of this programmer. Hello Tetrastes. Many thanks for pointing packages\fcl-process\src\win\process.inc But there is this as note line 280: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i // Beware: CreateProcess can alter the strings So I will study the code using CreateProcessW. [EDIT] I did try using CreateProcessW, with the vars adapted for widestring and following the code of process.inc but there is still garbage in the output strings.  So there is something I don't catch. I pass my turn. ---------------------------------------------------------------------- Roland57 · 2024-11-06 12:03 Quote from: tetrastes on October 18, 2024, 10:48:39 pmThe problem is in WriteFile. You must dereference str_: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i WriteFile(pFilter^.hNachGui, str_^, len + 1, AnzGeschrieben, nil) Then the first variant works like C code, the second has some regression. I finally could test that modification and indeed it solves the issue. So thank you very much!  I have to do a last review of the whole code, for proper line endings management, as suggested by TRon. ---------------------------------------------------------------------- Roland57 · 2024-11-08 07:25 I fixed a last (?) error. This line was not properly translated: Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i sscanf(str, "%15s", Kommando); My first translation was: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     StrLCopy(Kommando, str_, SizeOf(Kommando) - 1); Which is obviously wrong. Now it is: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     FillChar(Kommando, SizeOf(Kommando), #0);    i := 0;    while (i < len) and (i < SizeOf(Kommando)) and ((str_ + i)^ <> #32) do    begin    Kommando[i] := (str_ + i)^;      Inc(i);    end; I know there is a Pascal sscanf function, but I didn't find how to use it in that case.  @TRon After reflection, '\n' (please notice the single quotes) is the equivalent of #10, not #13#10. So, it would probably be better to use #13#10 as you suggested, but as my first goal was to translate the C program as it is, I went back to my first translation. Here is the final (?) state of the program, if ever someone is interested. Thank you all for your help!