TMemoryStreamUTF8 cannot LoadFromFile lazarus / 21-General ---------------------------------------------------------------------- martinrame · 2022-06-13 21:02 Hi, I need to load a file whose name contains accented characters, such as: "/mnt/Informes/202205/PEÑALOZA JORGE.pdf" The fpc program is a CGI, which runs in a FreeBSD system. The files reside in a Windows server mounted on /mnt/Informes on the FreeBSD system. The filename is returned from a database query, and I store it in a string, such as: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i lFile := lQuery.FieldByName('filename').AsString; Then, I check if the file exists: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i if FileExists(lFile) thenbegin...end; If the file exists I do this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i lPdfStream := TMemoryStreamUTF8.Create;lPdfStream.LoadFromFile(lFile);...  The file exists, but when I call LoadFromFile I get "Unable to open file ...". What can I do to open it? ---------------------------------------------------------------------- martinrame · 2022-06-13 21:15 I also tried with a TFileStream (and TFileStreamUTF8), with the same results. ---------------------------------------------------------------------- martinrame · 2022-06-13 22:39 I created a simple program to traverse all files in the directory and got the names with "ñ" where replaced with "??": Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure AddAllFilesInDir(const Dir: string);var  SR: TSearchRec;begin  if FindFirst(IncludeTrailingBackslash(Dir) + '*.*', faAnyFile or faDirectory, SR) = 0 then    try      repeat        if (SR.Attr and faDirectory) = 0 then          ListBox1.Items.Add(SR.Name)        else if (SR.Name <> '.') and (SR.Name <> '..') then          AddAllFilesInDir(IncludeTrailingBackslash(Dir) + SR.Name);  // recursive call!      until FindNext(Sr) <> 0;    finally      FindClose(SR);    end;end; This is the result: Quote.. PERALTA ANDRES.pdf PE??ALOZA JORGE.pdf .. ---------------------------------------------------------------------- Gustavo 'Gus' Carreno · 2022-06-14 00:07 Hey Martin, Let me start by saying that this is not an answer to your problem, it's more of pointing you in a direction so that you can solve your problem. With that said, if you look at this page: FindFirst, you'll find that there are 2 overloads for the same function. One uses a UnicodeString and the other uses a RawByteString. And both return a different type of SearchRec. Why can't I help you more? Well, that's because I've never dug deep into the magic that is the transformation the type String gets involved in when it's passed around with UnicodeString, WideString, UTF8String and all the others that I'm not aware of. Because of that the issue could be anywhere since you're passing around a string with UTF content, but some of the LCL objects don't really know how to deal with that. And good luck trying to identify what type of string it is. And how or where it gets mangled. And if you're planing on cross platform, if you get it running on Windows, then nothing guarantees you it will work on Linux due mainly to the OS, not the FCL or LCL. So yeah, while it's a pain, and believe me, I'm Portuguese and we have to deal with the same issues, it's something that you'll have to dig around and see what type of string is being returned an then do stuff accordingly. Unfortunately, you'll have to wait for the people that have more experience with UTF containing strings, cuz I'm pretty crap at it myself, and for that I apologise profusely. Cheers, Gus ---------------------------------------------------------------------- PascalDragon · 2022-06-14 06:57 Quote from: martinrame on June 13, 2022, 11:02:05 pmThe fpc program is a CGI, which runs in a FreeBSD system. Make sure that your main program uses either cwstring or fpwidestring. If that's not enough please provide the output of the locale (at least I hope that also exists on FreeBSD). ---------------------------------------------------------------------- marcov · 2022-06-14 09:03 Afaik on all *nix, the apis don't transcode filenames, you get them binary as written in the filesystem directory structure. But if you then have filesystems with differing encodings (e.g. freebsd native utf8, and Windows some ansi encoding), you have a problem. The best bet is to look into mount options for the said SMB share, and see if you can fix the encoding the FS driver gives to the kernel on that end. ---------------------------------------------------------------------- martinrame · 2022-06-14 10:36 Hi, the locale is this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LANG=en_US. UTF-8LC_CTYPE="en_US.UTF-8"LC_COLLATE="en_US.UTF-8"LC_TIME="en_US.UTF- 8"LC_NUMERIC="en_US.UTF-8"LC_MONETARY="en_US.UTF- 8"LC_MESSAGES="en_US.UTF-8"LC_ALL= The smb filesystem is mounted specifying this encodings (in /etc/nsmb.conf): Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i charsets=utf-8:iso-8859-1 That means, the local filesystem is UTF8, and the remote (Windows) is ISO-8850-1. ---------------------------------------------------------------------- martinrame · 2022-06-14 10:38 Quote from: PascalDragon on June 14, 2022, 08:57:57 amQuote from: martinrame on June 13, 2022, 11:02:05 pmThe fpc program is a CGI, which runs in a FreeBSD system. Make sure that your main program uses either cwstring or fpwidestring. If that's not enough please provide the output of the locale (at least I hope that also exists on FreeBSD). Hi PascalDragon, how can I tell which type of strings?. ---------------------------------------------------------------------- PascalDragon · 2022-06-14 11:26 Quote from: martinrame on June 14, 2022, 12:38:00 pmQuote from: PascalDragon on June 14, 2022, 08:57:57 amQuote from: martinrame on June 13, 2022, 11:02:05 pmThe fpc program is a CGI, which runs in a FreeBSD system. Make sure that your main program uses either cwstring or fpwidestring. If that's not enough please provide the output of the locale (at least I hope that also exists on FreeBSD). Hi PascalDragon, how can I tell which type of strings?. This is not about types of strings. You need to put either cwstring or fpwidestring into the uses-clause of your main program (first place; second or third if you also use cthreads and/or cmem). The locale output looks good, FPC should detect that as UTF-8 then (as long as you use either cwstring or fpwidestring). ---------------------------------------------------------------------- martinrame · 2022-06-14 11:35 Thanks PascalDragon, but unfortunately using Quotecwstring or Quotefpwidestring doesn't change the result. ---------------------------------------------------------------------- martinrame · 2022-06-14 12:12 I did an: ls /directory/file|hexdump -C to see the actual ascii codes of the filename and found the Ñ is replaced to the hex a5, then I did this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i lFile := AnsiReplaceStr(lFile, 'Ñ', chr($A5));                                      lPdfStream.LoadFromFile(lFile); But still I can't open the file with LoadFromFile. ---------------------------------------------------------------------- martinrame · 2022-06-14 13:51 Traversing char by char, the string returned by the database I'm getting this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i P=80E=69=195=145A=65L=76O=79Z=90A=65  Compared to the output of hexdump I get: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ls /mnt/Informes/202205/PE*19900.pdf|hexdump -C -s 36 -n 800000024  2f 50 45 a5 41 4c 4f 5a                           |/PE.ALOZ|0000002c  The difference is the file name contains a5 (Ascii 165) in the place of the Ñ, while the string returned by the database contains ASCII 195 followed by 145. It looks like the database is effectively using UTF8, while the file naming is ASCII. Is there a way to convert between those incodings? ---------------------------------------------------------------------- martinrame · 2022-06-14 14:09 I replaced that with: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i lFile := StringReplace(lFile, #195#145, #165, [rfReplaceAll]); But still getting the "Unable to open file...No such file or directory". ---------------------------------------------------------------------- martinrame · 2022-06-14 14:28 The funny thing is that FileExists returns true when I check both the UTF8 file name and the one whith replaced #195#145 with #165. But LoadFromFile raises the "Unable to open file ... No such file or directory" error. ---------------------------------------------------------------------- martinrame · 2022-06-16 00:54 I still can't find a solution, but I'm still trying. I made this simple recursive file search, for finding part of the file. I know all files end with a number followed by .pdf, so I call: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i FindFile('/path/to/file/', '1234.pdf'); That finds the first file that ends in '1234.pdf' and I try to open using LoadFromFile, but I still get the famous "Unable to open file..." ONLY ON FILES WITH SPECIAL CHARACTERS, like Ñ. Why I can't open the file, when FindFirst/FindNext finds it?. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function TTest.FindFile(const Dir: string; AMatch: string): string;               var                                       SR: TUnicodeSearchRec;                                                                       lFiles: string;                          begin                                                                                           if FindFirst(IncludeTrailingBackslash(Dir) + '*.pdf', faAnyFile or faDirectory, SR) = 0 then        try                                                                                               repeat                                                                                          if (SR.Attr and faDirectory) = 0 then                                                             if Pos(AMatch, SR.Name) > 0 then          begin                                                                                             lFiles := SR.Name;                                                                              with TStringList.Create do               begin                                                                                            LoadFromFile(Dir + SR.Name); // <-- Unable to open file....            end;                                                                                            break;                                                                                      end                                                                                           else if (SR.Name <> '.') and (SR.Name <> '..') then          FindFile(IncludeTrailingBackslash(Dir) + SR.Name, AMatch);  // recursive call!      until FindNext(Sr) <> 0;                                                                        Result := lFiles;    finally      FindClose(SR);    end;end; ---------------------------------------------------------------------- martinrame · 2022-06-16 01:04 A similar program in Python is this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i import os # folder pathdir_path = r'/path/to/file/' # Iterate directoryfor path in os.listdir(dir_path):    # check if current path is a file    if os.path.isfile(os.path.join(dir_path, path)):        if path.endswith("19900.pdf"):            f = open(dir_path + path, "rb")            content = f.read()            print(content) And it can open the file without issues. ---------------------------------------------------------------------- PascalDragon · 2022-06-16 09:15 Quote from: martinrame on June 16, 2022, 02:54:34 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i           with TStringList.Create do               begin                                                                                              LoadFromFile(Dir + SR.Name); // <-- Unable to open file....          end;  Sidenote: this will lead to a memory leak. ---------------------------------------------------------------------- SymbolicFrank · 2022-06-17 07:10 Quote from: martinrame on June 14, 2022, 02:12:45 pmI did an: ls /directory/file|hexdump -C to see the actual ascii codes of the filename and found the Ñ is replaced to the hex a5, then I did this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i lFile := AnsiReplaceStr(lFile, 'Ñ', chr($A5));                                      lPdfStream.LoadFromFile(lFile); But still I can't open the file with LoadFromFile. The ASCII code of 'Ñ' in ISO-8850-1 is D1, not A5. So, if your settings are as you said, they translate the wrong code page. Find out what code page is used and translate it yourself. There's almost certainly a Free Pascal function somewhere to do that for you. ---------------------------------------------------------------------- Josh · 2022-06-17 08:02 does SysToUTF8 function in lazutf8 unit solve the problem? Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i LoadFromFile(SysToUTF8(Dir + SR.Name)); ---------------------------------------------------------------------- tetrastes · 2022-06-17 08:14 Quote from: SymbolicFrank on June 17, 2022, 09:10:16 amThe ASCII code of 'Ñ' in ISO-8850-1 is D1, not A5. So, if your settings are as you said, they translate the wrong code page. Code for Ñ is $A5 in CP437 and CP850, which are used for Windows console in USA or Western Europe. As I understand, Windows uses console CP for network mounts. ---------------------------------------------------------------------- Jorg3000 · 2022-06-17 09:46 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var lFile: RawByteString;...lFile:= ......SetCodepage(lFile,Windows.GetConsoleCP, false);lFile:=Utf8Encode(lFile); ---------------------------------------------------------------------- PascalDragon · 2022-06-17 11:27 Quote from: Jorg3000 on June 17, 2022, 11:46:31 amCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var lFile: RawByteString;...lFile:= ......SetCodepage(lFile,Windows.GetConsoleCP, false);lFile:=Utf8Encode(lFile);  martinrame is on FreeBSD, so using the Windows unit is not an option. ---------------------------------------------------------------------- Jorg3000 · 2022-06-17 12:44 Quotemartinrame is on FreeBSD, so using the Windows unit is not an option. Oops, I confused that with another post that was explicitly about Windows. Sorry. PS: If the codepage is known, e.g. 437 or 850, my code snippet could be used for conversion. QuoteCode for Ñ is $A5 in CP437 and CP850 ---------------------------------------------------------------------- martinrame · 2022-06-23 19:16 Hi!, thanks for taking some time to study this, and sorry for not showing up (I got the flu and wasn't able to work). I created a small program that basically does the same as the CGI (read the file name from the database in UTF8, and open that file from the mounted filesystem)...and it worked!. So, now I'm starting to think the problem is related to the CGI and/or the Apache24 web server. ---------------------------------------------------------------------- martinrame · 2022-06-23 20:27 This is the console program that works: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test; {$mode objfpc}{$H+}uses  Classes, SysUtils, pqconnection, sqldb; var  lStream: TFileStream;  PQConnection2: TPQConnection;  SQLTransaction2: TSQLTransaction;  lQuery: TSqlQuery;begin  PQConnection2 := TPQConnection.Create(nil);  SQLTransaction2 := TSqlTransaction.Create(nil);  lQuery := TSqlQuery.Create(nil);  try  PQConnection2.HostName:= '192.168.0.130';    PQConnection2.DatabaseName:= 'app2';    PQConnection2.UserName:= '******;    PQConnection2.Password:= '******';    PQConnection2.Transaction:= SQLTransaction2;    PQConnection2.Connected:= True;    lQuery.Database := PQConnection2;  lQuery.SQL.Text := 'select ' +      'replace(' +      '   replace(' +      '      replace(' +      '         archivo,''\\'', ''\''),' +      '      ''SERVIDOR\Sistema\Proyecto'',''mnt/assist''),' +      '   ''\'',''/'') as archivo ' +      'from vista_pacs where idsld_pedidoestudio_det=:accession';    lQuery.ParamByName('accession').AsInteger := StrToInt(argv[1]);    lQuery.Open;    lStream := TFileStream.Create(lQuery.FieldByName('archivo').AsString, fmOpenRead);    Writeln(lStream.Size);    lStream.Free;  finally    lQuery.Free;    SQLTransaction2.Free;    PQConnection2.Free;  end;end. And this is the unit, part of the CGI (using Brook Framework), that doesn't work: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i unit acttest;                                                                                             {$mode objfpc}{$H+}                                                            interface                                                               uses                                                                                        BrookAction,                                                                              Classes,                     SysUtils,  sqldb,        dm;                                               type                                              TActPDFOnTheFlyTEST = class(TBrookAction)                                                   private  public    procedure Get; override;                                                                end; implementation procedure TActPDFOnTheFlyTEST.Get;var  lPdfStream: TFileStream;  lAccession: string;  lQuery: TSqlQuery;  lFile: string;begin  lQuery := TSQLQuery.Create(nil);  try    lAccession := Variable['accession'];  lQuery.DataBase := DataModule1.PQConnection2;    lQuery.SQL.Text := 'select ' +       'replace(' +       '   replace(' +       '      replace(' +       '         archivo,''\\'', ''\''),' +       '      ''SERVIDOR\Sistema\Proyecto'',''mnt/assist''),' +       '   ''\'',''/'') as archivo ' +       'from vista_pacs where idsld_pedidoestudio_det=:accession';    lQuery.ParamByName('accession').AsInteger := StrToInt(lAccession);    lQuery.Open;    lPdfStream := TFileStream.Create(lQuery.FieldByName('archivo').AsString, fmOpenRead);    raise exception.create('aaaa'); // this is never reached, because the previous line returns Unable to open file ... No such file or directory on files with Ñ.    //lPdfStream.LoadFromFile(lFile);    lPdfStream.Position := 0;    HttpResponse.ContentStream := lPdfStream;    HttpResponse.ContentType:='application/pdf';    HttpResponse.ContentLength:= lPdfStream.Size;    HttpResponse.SendContent;  finally    lQuery.Free;  end;end; initialization  TActPDFOnTheFlyTEST.Register('/pdfontheflytest/:accession'); end.  The partial conclusion is the CGI has some limitation to opening this the file, but it can open files without Ñ wihout issues. ---------------------------------------------------------------------- martinrame · 2022-06-23 22:20 I'm re-trying with cwstring, as suggested by PascalDragon many replies ago. And it seems to try opening the file, but the program hangs (100% cpu usage...) on this line: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i lPdfStream := TFileStream.Create(lQuery.FieldByName('archivo').AsString, fmOpenRead); ---------------------------------------------------------------------- martinrame · 2022-06-23 22:43 As the original file is in a SMB filesystem I copied it to my home directory and adapted the program to read from there, but now I get: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 500 - Internal server error @error What's that @error? ---------------------------------------------------------------------- martinrame · 2022-06-23 22:45 Is the default template for the Brook Framework's 500 error. ---------------------------------------------------------------------- Thaddy · 2022-06-24 08:45 TmemorystreamUTF8 is a nonsense. A stream is a stream and always byte sized. I would have fired the author. Simply because of proven lack of knowledge. ---------------------------------------------------------------------- bytebites · 2022-06-24 08:56 TmemorystreamUTF8... QuoteImplements a memory stream which supports file names with UTF-8 encoding ---------------------------------------------------------------------- BeniBela · 2022-06-24 09:29 Quote from: Thaddy on June 24, 2022, 10:45:09 amTmemorystreamUTF8 is a nonsense. A stream is a stream and always byte sized. I would have fired the author. Simply because of proven lack of knowledge. All the streams are nonsense. In Pascal, you read files with AssignFile/Reset/Read. You do not need classes for that ---------------------------------------------------------------------- PascalDragon · 2022-06-24 11:00 Quote from: BeniBela on June 24, 2022, 11:29:16 amIn Pascal, you read files with AssignFile/Reset/Read. You do not need classes for that In Object Pascal - especially when compatible with LCL/VCL -  file handling is done with TFileStream and friends, not with Pascal I/O. ---------------------------------------------------------------------- martinrame · 2022-06-24 11:56 Well, as this is taking too long I finally somewhat solved this way: 1. The CGI program gets the Id file (from the database, there's a relation between the filename and an Id). 2. The CGI runs (using TProcess) an external executable passing the ID as param. 3. The executable receives the ID, do an SQL query to find the filename, loads it into a TMemoryStream, saves to a new file called /tmp/.pdf and writes 'ok' to stdout. 4. The CGI reads the result of the program, if it's 'ok' then reads the file into a TMemoryStream and sends its content to the client. Awful, but works. ---------------------------------------------------------------------- SymbolicFrank · 2022-06-24 12:30 Did you try to convert the name from the console code page? ---------------------------------------------------------------------- martinrame · 2022-06-24 12:56 I tried everything, it looks like an issue related to Apache and/or the CGI, because the same program from command line works without issues. ---------------------------------------------------------------------- Thaddy · 2022-06-24 14:25 The file name, not the content...... < sigh> and even that is nonsense. That class has no right to live.... ---------------------------------------------------------------------- dsiders · 2022-06-24 17:15 Quote from: Thaddy on June 24, 2022, 04:25:13 pmThe file name, not the content...... < sigh> and even that is nonsense. That class has no right to live.... Well, it has been marked as deprecated, so... ---------------------------------------------------------------------- Thaddy · 2022-06-24 19:28 ....it should be removed... ---------------------------------------------------------------------- PascalDragon · 2022-06-25 16:31 Quote from: martinrame on June 24, 2022, 02:56:42 pmI tried everything, it looks like an issue related to Apache and/or the CGI, because the same program from command line works without issues. So to clear that up: if you run the program without any involvment of Apache you can correctly open the file with the non-ASCII characters? And if you do it inside Apache you can't? ---------------------------------------------------------------------- martinrame · 2022-06-27 11:50 Quote from: PascalDragon on June 25, 2022, 06:31:42 pmQuote from: martinrame on June 24, 2022, 02:56:42 pmI tried everything, it looks like an issue related to Apache and/or the CGI, because the same program from command line works without issues. So to clear that up: if you run the program without any involvment of Apache you can correctly open the file with the non-ASCII characters? And if you do it inside Apache you can't? Exactly. ---------------------------------------------------------------------- PascalDragon · 2022-06-28 11:28 Would you please dump the environment variables for both cases? Maybe there is some difference that can point to the issue.