accented char lazarus / 21-General ---------------------------------------------------------------------- Paolo · 2021-08-08 09:20 Hello, I have a small program that read a text file row by row and display it in a ListBox. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.OpenFile(const AFileName : string; ALstBox : TListBox; ALabel : TLabel);var  FilInp : TextFile;  Str1: string;begin  AssignFile(FilInp, AFileName);  try    Reset(FilInp);    while not(EOF(FilInp)) do begin      Readln(FilInp, Str1);      ALstBox.Items.Add(Str1);    end;  finally    CloseFile(FilInp);  end;end;  but I am facing the following problems: it happens that opening two files with the "same content" (at least opened wiht a simple text editor) one is correctly displaied the other has the characters like à,è,etc.. diplayed with other simbols, I really don't understand way. NB: the file that has bad character is coming from a pc with win-7 pro and saved either on USB or CD-Rom, then it is opend on the my machine win-10 6abit any help ? win-10 64, laz 2.0.12, FPC 3.2.0 ---------------------------------------------------------------------- JuhaManninen · 2021-08-08 10:03 Quote from: Paolo on August 08, 2021, 11:20:39 amI am facing the following problems: it happens that opening two files with the "same content" (at least opened wiht a simple text editor) one is correctly displaied the other has the characters like à,è,etc.. diplayed with other simbols, I really don't understand way. NB: the file that has bad character is coming from a pc with win-7 pro and saved either on USB or CD-Rom, then it is opend on the my machine win-10 6abit Lazarus GUI apps use Unicode with UTF-8 encoding by default. Your text files use a local Windows codepage. Windows codepages are a source of problems which Unicode solved a long time ago. The best solution is to convert the files to UTF-8. You can use one of the utility conversion programs out there to convert them all. Another solution is to convert the data just after reading it in your code. This page gives hints for it:  https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus ---------------------------------------------------------------------- Paolo · 2021-08-08 13:57 Thanks @Juha, I did Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.OpenFile(const AFileName : string; ALstBox : TListBox; ALabel : TLabel);var  FilInp : TextFile;  Str1: string;  StrIn: RawByteString;begin  AssignFile(FilInp, AFileName);  try    Reset(FilInp);    while not(EOF(FilInp)) do begin      Readln(FilInp, StrIn);      Str1:=WinCPToUTF8(StrIn);      ALstBox.Items.Add(Str2);  end;    ALabel.Caption:=AFileName;  finally    CloseFile(FilInp);  end;end;  now the file initially showing wrong char is fine with accented char, but the one that was good at the begin showed different chars... At the moment the solution is to have an added flag to mange the two cases, but what is happening ? How can I detect if the "WinCPToUTF8" is necessary ? ---------------------------------------------------------------------- Jurassic Pork · 2021-08-08 15:25 hello, Try the ChsDet (CharacterSet Detector) available through Online- Package-Manager. have a look here :  http://chsdet.sourceforge.net/ comparison of 3 methods to detect character set in attachment. content of the files : QuoteEn écrivant ma pensée, elle m'échappe quelquefois; mais cela me fait me souvenir de ma faiblesse, que j'oublie à toute heure; ce qui m'instruit autant que ma pensée oubliée, car je ne tends qu'à connaître mon néant. Pascal, Pensées, Première partie, Chapitre II. Friendly, J.P ---------------------------------------------------------------------- JuhaManninen · 2021-08-08 15:45 Quote from: Paolo on August 08, 2021, 03:57:17 pmnow the file initially showing wrong char is fine with accented char, but the one that was good at the begin showed different chars... At the moment the solution is to have an added flag to mange the two cases, but what is happening ? I guess those files are already encoded as UTF-8. You really should use the same UTF-8 for them all. QuoteHow can I detect if the "WinCPToUTF8" is necessary ? Unit LConvEncoding has function GuessEncoding(). It uses heuristics and may not be 100% accurate always. ---------------------------------------------------------------------- Jurassic Pork · 2021-08-08 16:03 Juha in the comparison that i have added in my previous message lconvencoding.pas is using GuessEncoding : seems to be not so bad ---------------------------------------------------------------------- Paolo · 2021-08-08 16:22 Thaks to both, I'll check as soon as possible your suggestions. Let me explain the situation: I write code on pc A, win-10 either lazarus or delphi, copy the file (simple pas unit), save on the pc B with delphi on win-7, do modification, then save, copy and paste on pc A, try to compare the chages, and here the problems. ---------------------------------------------------------------------- Jurassic Pork · 2021-08-08 16:31 Quote from: Paolo on August 08, 2021, 06:22:52 pmThaks to both, I'll check as soon as possible your suggestions. Let me explain the situation: I write code on pc A, win-10 either lazarus or delphi, copy the file (simple pas unit), save on the pc B with delphi on win-7, do modification, then save, copy and paste on pc A, try to compare the chages, and here the problems. what  are your versions of lazarus and delphi ? may be the editor in the Delphi IDE uses ANSI encoding ---------------------------------------------------------------------- Paolo · 2021-08-08 16:36 Tokyo, laz 2.0.12, fpc 3.2.0 ---------------------------------------------------------------------- engkin · 2021-08-08 17:55 I think the problem is in the text files, more precisely the way you generate these file. How do you generate your files? ---------------------------------------------------------------------- Paolo · 2021-08-08 18:07 The origin of pas file is delphi (since delphi 3 up to tokyo) , then now manipulated and modified by lazarus/fpc. What I think is that the problem is originated by the going back an forth of file between different pc, i cannot now check but the file opened in the editor either delphi on pc A or laz on pc B, all seems fine. ---------------------------------------------------------------------- engkin · 2021-08-08 18:15 The code in your first post is used to read pas files? Edit QuoteI have a small program that read a text file row by row and display it in a ListBox ---------------------------------------------------------------------- Paolo · 2021-08-08 19:46 Yes, to compare the content of two pas file. If I'll be able to reproduce the situation I'll post both the files and the code to read them. ---------------------------------------------------------------------- Paolo · 2021-08-09 16:57 attached the extracted project that reads the file Prova-1.pas and Prova-2.pas and show them on listboxes. Use the first button to load Prova-1.pas and the second one to load Prova-2.pat. You should see the accented char "è" in the comment correctly (See the code why), if you reverse the order, load Prova-2.pas with the first button and viceversa, both are wrongly shown. ---------------------------------------------------------------------- engkin · 2021-08-09 18:05 Prova-1.pas is utf8 Prova-2.pas is not Open both files in Lazarus and right click on each File Settings   Encoding     See which one is what and change to utf8 ---------------------------------------------------------------------- Paolo · 2021-08-10 12:23 thanks engkin, however in the real situation the file is just one file, say "prova.pas", as soon as possible I'll try to verify "who" or "what" make the file different... ---------------------------------------------------------------------- engkin · 2021-08-10 15:29 You can check if a string is a valid UTF8 string by using FindInvalidUTF8Codepoint from unit LazUtf8, as in: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses ...LazUtf8;..  s:string;  p:PtrInt;begin  s:='ab'#$88'cd';  //<--- not valid because of the #$88  p:=FindInvalidUTF8Codepoint(@s[1],Length(s),True);  if p<>-1 then //NOT Valid  If your code is for your personal use, or limited to one ANSI codepage, you can use WinCPToUTF8 to correct the string. Otherwise you can provide some mean for the user to choose the correct codepage.