[Windows] UTF8 encoding with ReadLn lazarus / 34-Windows ---------------------------------------------------------------------- bstewart · 2026-01-20 15:41 Consider this simple source: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test; {$CODEPAGE UTF8}{$MODE OBJFPC}{$MODESWITCH UNICODESTRINGS} uses  Windows; var  S: string; begin  Write('Enter δείγμα: ');  ReadLn(S);  WriteLn('S := ' + S);end.  I run this in a cmd.exe session and enter the command chcp 65001 to set the console input and output encoding to UTF8, run it, and enter a UTF8 string. Output: Code: [Select]C:\Users\username\FPC\source\test>chcp 65001 Active code page: 65001 C:\Users\username\FPC\source\test>fpc test.pp Free Pascal Compiler version 3.2.2 [2021/05/15] for i386 Copyright (c) 1993-2021 by Florian Klaempfl and others Target OS: Win32 for i386 Compiling test.pp Linking test.exe 17 lines compiled, 0.1 sec, 30480 bytes code, 1348 bytes data C:\Users\username\FPC\source\test>.\test Enter δείγμα: δείγμα S := de??µa For the ReadLn call, I copy and paste the string δείγμα and hit Enter. I as expecting the WriteLn call to output the UnicodeString as I entered it. What would be needed to get ReadLn to read a UTF8 string properly? ---------------------------------------------------------------------- andersonscinfo · 2026-01-20 16:05 The issue you're experiencing is related to how Free Pascal handles console I/O on Windows when dealing with UTF-8 encoded input. Even though you've set the codepage to UTF-8 (65001) and enabled Unicode strings in FPC, the standard ReadLn function doesn't properly handle UTF-8 input from the Windows console. Here's the solution: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test;{$CODEPAGE UTF8}{$MODE OBJFPC}{$MODESWITCH UNICODESTRINGS} uses  Windows; function ConsoleReadLn: String;var  hConsole: THandle;  Buffer: packed array[0..1023] of WideChar;  NumRead: DWORD;  UTF8Str: UTF8String;begin  hConsole := GetStdHandle(STD_INPUT_HANDLE);  NumRead := 0;    // Read wide characters from console  ReadConsoleW(hConsole, @Buffer[0], SizeOf(Buffer) div SizeOf(WideChar) - 1, NumRead, nil);  // Null terminate  Buffer[NumRead] := #0;    // Convert WideString to UTF8String  SetLength(UTF8Str, WideCharLenToString(nil, @Buffer[0], NumRead));  WideCharToMultiByte(CP_UTF8, 0, @Buffer[0], NumRead, PAnsiChar(UTF8Str), Length(UTF8Str), nil, nil);    Result := String(UTF8Str);    // Remove trailing newline if present  if (Length(Result) > 0) and (Result[Length(Result)] = #10) then    Delete(Result, Length(Result), 1);  if (Length(Result) > 0) and (Result[Length(Result)] = #13) then    Delete(Result, Length(Result), 1);end; var  S: string;begin  Write('Enter δείγμα: ');  S := ConsoleReadLn;  WriteLn('S := ' + S);end.  Alternatively, you can use a simpler approach by using the Windows API directly: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test;{$CODEPAGE UTF8}{$MODE OBJFPC}{$MODESWITCH UNICODESTRINGS} uses  Windows, SysUtils; function ReadUTF8Line: String;var  hInput: THandle;  Buffer: array[0..1023] of WideChar;  NumRead: DWORD;  UTF8Str: UTF8String;begin  hInput := GetStdHandle(STD_INPUT_HANDLE);  NumRead := 0;    ReadConsoleW(hInput, @Buffer[0], SizeOf(Buffer) div SizeOf(WideChar) - 1, NumRead, nil);    if NumRead > 0 then  begin    Buffer[NumRead] := #0;    SetLength(UTF8Str, WideCharLenToString(nil, @Buffer[0], NumRead));    WideCharToMultiByte(CP_UTF8, 0, @Buffer[0], NumRead, PAnsiChar(UTF8Str), Length(UTF8Str), nil, nil);    Result := String(UTF8Str);        // Remove carriage return and line feed    if (Length(Result) > 0) and (Result[Length(Result)] = #13) then      SetLength(Result, Length(Result) - 1);    if (Length(Result) > 0) and (Result[Length(Result)] = #10) then      SetLength(Result, Length(Result) - 1);  end  else    Result := '';end; var  S: string;begin  Write('Enter δείγμα: ');  S := ReadUTF8Line;  WriteLn('S := ' + S);end.  The problem occurs because the standard ReadLn function in FPC doesn't properly translate UTF-8 input from the Windows console. The Windows console API functions (ReadConsoleW) handle Unicode properly, so using them directly resolves the issue. The key points are: 1. Using ReadConsoleW to read wide characters from the console 2. Converting the wide character buffer to UTF-8 using WideCharToMultiByte 3. Properly handling the conversion to ensure the UTF-8 string is correctly formed This approach will properly capture and display your Greek text "δείγμα" as expected. ---------------------------------------------------------------------- jcmontherock · 2026-01-20 16:41 With the "Keysboard" component I never met encoding problems. It's more better than ReadLn... ---------------------------------------------------------------------- bstewart · 2026-01-20 16:58 I was really looking for more of a confirmation that Read/ReadLn is broken. Seems so. Using the Windows APIs is non-starter for other platforms (understood that I posted in the Windows OS section; I'm merely pointing it out). Perhaps the ongoing overhaul of the RTL will address this limitation. ReadLn shouldn't fail in this manner... ---------------------------------------------------------------------- andersonscinfo · 2026-01-20 17:08 try this Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program ConsoleUTF8Test; {$MODE OBJFPC}      // Modo padrão moderno Object Pascal{$H+}               // Strings longas (AnsiString) ativadas{$CODEPAGE UTF8}    // Define a codepage do fonte para UTF-8 uses  {$IFDEF UNIX}  BaseUnix,         // Em Linux, o ReadLn padrão geralmente funciona bem com UTF-8  {$ENDIF}  {$IFDEF WINDOWS}  Windows,          // Necessário para as APIs de console  {$ENDIF}  SysUtils; // Função encapsulada para ler UTF-8 do Console de forma Cross-Platformfunction ConsoleReadLn: string;{$IFDEF WINDOWS}var  hStdIn: THandle;  NumRead: DWORD;  Buffer: array[0..4095] of WideChar; // Buffer generoso para WideString  ResWide: UnicodeString;begin  hStdIn := GetStdHandle(STD_INPUT_HANDLE);    // Tenta ler do console usando a API Unicode do Windows  if ReadConsoleW(hStdIn, @Buffer, Length(Buffer), NumRead, nil) then  begin    // Define o tamanho da string Unicode baseada no que foi lido    SetLength(ResWide, NumRead);    Move(Buffer[0], ResWide[1], NumRead * SizeOf(WideChar));        // Converte nativamente de UTF-16 para a String padrão (UTF-8 definido no CP)    Result := UTF8Encode(ResWide);  end  else    Result := ''; // Falha na leitura      // Tratamento clássico de fim de linha (CR/LF)  while (Length(Result) > 0) and (Result[Length(Result)] in [#13, #10]) do    SetLength(Result, Length(Result) - 1);end;{$ELSE}begin  // Em Unix/Linux, o sistema de arquivos e console já costumam ser UTF-8 nativo  ReadLn(Result);end;{$ENDIF} var  Entrada: string;begin  // Configuração essencial para SAÍDA correta no Windows  {$IFDEF WINDOWS}  SetConsoleOutputCP(CP_UTF8);  SetConsoleCP(CP_UTF8);  // Ajuste de fonte pode ser necessário no terminal do Windows (ex: Lucida Console)  {$ENDIF}   Write('Digite algo (ex: ελληνικά / Português): ');    // Usamos nossa função robusta  Entrada := ConsoleReadLn;   WriteLn('Você digitou: ', Entrada);  WriteLn('Tamanho em bytes: ', Length(Entrada));  WriteLn('Tamanho em caracteres (UTF8Length): ', UTF8Length(Entrada)); // Requer LazUTF8 ou implementação manual em FPC puro   WriteLn('Pressione ENTER para sair...');  ReadLn;end. ---------------------------------------------------------------------- cdbc · 2026-01-20 17:23 Hi @bstewart: Try this:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test; {$MODE OBJFPC}{$H+} uses sysutils; var  S: string; begin  Write('Enter δείγμα: ');  ReadLn(S);  WriteLn('S := ' + S);end.it produces this: Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i bc@red bstewart$ fpc test.pp Free Pascal Compiler version 3.2.2 [2024/07/26] for x86_64Copyright (c) 1993-2021 by Florian Klaempfl and othersTarget OS: Linux for x86-64Compiling test.ppLinking test15 lines compiled, 0.3 secbc@red bstewart$ ./test Enter δείγμα: δείγμαS := δείγμαbc@red bstewart$  Ofc. this was on Linux, but I think you're overcomplicating things... Regards Benny ---------------------------------------------------------------------- LV · 2026-01-20 18:14 Hi! Try this. ---------------------------------------------------------------------- marcov · 2026-01-20 18:14 Quote from: bstewart on January 20, 2026, 04:41:22 pmI run this in a cmd.exe session and enter the command chcp 65001 to set the console input and output encoding to UTF8, run it, and enter a UTF8 string. Output: All too complicated. Note that $codepage only influences the codepage of the strings in the source, it doesn't do much for thje rest. For Windows, go the the application tab in the "project->options", turn the manifest on, and turn on "ANSI codepage is UTF-8, (Windows 10 1903+)" and it will simply work unless you are on unsupported Windows versions. ---------------------------------------------------------------------- bstewart · 2026-01-20 18:25 QuoteAll too complicated. Note that $codepage only influences the codepage of the strings in the source, it doesn't do much for thje rest. Yes, aware of that; I added it to the sample to tell FPC the source is UTF8. QuoteFor Windows, go the the application tab in the "project->options", turn the manifest on, and turn on "ANSI codepage is UTF-8, (Windows 10 1903+)" and it will simply work unless you are on unsupported Windows versions. I know this is a Lazarus forum, but I'm actually not using Lazarus. How do I do this if compiling from the command line? ---------------------------------------------------------------------- marcov · 2026-01-20 18:28 Quote from: bstewart on January 20, 2026, 07:25:47 pmI know this is a Lazarus forum, but I'm actually not using Lazarus. How do I do this if compiling from the command line? Basically that tab in Lazarus is only wizard to click together a manifest that is then translated to a .res. You can look it up in MSDN and create the manifest and then compile it with windres or fpcres. Then link it to the EXE with $R. ---------------------------------------------------------------------- LV · 2026-01-20 18:45 Quote from: bstewart on January 20, 2026, 07:25:47 pmI know this is a Lazarus forum, but I'm actually not using Lazarus. How do I do this if compiling from the command line? ---------------------------------------------------------------------- bstewart · 2026-01-20 18:46 QuoteThen link it to the EXE with $R. Confirmed working, thanks! How do we fix ReadLn without a manifest so it runs on older platforms? Or is it not possible? ---------------------------------------------------------------------- bstewart · 2026-01-20 19:09 Regarding the manifest: Here's the MCVE. File test.rc: Code: [Select]1 24 "test.manifest" File test.manifest: Code: [Select]               UTF-8       Compile test.rc to test.res: Code: [Select]windres -i test.rc -o test.res Finally, compile using the resource by adding the following to the top of the source file: Code: [Select]{$R *.res} ---------------------------------------------------------------------- LV · 2026-01-20 19:20 Quote from: bstewart on January 20, 2026, 07:46:59 pmHow do we fix ReadLn without a manifest so it runs on older platforms? Or is it not possible? 😉 ---------------------------------------------------------------------- bstewart · 2026-01-20 19:25 @andersonscinfo - beat you to it ;-) @LV - run your sample without calling the extra Windows APIs and without a resource (*.res) file. ---------------------------------------------------------------------- bstewart · 2026-01-20 19:44 This works without a manifest: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test; {$CODEPAGE UTF8}{$MODE OBJFPC}{$MODESWITCH UNICODESTRINGS} uses  windows; var  S: string; begin  SetMultiByteConversionCodePage(CP_UTF8);  Write('Enter δείγμα: ');  ReadLn(S);  WriteLn('S := ' + S);end.  This means, though, that there's an implicit conversion happening. The goal is to read and write UTF8 strings without any conversions. ---------------------------------------------------------------------- LV · 2026-01-20 20:05 Quote from: bstewart on January 20, 2026, 08:25:49 pm@LV - run your sample without calling the extra Windows APIs and without a resource (*.res) file. Running this code ├─ utf8_2    ├─ test.lpr Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test; {$MODE OBJFPC}{$H+}{$codepage utf8} uses  SysUtils, Windows; var  S: String; begin  SetConsoleCP(CP_UTF8);  SetConsoleOutputCP(CP_UTF8);   SetMultiByteConversionCodePage(CP_UTF8);  SetTextCodePage(Output, CP_UTF8);  SetTextCodePage(Input, CP_UTF8);   Write('Enter 🤔 δείγμα 😊: ');  ReadLn(S);  WriteLn('S := ', S);   WriteLn('Windows 11; FPC 3.2.2');  ReadLn;end.  from the cmd Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i @echo offset FPC="C:\lazarus\fpc\3.2.2\bin\x86_64-win64\fpc.exe" %FPC% -Twin64 -Px86_64 -otest64.exe S:\utf8_2\test.lpr%FPC% -Twin32 -Pi386  -otest32.exe S:\utf8_2\test.lpr echo Donepause  🫡 ---------------------------------------------------------------------- bstewart · 2026-01-20 20:08 Correct; as I acknowledged, it works when you call SetMultiByteConversionCodePage. However, as I'll say again, this works because there's an implicit conversion happening. As I wrote previously: The goal is for it to work without implicit conversions. ---------------------------------------------------------------------- andersonscinfo · 2026-01-20 22:38 QuoteQuote from: bstewart on Today at 07:46:59 pm [[https://forum.laza rus.freepascal.org/index.php/topic,73290.msg574592.html#msg574592](htt ps://forum.lazarus.freepascal.org/index.php/topic,73290.msg574592.html #msg574592)]How do we fix ReadLn without a manifest so it runs on older platforms? Or is it not possible? Unfortunately, **it's not possible to reliably fix `ReadLn` for UTF-8 input on older Windows versions without a manifest**. Here’s why: The issue stems from how the Windows console (cmd.exe) handles Unicode input/output before Windows 10 version 1903. Prior to this, the console was fundamentally designed around ANSI codepages, and even when you set `chcp 65001`, the underlying I/O functions used by the RTL often encounter bugs or limitations with multi-byte sequences. ### Why the Manifest Works The manifest enables the "ANSI codepage is UTF-8" feature introduced in Windows 10 1903. This tells the OS to treat all ANSI APIs (like those used by FPC’s `ReadLn`) as UTF-8, effectively making them work correctly with Unicode strings. Without this, the console’s internal handling of UTF-8 sequences breaks, leading to garbled output like `de??µa`. ### Alternatives for Older Platforms If you need compatibility with older Windows versions (7/8/early 10), here are some workarounds: 1. **Use Windows API (`ReadConsoleW`)** Standard `ReadLn` relies on ANSI system calls. To properly read Unicode on older Windows, you must bypass the standard input mechanism and use `ReadConsoleW`, which reads UTF-16 (WideString) directly from the console buffer. You can then convert the result to UTF-8. ```pascal Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program test;{$MODE OBJFPC}{$H+}uses Windows, SysUtils; var  hStdIn: THandle;  dwRead: DWORD;  Buffer: array[0..1023] of WideChar;  WStr: WideString;  S: String;begin  // Set output to UTF-8 so we can print the result correctly  SetConsoleOutputCP(CP_UTF8);  Write('Enter δείγμα: ');   hStdIn := GetStdHandle(STD_INPUT_HANDLE);   // Read WideChars directly to avoid ANSI conversion issues  if ReadConsoleW(hStdIn, @Buffer, Length(Buffer), dwRead, nil) then  begin    SetString(WStr, Buffer, dwRead);     // Trim CR/LF from the end manually    while (Length(WStr) > 0) and (WStr[Length(WStr)] < #32) do      Delete(WStr, Length(WStr), 1);     S := UTF8Encode(WStr); // Convert safely to UTF-8  end;   WriteLn('S := ' + S);end.  ``` 2. **Use a Different Console Emulator** Tools like ConEmu or Windows Terminal provide better Unicode support than the default cmd.exe, even on older Windows versions. However, this shifts the burden to the user rather than solving it programmatically. 3. **Switch to GUI Applications** If feasible, consider building a GUI application using Lazarus or another framework. GUI controls handle Unicode input much more reliably across different Windows versions. ### Conclusion While there are workarounds, none are as seamless as using the manifest approach. For legacy support on older Windows, using `ReadConsoleW` is the most robust programmatic solution, as the standard `ReadKey` or `ReadLn` functions are prone to encoding errors with multi-byte characters in those environments. Hope this helps! 😊 ---------------------------------------------------------------------- bstewart · 2026-01-20 23:08 QuoteStandard ReadLn relies on ANSI system calls. ...and thus this issue. In my thinking, {$MODESWITCH UNICODESTRINGS} should use the W-suffix API calls on the Windows side. Perhaps this is coming at some point. ---------------------------------------------------------------------- Thaddy · 2026-01-21 09:42 No because different windows versions have different codepages: you either have to explicitly set your codepage for the terminal in your Windows version to UTF8/65001 OR use the suggestion by LV which is the only correct one: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   SetConsoleCP(CP_UTF8);  SetConsoleOutputCP(CP_UTF8);   SetMultiByteConversionCodePage(CP_UTF8);  SetTextCodePage(Output, CP_UTF8);  SetTextCodePage(Input, CP_UTF8); For a good example on how this works, see the console code from member Warfley https://github.com/Warfley/LazTermUtils That code is highly recommended to study the matter at hand. That also shows how to keep your code compatible over multiple platforms. (And YES he also uses the above suggestion by LV.....because that is how it should be done) The thing is that simply SetMultiByteConversionCodePage is not enough to cover full console input/output, subs readln/writeln and consorts on Windows. (Just to be sure: if you use that code from a GUI app, make sure there is a console in the first place, either through the Windows API or through compiling with {$apptype console} the latter automatically creates a console window in GUI apps) LV did it a bit better, though, capturing  SetTextCodePage(Input, CP_UTF8) too and that is for your readln. ---------------------------------------------------------------------- PascalDragon · 2026-01-22 20:51 Quote from: bstewart on January 21, 2026, 12:08:42 amQuoteStandard ReadLn relies on ANSI system calls. ...and thus this issue. In my thinking, {$MODESWITCH UNICODESTRINGS} should use the W-suffix API calls on the Windows side. Perhaps this is coming at some point. No, because {$ModeSwitch UnicodeString} is only unit local, thus does not influence functions in other units (most importantly the Windows unit). Also it's best to simply not use that switch. In the development version the correct approach is to compile the RTL for Unicode which will make use of the -W-functions and use String = UnicodeString in general. ---------------------------------------------------------------------- bstewart · 2026-01-26 16:56 QuoteNo, because {$ModeSwitch UnicodeString} is only unit local, thus does not influence functions in other units (most importantly the Windows unit). Of course; what I was trying to say (rather poorly) is that the RTL and Windows unit should be compiled to use Unicode strings (i.e., the 'W'-suffix API calls). QuoteAlso it's best to simply not use that switch. If the entire RTL can be compiled on Windows using the 'W'-suffix APIs, then I would agree. Just to state it explicitly, one of the primary purposes of using the 'W' APIs [Unicode] is to avoid these kinds of issues. QuoteIn the development version the correct approach is to compile the RTL for Unicode... Is there a how-to guide somewhere on how to accomplish this? ---------------------------------------------------------------------- PascalDragon · 2026-01-27 19:58 Quote from: bstewart on January 26, 2026, 05:56:56 pmQuoteAlso it's best to simply not use that switch. If the entire RTL can be compiled on Windows using the 'W'-suffix APIs, then I would agree. Just to state it explicitly, one of the primary purposes of using the 'W' APIs [Unicode] is to avoid these kinds of issues. You currently get more confusion and conversion by using that modeswitch, because the remainder of the RTL still uses String = AnsiString and thus the unit where you enable the switch will convert (e.g. if you use a TStringList) while if the mode is disabled the strings will mainly be passed through. Quote from: bstewart on January 26, 2026, 05:56:56 pmQuoteIn the development version the correct approach is to compile the RTL for Unicode... Is there a how-to guide somewhere on how to accomplish this? Here.