Extended ASCII use - 2 lazarus / 21-General ---------------------------------------------------------------------- raymond · 2022-01-06 14:28 In the UTF8 code set 62 (55%) of the characters are for 'European' characters. !!! ( fpc for FreeDOS + code set 850 was nirvana ). Does anybody KNOW how to manipulate strings/arrays of 'European' characters ?? Proved examples, please. I would faint with gratitude. Many thanks. ---------------------------------------------------------------------- JuhaManninen · 2022-01-06 14:53 What do you mean? UTF-8 encoding supports the full Unicode. If you mean the 7-bit ASCII by 'European' characters, then it gets easy because UTF-8 is compatible with 7-bit ASCII. ---------------------------------------------------------------------- engkin · 2022-01-06 16:05 Use unit LazUTF8. If you are working on a terminal/console app, you need to add LazUtils package where LazUTF8 is. You do that in:   Project - Project Inspector     Add - Add New Requirement       Type LazU and choose LazUtils What you call character is actually more of a string Use UTF8Copy, UTF8Insert...etc ---------------------------------------------------------------------- engkin · 2022-01-06 16:14 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program Project1; {$mode objfpc}{$H+} uses  {$IFDEF UNIX}{$IFDEF UseCThreads}  cthreads,  {$ENDIF}{$ENDIF}  Classes  ,LazUTF8  { you can add units after this }; var  s:string;  s1,s2:string;begin  s := 'ÄÇ';  WriteLn(s);  WriteLn(Length(s));//===> 4  WriteLn(UTF8Length(s));//===> 2  s1:=UTF8Copy(s,1,1);  WriteLn(s1); // Ä  s2:=UTF8Copy(s,2,1);  WriteLn(s2); // Ç  UTF8Insert(s2,s,1); // s is ÇÄÇ  WriteLn(s);  UTF8Delete(s,2,1);  // s is ÇÇ  WriteLn(s);  ReadLn;end. From your previous post, add cwstring unit if you are using Linux ---------------------------------------------------------------------- engkin · 2022-01-06 16:25 Quote from: raymond on January 06, 2022, 03:28:40 pmIn the UTF8 code set 62 (55%) of the characters are for 'European' characters. !!! Not sure where you got that. UTF8 is ASCII compatible encoding of Unicode. Unicode codepoints can take up to 4 bytes when encoded using UTF8. A is one byte and is ASCII compatible. Ä is two bytes and is not compatible with ASCII. A "character" can use more than one codepoint. The same "character" could be represented with different codepoints. ---------------------------------------------------------------------- JuhaManninen · 2022-01-06 16:26 Yes. The sample code applies to any UTF-8 text. I didn't quite understand what 'European' characters meant. ---------------------------------------------------------------------- engkin · 2022-01-06 16:28 Me neither, but I looked at his other post. ---------------------------------------------------------------------- raymond · 2022-01-11 13:52 Whilst I'm evaluating the replies I would say ASCII means 'American Standard Code ..' and is 128 values. An accented character only occurs in a European language and features in the 'Extended' part of  ASCII (in the UTF8 code set). My interest is in using fpc without Lazarus, as was possible with fpc and FreeDOS and code set IBM 850, which covers all the 'European' characters. UTF8 is a single-byte code set and should be usable as such. In fpc can one use UTF8 as single bytes in arrays ? ---------------------------------------------------------------------- tetrastes · 2022-01-11 14:20 UTF8 is NOT single-byte code set. It uses one byte only for first 128 (ASCII) characters. For others it uses up to 4 bytes. So if you want what you call "'extended' part of ASCII" to be one- byte, change your system codepage to IBM850, and fpc will work as you want. ---------------------------------------------------------------------- Thaddy · 2022-01-11 14:53 Quote from: raymond on January 11, 2022, 02:52:51 pmUTF8 is a single- byte code set and should be usable as such. In fpc can one use UTF8 as single bytes in arrays ? No. UTF8 is NOT a single byte code set! Indeed, CP_UTF8 can use up to 4 bytes and for e.g. arrays and database field you will have to reserve 4 bytes per char! This is often overlooked. So the answer is no. You can not treat UTF8 as single byte. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-11 16:25 C̵̢̦̖̭̯̮̦̼̺̮̮̺̳̹̰̤̆̿̈́̓͆̄͌̉̂̏̑̌͋͝ŗ̴̡̻̫̻͖͍̙̥̬͔̩̖̰̱̈̌́̈́̀͛͋͌̈́͂̿̃͊̿͑̀̚à̴̈́ ̢̦͓͕͍̘̰̯͔͕͋̈̃̀̒̐͛͜ͅz̸̢̧̛̦̘̥̥̩͓͍̖͇͔͍̦͚̤͒̐̅̄͊͋̍͑́͒̅̎͝͝ỳ̸̨̫̳̤͕̬͈̹̭̜̠̻̹̅ ̵̢̨̧͉̥̹̜͔̬̬̞́̽̉̓̇̅̾͝ŝ̸͔̲̹̄̚ͅt̸̡̯͎̳̩̪͍͈̔̍̈́͘͜ũ̶̞͕̠̺̈́́̌͑̐̿̔̌͒͋̋͌̕͠f̸̀̐̉ ̡͚̾̋̑̕͠͠f̷̛̩͔̰̄̊͗̋͗̆̅̌͒̄̒͐͛͐͝,̸̢̮͙̜̬͍̱͈͙̟̳̬̑͗̾́̋̄̈́͜ͅ ̸̟͈͎̘͕̙̺͖̙̝̺̬͑e̷̽ ̧̧̢͉͍͖͚̗̻̳͎͛͗̀̽͂͌̅͌͊̈̈̋̈́͘͘̚h̵͖̘͇̦̞̪̓̈̈́̄̆͗̅̆̓̿̒̄̂͋̓̚?̷̟̪͍̜̳̹͖̯͊̏̑͠͝ 16 chars, 676 bytes. ---------------------------------------------------------------------- JuhaManninen · 2022-01-11 16:26 @raymond, I suggest you forget IBM 850 and all other 'Extended ASCII' codepages. They lead to all kinds of agony when data is exchanged. Just convert your data to Unicode. It has solved such problems already decades ago. [Edit] Crazy stuff, eh? Try to do that with Extended ASCII codepages! ---------------------------------------------------------------------- engkin · 2022-01-11 18:58 @raymond, Quote from: raymond on January 11, 2022, 02:52:51 pmMy interest is in using fpc without Lazarus, as was possible with fpc and FreeDOS and code set IBM 850, which covers all the 'European' characters. 1-Make sure to tell the compiler the codepage/encoding you use in your source code. For instance, if a file is using the same codepage you want, IBM 850, then add this to the top of file: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc}{$H+}{$Codepage cp850}//<--- Tell the compiler about the codepage used in this file uses 2-Make sure your dos is using the same codepage by calling chcp What output do you get for this test project: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc}{$H+}{$Codepage cp850} var  s:array[0..3] of String;  ss:string;  i:byte;  c:char;begin  WriteLn('DefaultSystemCodePage: ',DefaultSystemCodePage);  WriteLn('TextRec(Output).CodePage: ',TextRec(Output).CodePage);  ss:='ÄÇýÝ';  WriteLn(ss);   //CP850: #$80..#$FF  s[0]:='ÇüéâäàåçêëèïîìÄÅÉæÆôöòûùÿÖÜø£Ø×ƒ';  s[1]:='áíóúñѪº¿®¬½¼¡«»░▒▓│┤ÁÂÀ©╣║╗╝¢¥┐';  s[2]:='└┴┬├─┼ãÃ╚╔╩╦╠═╬¤ðÐÊËÈıÍÎÏ┘┌█▄¦Ì▀';  s[3]:='ÓßÔÒõÕµþÞÚÛÙýݯ´­±‗¾¶§÷¸°¨·¹³²■';  for i:=$80 to $FF do  begin    if (i mod 32)=0 then    begin      WriteLn();      WriteLn();      WriteLn(s[i div 32 - 4]);    end;    if i in [$07,$08,$09,$0A,$0D] then      c:=' '    else      c:=char(i);    Write(c);  end;end. ---------------------------------------------------------------------- tetrastes · 2022-01-11 19:48 Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i   for i:=$80 to $FF do  begin    if (i mod 32)=0 then    begin      WriteLn();      WriteLn();      WriteLn(s[i div 32 - 4]);    end;    if i in [$07,$08,$09,$0A,$0D] then      c:=' '    else      c:=char(i);   // ! This is invalid in utf8 and leads to runtime error 101    Write(c);  end; ---------------------------------------------------------------------- Thaddy · 2022-01-11 20:01 Note that if you do not use Lazarus (As you stated) but only FPC, you can use {$mode delphiunicode} and char would be unicodechar. But that is by no means single byte, mostly double byte (the ucs2 part) and possibly  - again - 4 bytes. This mode will make your life easier, though. This example covers technically UCS2, but that is an ancestor to current unicode16 and a 4 byte unicodechar is not implemented as such:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program size;{$mode delphiunicode}begin  writeln(SizeOf(Char));end.In an ideal world it should return 4, imho.  OTOH this was defined before the unicode extensions in the standard and fpc can handle the 4 case as well. Just like Delphi. That does mean you still have to reserve 4 bytes per UnicodeChar, not two as per my example. ---------------------------------------------------------------------- engkin · 2022-01-11 20:36 Quote from: tetrastes on January 11, 2022, 08:48:48 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i       c:=char(i);   // ! This is invalid in utf8 and leads to runtime error 101  What would be the correct way considering: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc}{$H+}{$Codepage cp850} var..  i:byte;  c:char; ---------------------------------------------------------------------- tetrastes · 2022-01-11 20:52 Quote from: Thaddy on January 11, 2022, 09:01:35 pmIn an ideal world it should return 4, imho.  By the way, at linux Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i sizeof(wchar_t)returns 4. ---------------------------------------------------------------------- Thaddy · 2022-01-11 21:00 Yes. ---------------------------------------------------------------------- tetrastes · 2022-01-11 21:06 @engkin To imitate CP850 at UTF8 console without errors: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program project1; {$mode objfpc}{$H+}{$Codepage cp850} {$IFDEF UNIX}uses cwstring;    // We need properly working UnicodeManager, of course{$ENDIF} type  CP850String = type AnsiString(850); var  s:array[0..3] of String;  ss:string;  i:byte;  c:char;begin  WriteLn('DefaultSystemCodePage: ',DefaultSystemCodePage);  WriteLn('TextRec(Output).CodePage: ',TextRec(Output).CodePage);  ss:='ÄÇýÝ';  WriteLn(ss);   //CP850: #$80..#$FF  s[0]:='ÇüéâäàåçêëèïîìÄÅÉæÆôöòûùÿÖÜø£Ø×ƒ';  s[1]:='áíóúñѪº¿®¬½¼¡«»░▒▓│┤ÁÂÀ©╣║╗╝¢¥┐';  s[2]:='└┴┬├─┼ãÃ╚╔╩╦╠═╬¤ðÐÊËÈıÍÎÏ┘┌█▄¦Ì▀';  s[3]:='ÓßÔÒõÕµþÞÚÛÙýݯ´­±‗¾¶§÷¸°¨·¹³²■';  for i:=$80 to $FF do  begin    if (i mod 32)=0 then    begin      WriteLn();      WriteLn();      WriteLn(s[i div 32 - 4]);    end;    if i in [$07,$08,$09,$0A,$0D] then      c:=' '    else      c:=char(i);    Write(CP850String(c));    // SIC!  end;end. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-11 21:38 Quote from: Thaddy on January 11, 2022, 09:01:35 pmIn an ideal world it should return 4, imho.  Totally agree. Go Linux! ---------------------------------------------------------------------- engkin · 2022-01-11 21:55 @tetrastes, If chcp gave 850, what's the difference between: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     Write(c); and Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     Write(CP850String(c)); ---------------------------------------------------------------------- tetrastes · 2022-01-11 22:24 Casting char to string? The output is the same. It's funny that at windows there is difference between Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i write(c)which outputs in console CP, and Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i write(string(c))which outputs in default system CP. And you may change them independently. ---------------------------------------------------------------------- engkin · 2022-01-11 22:37 Quote from: tetrastes on January 11, 2022, 11:24:16 pmCasting char to string? The output is the same. It's funny that at windows there is difference between Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i write(c)which outputs in console CP, and Right, and that is what raymond wants. ---------------------------------------------------------------------- tetrastes · 2022-01-11 22:48 But he wants it at linux with UTF8, where write(c) works only for c<128. ---------------------------------------------------------------------- engkin · 2022-01-11 23:06 Quote from: tetrastes on January 11, 2022, 11:48:12 pmBut he wants it at linux with UTF8, where write(c) works only for c<128. In this case you are right, I missed that part. Thanks for correcting me. ---------------------------------------------------------------------- Thaddy · 2022-01-12 06:24 Anybody knows why UnicodeChar differs from wchar_t ? (or similar in any other unicode supporting language!) Currently it is just UCS2 size. It really should be size 4. So OK on Delphi it is also size 2, I guess. I suspect that a size 4 greatly simplifies handling codepoints. ---------------------------------------------------------------------- tetrastes · 2022-01-12 07:54 Not only on Delphi, but windows itself. Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i sizeof(wchar_t)is 2 at windows, 32 or 64 bit, gcc or cl. So fpc loves windows. ---------------------------------------------------------------------- PascalDragon · 2022-01-12 08:03 Quote from: Thaddy on January 12, 2022, 07:24:57 amAnybody knows why UnicodeChar differs from wchar_t ? (or similar in any other unicode supporting language!) Currently it is just UCS2 size. It really should be size 4. So OK on Delphi it is also size 2, I guess. I suspect that a size 4 greatly simplifies handling codepoints. Because Delphi defines UnicodeChar to be used with UTF-16 which has 2 Byte codepoints and Delphi in turn uses UTF-16, because Windows uses UTF-16. For UTF-32 there is UCS4Char (though this is not a true, builtin character type, but simply an alias to LongWord). Quote from: tetrastes on January 12, 2022, 08:54:11 amNot only on Delphi, but windows itself. Code: C  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i sizeof(wchar_t)is 2 at windows, 32 or 64 bit, gcc or cl. So fpc loves windows.  No, for FPC Delphi-compatibility is the main driver and thus UnicodeChar is declared the same as it is in Delphi which - as written above - uses UTF-16 (and thus 2 Byte codepoints), because Windows uses UTF-16. ---------------------------------------------------------------------- tetrastes · 2022-01-12 08:14 FPC follows Delphi, Delphi follows Windows... The leading sheep is Windows, anyway. ---------------------------------------------------------------------- PascalDragon · 2022-01-12 08:41 Even if Delphi or FPC would use a 4 Byte UnicodeChar there'd still be a need for a 2 Byte character to handle UTF-16. It's after all a reason why C++ introduced distinct char16_t and char32_t types in C++11 (see here). ---------------------------------------------------------------------- tetrastes · 2022-01-12 09:05 Just interesting and having no time to read sources right now: if at unix we have to use clib UnicodeStringManager (uses cwstring), there is overhead converting 2-byte UnicodeChar to 4-byte wchar_t? ---------------------------------------------------------------------- tetrastes · 2022-01-12 10:05 Quote from: engkin on January 11, 2022, 11:37:44 pmRight, and that is what raymond wants. By the way, it seems that text-mode Free Pascal IDE (fp) at linux does what he wants, at least at my Debian 10 with fpc 3.2.0. It uses CP437 (though not 850). Opening my sources I see pseudo graphics instead of UTF8 letters, and Tools -> Ascii table shows the CP437 table. I don't understand this decision. I think it needs some efforts not to use UTF8 at UTF8 system.  Not saying that it makes fp rather useless, if you need non-ASCII output from your apps. ---------------------------------------------------------------------- raymond · 2022-01-12 10:24 Thank you for all the help and comments. Having used the logical simplicity and elegance of code page 850 for at least two decades, it's easy to be deceived by UTF'8', and difficult to understand why a value between hex 00 and hex ff gets to be compiled into 2 bytes. It would be tempting to convert back  to 850 but there is the issue of Linux (Mint) environment variables, let alone the editor (gedit). Bear in mind that in most 'natural' 'European' language texts there is an unpredictable mixing of '>127' chars and '< 128' chars. It looks as though I will have to fudge a solution for my programming purposes. Thank you once again. ---------------------------------------------------------------------- tetrastes · 2022-01-12 11:01 I am sorry, but I don't understand what is "the logical simplicity and elegance of code page 850", and why it is better than CP437, or 1252, or 8859-1, for example. I think there are may be problems with CP850 at linux, as with other DOS and Windows code pages, so if you want one-byte code  page, try 8859-1 (or 8859-15). They are native for unix, and once upon a time they were default. ---------------------------------------------------------------------- PascalDragon · 2022-01-12 12:53 Quote from: tetrastes on January 12, 2022, 10:05:31 amJust interesting and having no time to read sources right now: if at unix we have to use clib UnicodeStringManager (uses cwstring), there is overhead converting 2-byte UnicodeChar to 4-byte wchar_t? Not in the sense you think, because essentially no POSIX API expects wchar_t. Thus the TUnicodeStringManager never needs to convert from UTF-16 to UTF-32. But it needs to convert from UTF-16 to UTF-8 (assuming the system is set to UTF-8 which is essentially the default nowadays). Quote from: tetrastes on January 12, 2022, 11:05:14 amQuote from: engkin on January 11, 2022, 11:37:44 pmRight, and that is what raymond wants. By the way, it seems that text-mode Free Pascal IDE (fp) at linux does what he wants, at least at my Debian 10 with fpc 3.2.0. It uses CP437 (though not 850). Opening my sources I see pseudo graphics instead of UTF8 letters, and Tools -> Ascii table shows the CP437 table. I don't understand this decision. I think it needs some efforts not to use UTF8 at UTF8 system.  Not saying that it makes fp rather useless, if you need non-ASCII output from your apps. Free Vision currently simply does not support Unicode. There is work underway (and more or less already finished) that extends Free Vision and thus also the text mode IDE with Unicode support. Quote from: raymond on January 12, 2022, 11:24:37 amThank you for all the help and comments. Having used the logical simplicity and elegance of code page 850 for at least two decades, it's easy to be deceived by UTF'8', and difficult to understand why a value between hex 00 and hex ff gets to be compiled into 2 bytes. It would be tempting to convert back  to 850 but there is the issue of Linux (Mint) environment variables, let alone the editor (gedit). Bear in mind that in most 'natural' 'European' language texts there is an unpredictable mixing of '>127' chars and '< 128' chars. It looks as though I will have to fudge a solution for my programming purposes. Thank you once again. There needs to be some kind of encoding, because Unicode supports > $FF characters. So the code points < $80 are used as-is (which is just ASCII), but anything greater (in the first Byte) is used to encode how many further Bytes follow as UTF-8 supports code points with 1 to 4 Byte. This way the full range of Unicode code points can be used without wasting space for many 0s like UTF-16 and UTF-32 need (assuming the text in question is mainly ASCII). ---------------------------------------------------------------------- SymbolicFrank · 2022-01-12 13:57 Now only if there was a Unicode variant where each glyph took up the same amount of space... UTF-32 sounds great, but that is still not fixed length, because of combining characters and grapheme clusters. Also, while using it seems wasteful to users of the Latin alphabet, because most glyphs take up just 7 bits, many other language users think it wastes space because only 21 bits are used. They need multiple 32-bit 'chars' to create the grapheme clusters they use for written language. And the information density of UTF-8 is even worse. But because it is now the standard, don't hold your breath waiting for a new UTF-32 that actually uses all those bits to code for all the possible glyphs / grapheme clusters. So, overall, Unicode does work, but just barely. Technically it's a jack of all trades, master of none. Best example: sorting. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-13 10:18 I made a nice example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 'Ḝ' <> 'Ḝ' <> 'Ḝ' They might look the same, but aren't. Depending on your sort order and how it is sorted, they might end up together, but they might not. Unicode and the applications that display the resulting glyphs are strangely inconsistent as to what can be combined into what. ---------------------------------------------------------------------- engkin · 2022-01-13 16:49 Quote from: SymbolicFrank on January 13, 2022, 11:18:02 amI made a nice example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i 'Ḝ' <> 'Ḝ' <> 'Ḝ' They might look the same, but aren't. Depending on your sort order and how it is sorted, they might end up together, but they might not. Unicode and the applications that display the resulting glyphs are strangely inconsistent as to what can be combined into what. To compare them,try using: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses  unicodedata; function Test:boolean;var  s,o:array of string;  i:integer;begin  Result:=False;  s:=['Ḝ','Ḝ','Ḝ'];  SetLength(o,Length(s));  for i:=Low(s) to High(s) do  begin    o[i]:=NormalizeNFD(s[i]);    if o[Low(s)]<>o[i] then      exit;  end;  Result:=True;end; ---------------------------------------------------------------------- tetrastes · 2022-01-14 12:07 Quote from: PascalDragon on January 12, 2022, 01:53:54 pmQuote from: tetrastes on January 12, 2022, 10:05:31 amJust interesting and having no time to read sources right now: if at unix we have to use clib UnicodeStringManager (uses cwstring), there is overhead converting 2-byte UnicodeChar to 4-byte wchar_t? Not in the sense you think, because essentially no POSIX API expects wchar_t. Thus the TUnicodeStringManager never needs to convert from UTF-16 to UTF-32. But it needs to convert from UTF-16 to UTF-8 (assuming the system is set to UTF-8 which is essentially the default nowadays). It seemed strange to me, that there is unused type, so I looked in cwstring.pp, and found some functions, POSIX or not, using wchar_t. And as they are used in cwstring.pp, there is overhead in the sense I think. For example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function wcscoll (__s1:pwchar_t; __s2:pwchar_t):cint;cdecl;external clib name 'wcscoll'; ... function CompareWideString(const s1, s2 : WideString; Options : TCompareOptions) : PtrInt;{$if not(defined (aix) and defined(cpupowerpc32))}  var    hs1,hs2 : UCS4String;    us1,us2 : WideString;      begin    { wcscoll interprets null chars as end-of- string -> filter out }    if coIgnoreCase in Options then      begin    us1:=UpperWideString(s1);      us2:=UpperWideString(s2);      end       else            begin         us1:=s1;      us2:=s2;      end;   hs1:=WideStringToUCS4StringNoNulls(us1);    hs2:=WideStringToUCS4StringNoNulls(us2);    result:=wcscoll(pwchar_t(hs1),pwchar_t(hs2));  end;{$else}  { AIX/PPC32 has a 16 bit wchar_t }  where WideStringToUCS4StringNoNulls converts UTF-16 string to UTF-32, naturally. ---------------------------------------------------------------------- Thaddy · 2022-01-14 12:13 Note it is a misunderstanding that UTF16 char is always two bytes. It can be 4 too. Only UCS2 is always two bytes. Problem is that Delphi declared it before the UTF16 standard was enhanced and FPC adheres to the Delphi declaration. So wchar_t  refers to UTF8, UTF16 and UTF32. Sooner or later this will lead to problems. Note UCS2 equalled UTF16, but nowadays it is simply a 2 byte subset of UTF16. The whole issue is surrogate pairs, which can make it size 4. https://en.wikipedia.org/wiki/UTF-16 clearly states UTF16 is a variable length encoding. ---------------------------------------------------------------------- tetrastes · 2022-01-14 12:46 cwstring.pp knows that. There is code in WideStringToUCS4StringNoNulls for converting surrogate pairs. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-14 15:55 Quote from: engkin on January 13, 2022, 05:49:44 pmTo compare them,try using: Very interesting, thanks! ---------------------------------------------------------------------- PascalDragon · 2022-01-14 20:01 Quote from: tetrastes on January 14, 2022, 01:07:21 pmQuote from: PascalDragon on January 12, 2022, 01:53:54 pmQuote from: tetrastes on January 12, 2022, 10:05:31 amJust interesting and having no time to read sources right now: if at unix we have to use clib UnicodeStringManager (uses cwstring), there is overhead converting 2-byte UnicodeChar to 4-byte wchar_t? Not in the sense you think, because essentially no POSIX API expects wchar_t. Thus the TUnicodeStringManager never needs to convert from UTF-16 to UTF-32. But it needs to convert from UTF-16 to UTF-8 (assuming the system is set to UTF-8 which is essentially the default nowadays). It seemed strange to me, that there is unused type, so I looked in cwstring.pp, and found some functions, POSIX or not, using wchar_t. And as they are used in cwstring.pp, there is overhead in the sense I think. For example: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function wcscoll (__s1:pwchar_t; __s2:pwchar_t):cint;cdecl;external clib name 'wcscoll'; ... function CompareWideString(const s1, s2 : WideString; Options : TCompareOptions) : PtrInt;{$if not(defined (aix) and defined(cpupowerpc32))}  var    hs1,hs2 : UCS4String;    us1,us2 : WideString;      begin    { wcscoll interprets null chars as end-of- string -> filter out }    if coIgnoreCase in Options then      begin    us1:=UpperWideString(s1);      us2:=UpperWideString(s2);      end       else            begin         us1:=s1;      us2:=s2;      end;   hs1:=WideStringToUCS4StringNoNulls(us1);    hs2:=WideStringToUCS4StringNoNulls(us2);    result:=wcscoll(pwchar_t(hs1),pwchar_t(hs2));  end;{$else}  { AIX/PPC32 has a 16 bit wchar_t }  where WideStringToUCS4StringNoNulls converts UTF-16 string to UTF-32, naturally. *shrugs* That's how it is. But UnicodeString isn't used that often on Unix anyway. Lazarus uses UTF-8 and that goes through the AnsiString routines. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-15 12:38 It seems that many applications, like Word and OpenOffice only render combining characters correctly when there already is another character that looks the same. They can only display a single glyph at any location. While a browser stacks them with an offset. In other words, what you see depends on the rendering engine used. Is there one that actually combines them as is intended? LaTeX? But that also means, that each application (depending on the Unicode table and rendering engine used), has it's own Unicode subset, that might or might not look the same as any of the others when put on the screen or printer. And I think the best way to compare Unicode chars would be to split them out into the base shape and the separate, combining characters. Then again, that would require expanding those, as there are "attachments" not covered by them. Ok, that would make it even harder to determine how much storage space you need to reserve. Actually, the best way to store them would probably be like Huffman encoding (7-zip etc). Expand each character you come across, make a list and only store the index in your string or table. That way, they will all be the same when they look the same and fit in a single, 32-bit value. And always display them multi-pass, the parts on top of each other. I think that's how Unicode should have been. On the other hand, that won't fix the sorting problem. You still need a separate table for each language. Although you can limit those to only the base shape and attachments that make a difference. ---------------------------------------------------------------------- engkin · 2022-01-15 14:17 Quote from: SymbolicFrank on January 15, 2022, 01:38:06 pmIt seems that many applications, like Word and OpenOffice only render combining characters correctly when there already is another character that looks the same. They can only display a single glyph at any location. While a browser stacks them with an offset. In other words, what you see depends on the rendering engine used. Sounds interesting. Any specific example. Quote from: SymbolicFrank on January 15, 2022, 01:38:06 pmIs there one that actually combines them as is intended? LaTeX? There used to be a "layout engine" called ICU, it was abandoned and replaced with HarfBuzz. Both are open source, which made it easy to include in Firefox and Android. Quote from: SymbolicFrank on January 15, 2022, 01:38:06 pmBut that also means, that each application (depending on the Unicode table and rendering engine used), has it's own Unicode subset, that might or might not look the same as any of the others when put on the screen or printer. Depends on their implementation, bugs, and settings. The same application/OS might have an extension to support, say, Complex Scripts. By default the extension is not installed or enabled because it has a small efficiency impact. You need to install/activate it for the rendering engine to give you the expected results. Quote from: SymbolicFrank on January 15, 2022, 01:38:06 pmAnd I think the best way to compare Unicode chars would be to split them out into the base shape and the separate, combining characters. Then again, that would require expanding those, as there are "attachments" not covered by them. Ok, that would make it even harder to determine how much storage space you need to reserve. Actually, the best way to store them would probably be like Huffman encoding (7-zip etc). Expand each character you come across, make a list and only store the index in your string or table. That way, they will all be the same when they look the same and fit in a single, 32-bit value. And always display them multi-pass, the parts on top of each other. I think that's how Unicode should have been. On the other hand, that won't fix the sorting problem. You still need a separate table for each language. Although you can limit those to only the base shape and attachments that make a difference. You might be up to something here, but without testing an actual code, it is hard to say. ---------------------------------------------------------------------- SymbolicFrank · 2022-01-15 15:12 Quote from: engkin on January 15, 2022, 03:17:07 pmSounds interesting. Any specific example. There are at least 3 plain 'E's in the default OpenOffice character set, one with a cedilla, one with a (rounded) breve and one with both a cedilla and a (pointy) breve. There is also a breve and cedilla as combining characters.So, at first, I expected I could make at least 6 characters that looked the same But from all those, I could only combine three. I didn't try other character sets, but some googling showed Word to do the same. What happens is that the combining chars become small, separate chars that sit just to the right. Like you see here (if your browser displays that the same as mine). Even when they should go in the middle. On the other hand, in a browser you can make Zalgo text, which stacks them all on top and below one another. QuoteYou might be up to something here, but without testing an actual code, it is hard to say. My first idea was to turn every glyph into separate strokes, like with Chinese. That's definitely the simplest way to do it, but font designers wouldn't stand for it (although they might not like many different 'E's that all might have to be different, either). And it would be very hard to describe all letters like that, although in a sense that is what vector fonts already do. Computers think spotting patterns is hard. Perhaps with an AI. The code you posted is interesting, they seem to analyze the separate parts (through tables and the amount of strokes). It is much easier to spot the base forms and combining marks, because we have a list. (Although not all the attachments exist as combining marks, like the rounded breve.) But that still results in glyphs that will have different byte lengths, especially the Arabic and Asian ones. If you put those in something like a TStringList (one line for each glyph), you can simply use the index to represent it. But, you would have to store that table (the StringList) in each file as well. Which is also how compressed archive files work: you take the first byte, put it in the list and replace it with a '1'. Take the next and replace it with a '2'. Etc. And you attach the table to the end of the file. Then again, archives use a binary tree to store the information, which is how they compress them. Which would be the UTF-8 equivalent. And at the other end of he scale you can use a fixed 32-bit index. That would result in files like this: - UTFX-8, UTFX-16 and UTFX-32, where the total amount of separate glyphs fits in that many bits. - UTFX-Z, (like: zipped), where each glyph takes up a different amount of bits. Like, 1 bit for lower case 'e'. Or even better: the word 'the'. Or the Japanese / Chinese equivalent. You would have to expand that into one of the other formats to be able to process it. But it would be very compact to store and transmit. Like with the other UTF files, you start with a 16-bit value that specifies the format. Followed with a 64-bit offset to the table used. Sorting and comparing becomes much easier, although you have to compare the values in the table, not the index value (which will be different for each file). In the end, that's almost he same as we do now. Only the symbol table isn't static. Which I highly doubt it could ever be as long as we don't all use the same language. ---------------------------------------------------------------------- PascalDragon · 2022-01-15 17:42 Quote from: SymbolicFrank on January 15, 2022, 01:38:06 pmActually, the best way to store them would probably be like Huffman encoding (7-zip etc). Expand each character you come across, make a list and only store the index in your string or table. That way, they will all be the same when they look the same and fit in a single, 32-bit value. And always display them multi-pass, the parts on top of each other. Please note that you're simplifying things. Not only do fonts support ligatures, meaning two characters next to each other might be rendered differently than if they are apart (and this can be used for rather insane things, like generating charts by justing writing text or especially highlighting symbols of programming languages). But then there are emojis which make use of the ZERO WIDTH NON JOINER to combine different characters to generate a new glyph. E.g. the rainbow and a flag to generate the rainbow flag or the gender modifiers for the various gendered emojis or the different color variations of the family and pairing emojis (though most of this is strongly related to ligatures and its underlying mechanisms). ---------------------------------------------------------------------- SymbolicFrank · 2022-01-16 12:20 Yes, I know that. But it makes no difference. In many languages, the "unit" of writing is not a letter, but a word. They can consist of smaller units, but they might not. Many such languages do both: words that are a single glyph and words that are combined. What you (the programmer) are interested in, are the individual glyphs. If that is a complex picture that exists by itself, that is fine. If it consists of different glyphs that are "glued" together with zero-width spaces, that works as well. Kerning between different glyphs can be calculated (it is part of most font files). More interesting are ones you can write different ways, like 'ẞ' (equal to 'SS'). Or in languages like Korean, where you can use two different alphabets (hangeul and hanja), mostly interchangeable. That can be done with either a table or multiple entries, although the latter would always print the same glyph, no matter which one was input. Still, it won't be a fast or easy project. But in the long term it would beat Unicode on all fronts. And you could actually parse it Edit: also interesting is Arabic, where most glyphs are connected together without spaces. But (as far as I know), while most 'letters' are individual units, there are also long 'sentences' that are treated as a single glyph. So, spaces are not the primary separators. Even more so, I expect that parts of those long sequences could be individual units in different contexts. Still, that isn't much different than glyphs that consist of multiple letters and combining marks. Also, I would be very interested to hear from people who don't use a variant of Latin, how it works for their language. Because I expect that most of those are 'tacked on afterwards'.