use special character in ncurses mvaddchar lazarus / 32-Linux ---------------------------------------------------------------------- atroesch · 2023-02-15 12:12 First hello, iam new here how do i use special characters ( ö ä ü) in ncurses mvaddchar function? there is still a side explain it for C... http://www.melvilletheatre.com/articles/ncurses-extended- characters/index.html somebody knows the right / similar pascal syntax? if somebody got a better idea to use these characters in screen buffering, please let me know ---------------------------------------------------------------------- Bogen85 · 2023-02-15 12:38 Quote from: atroesch on February 15, 2023, 01:12:23 pmFirst hello, iam new here Welcome to the Lazarus/FPC forum! Quote from: atroesch on February 15, 2023, 01:12:23 pmhow do i use special characters ( ö ä ü) in ncurses mvaddchar function? there is still a side explain it for C... http://www.melvilletheatre.com/articles/ncurses-extended- characters/index.html somebody knows the right / similar pascal syntax? if somebody got a better idea to use these characters in screen buffering, please let me know I might be able to help out, do you have some example code you can share that your are trying to get this working with? ---------------------------------------------------------------------- Bogen85 · 2023-02-15 13:10 Quote from: atroesch on February 15, 2023, 01:12:23 pmhow do i use special characters ( ö ä ü) in ncurses mvaddchar function? Do you have a working example for C? That would help in figuring out the correct Pascal code to do the same thing. ---------------------------------------------------------------------- atroesch · 2023-02-15 13:20 scroll down the page i posted. at the end there is an C example... with addch but still does the same... ---------------------------------------------------------------------- Bogen85 · 2023-02-15 13:52 Quote from: atroesch on February 15, 2023, 02:20:41 pmscroll down the page i posted. at the end there is an C example... with addch but still does the same... I saw that, but you are talking about mvaddchar... The more you can provide what you are specifically trying to do, the better. ---------------------------------------------------------------------- Thaddy · 2023-02-15 14:09 If I look at the alt_charset_table: The pascal syntax for that is simply chr(); But I doubt that ncurses did not change to UTF in the mean time..... That page is 13 years old. ---------------------------------------------------------------------- tetrastes · 2023-02-15 23:37 Quote from: atroesch on February 15, 2023, 01:12:23 pmhow do i use special characters ( ö ä ü) in ncurses mvaddchar function? FPC has ncurses package with good examples, and it works. But don't use mvaddch, it works only for ASCII chars at UTF-8. Use mvaddstr: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$mode objfpc} uses ncurses, initc; procedure setlocale(cat : integer; p : pchar); cdecl; external clib; const LC_ALL = 6; begin setlocale(LC_ALL, ''); initscr(); mvaddstr(1,3, curses_version); mvaddstr(7,3, PChar('ö')); mvaddstr(7,5, PChar('ä')); mvaddstr(7,7, PChar('ü')); mvaddch(9,3, $F6); // ö - does not work mvaddstr(9,7, PChar('Output of ö by mvaddch(y,x, $F6) does not work')); mvaddch(11,3, ord('o')); // works mvaddstr(15,7, PChar('Press any key to exit')); getch(); endwin();end. EDIT: the source is in UTF-8, of course. Quote from: atroesch on February 15, 2023, 01:12:23 pmthere is still a side explain it for C... http://www.melvilletheatre.com/articles/ncurses-extended- characters/index.html The code for generating the A_ALTCHARSET table from that link does not work at UTF-8 console, which was expected however. ---------------------------------------------------------------------- dbannon · 2023-02-16 03:18 I am playing with a similar problem. Quote from: tetrastes on February 16, 2023, 12:37:43 amCode: [Select] mvaddch(9,3, $F6); // ö - does not work Is that because $F6 is not a valid utf8 character (its Unicode char). The UTF8 code for ö is $C3B6, two bytes. I am unsure of the correct way to convert such a value to a UTF8 character but this works (without any code page fiddling) - Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Buff : string;....Buff := char($C3) + char($B6); I image mvaddch() would send only the first byte and even if it took a widechar then bad things would happen if you feed it a three or four byte utf8 char. I expect there is a library function to break up a hex value and convert it to char bytes but I cannot find it. So, coding up my own I guess. Davo ---------------------------------------------------------------------- atroesch · 2023-02-16 09:45 thank you. so i will use mvaddstr ---------------------------------------------------------------------- tetrastes · 2023-02-16 12:13 Quote from: dbannon on February 16, 2023, 04:18:08 amI expect there is a library function to break up a hex value and convert it to char bytes but I cannot find it. So, coding up my own I guess. There is https://lazarus- ccr.sourceforge.io/docs/lazutils/lazutf8/unicodetoutf8.html: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i uses ncurses, initc, LazUTF8;. . . mvaddstr(8,3, PChar(UnicodeToUTF8($F6))); mvaddstr(8,5, PChar(UnicodeToUTF8($1D575))); Just in case, this code gives interesting results: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var c: chtype;. . . c := $F6; mvaddchstr(11,3, @c); // This works c := $400F6; mvaddchstr(11,5, @c); c := $3400F6; mvaddchstr(11,7, @c); mvaddch(13,3, $6F); mvaddch(13,5, $4006F); mvaddch(13,7, $34006F); ---------------------------------------------------------------------- dbannon · 2023-02-16 23:04 Quote from: tetrastes on February 16, 2023, 01:13:05 pmQuote from: dbannon on February 16, 2023, 04:18:08 amI expect there is a library function to break up a hex value and convert it to char bytes but I cannot find it. So, coding up my own I guess. There is https://lazarus- ccr.sourceforge.io/docs/lazutils/lazutf8/unicodetoutf8.html: Hmm, I could do with something that converts a number, eg Numb : longint = 50102; to its equivalent UTF8 char in a string. Its an easy 10 or so lines of code but I am sure there is a better way. https://lazarus-ccr.sourceforge.io/docs/lazutils/lazutf8/index.html is not very helpful, seems to think UTF8LeftStr is the only function in there. Quote.... Just in case, this code gives interesting results: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var c: chtype;. . . c := $F6; mvaddchstr(11,3, @c); // This works c := $400F6; mvaddchstr(11,5, @c); c := $3400F6; mvaddchstr(11,7, @c); mvaddch(13,3, $6F); mvaddch(13,5, $4006F); mvaddch(13,7, $34006F); Yes, as you say interesting. I am not sure whats happening there, maybe triggering some side effect in mvaddch() ? ncurses is pretty old technology. And, just what is a "chtype" ? Is that some sort of UTF8 character type ? My system does not know what it is. I use string[4] which, of course would require mvaddchstr(). Davo ---------------------------------------------------------------------- tetrastes · 2023-02-17 09:06 Quote from: dbannon on February 17, 2023, 12:04:05 amQuote from: tetrastes on February 16, 2023, 01:13:05 pmQuote from: dbannon on February 16, 2023, 04:18:08 amI expect there is a library function to break up a hex value and convert it to char bytes but I cannot find it. So, coding up my own I guess. There is https://lazarus- ccr.sourceforge.io/docs/lazutils/lazutf8/unicodetoutf8.html: Hmm, I could do with something that converts a number, eg Numb : longint = 50102; to its equivalent UTF8 char in a string. Its an easy 10 or so lines of code but I am sure there is a better way. It seems that I don't understand what do you need. Do not functions from that link do that? Quote from: dbannon on February 17, 2023, 12:04:05 amhttps://lazarus- ccr.sourceforge.io/docs/lazutils/lazutf8/index.html is not very helpful, seems to think UTF8LeftStr is the only function in there. Click there at [Procedures and functions] -> https://lazarus- ccr.sourceforge.io/docs/lazutils/lazutf8/index-5.html or [Index] -> https://lazarus-ccr.sourceforge.io/docs/lazutils/lazutf8/index-8.html. Quote from: dbannon on February 17, 2023, 12:04:05 amYes, as you say interesting. I am not sure whats happening there, maybe triggering some side effect in mvaddch() ? ncurses is pretty old technology. I think it is intentionally, high bytes of parameter are used for such effects. I'll read ncurses docs when I have a time. QuoteAnd, just what is a "chtype" ? It's defined in fpc's packages/ncurses/src/ncurses.pp: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i type pchtype = ^chtype; chtype = culong; And culong in rtl/unix/ctypes.inc: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i {$ifdef cpu64} clong = int64; pclong = ^clong; cslong = int64; pcslong = ^cslong; culong = qword; pculong = ^culong;{$else} clong = longint; pclong = ^clong; cslong = longint; pcslong = ^cslong; culong = cardinal; pculong = ^culong;{$endif} ---------------------------------------------------------------------- Thaddy · 2023-02-17 09:14 Quote from: dbannon on February 16, 2023, 04:18:08 amI expect there is a library function to break up a hex value and convert it to char bytes but I cannot find it. So, coding up my own I guess. Davo Plz Davo, are you playing the ill-informed on purpose? It is a value (sick), not a representation in hex. That is silly. Also note that the type helper in sysutils can split a word value into two bytes. (But not the full range 1...4 of UTF8 without some tinkering, it works for UCS2, though, a subset of UTF16, and actually its predecessor) ---------------------------------------------------------------------- dbannon · 2023-02-18 03:07 Yes Thaddy, I value your concern about my mental state, its a good assessment ! Sorry, atroesch, to seem to hijack your thread, not intentional. My plan was to store the numerical value of a UTF8 character in config files. Its not a good idea (IMHO) to have actual UF8 characters in source/config because, Debian, those guardians of code quality, flag any such file as suspect, even, for example, when accented characters appear in an authors name. Sigh ... And it all worked quite nicely until I started testing .... Four byte UTF8 characters are bigger, numerically, than a longint and functions like inttostrdef() do not cope with the reverse process. So, I will have to record and check them as a string, not that much harder. Davo ---------------------------------------------------------------------- Bogen85 · 2023-02-18 03:53 A single display character can be more than one unicode character. See this thread where the OP was running into trouble because of a character with a diacritical marker: https://forum.lazarus.freepascal.org/index.php/topic,62150.msg470031.h tml One can't reliably in every situation treat unicode characters as individually characters. I've since updated the function that breaks a utf8 string up into into separate strings, each being one display character. (Ligatures and emoticons are not accounted for though). Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program unicode_ex2; {$mode objfpc}{$h+}{$codepage utf8} uses sysutils, types; function utf8DisplayedChars (const str_in: string; const withCombiningDiacriticals: boolean = true): TStringDynArray; procedure primary (const len: integer; offset: integer = 1; n_chars: integer = 0; remaining: integer = 0); procedure secondary (const n_bytes: integer); begin result[n_chars] := copy(str_in, offset, n_bytes); inc(offset, n_bytes); dec(remaining, n_bytes); inc(n_chars); end; begin setlength(result, len); remaining := len; while remaining > 0 do secondary(Utf8CodePointLen(@str_in[offset], remaining, withCombiningDiacriticals)); setlength(result, n_chars); end; begin result := []; primary(length(str_in)); end; const boo: string = 'ábcdéfghíÁ̊ÅÁǺÁwowe̊́!é'; var str: string; begin writeln(boo); for str in utf8DisplayedChars(boo) do writeln(format('%s: %d', [str, length(str)]));end. The lengths of those "characters" in that string range from 1 to 5 bytes. Code: Text [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i $ ./bin/examples/unicode_ex2ábcdéfghíÁ̊ÅÁǺÁwowe̊́!éá: 2b: 1c: 1d: 1é: 2f: 1g: 1h: 1í: 2Á̊: 5Å: 3Á: 3Ǻ: 5Á: 2w: 1o: 1w: 1e̊́: 5!: 1é: 3 So no single integer type is likely sufficient for displayed unicode characters, depending on the number of diacritical markers involved. Granted, some of the 5 byte sequences on this forum web page might show the diacritical marker over the colon, but that is not what showed up in my terminal output. (See attached image). ---------------------------------------------------------------------- Bogen85 · 2023-02-18 04:09 And converting the example to ncurses, it works when using mvaddstr: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i program unicode_ex2; {$mode objfpc}{$h+}{$codepage utf8}uses initc, ncurses, sysutils, types; procedure setlocale (cat: integer; p: pChar); cdecl; external clib;const LC_ALL = 6; function utf8DisplayedChars (const str_in: string; const withCombiningDiacriticals: boolean = true): TStringDynArray; procedure primary (const len: integer; offset: integer = 1; n_chars: integer = 0; remaining: integer = 0); procedure secondary (const n_bytes: integer); begin result[n_chars] := copy(str_in, offset, n_bytes); inc(offset, n_bytes); dec(remaining, n_bytes); inc(n_chars); end; begin setlength(result, len); remaining := len; while remaining > 0 do secondary(Utf8CodePointLen(@str_in[offset], remaining, withCombiningDiacriticals)); setlength(result, n_chars); end; begin result := []; primary(length(str_in)); end; const boo: string = 'ábcdéfghíÁ̊ÅÁǺÁwowe̊́!é'; var str: string; y: integer; begin setlocale(LC_ALL, ''); initscr(); mvaddstr(1, 3, curses_version); mvaddstr(3, 3, pChar(boo)); y := 5; for str in utf8DisplayedChars(boo) do begin mvaddstr(y, 2, pChar(format('%s: %d', [str, length(str)]))); inc(y); end; mvaddstr(y + 2, 7, pChar('Press any key to exit')); getch(); endwin();end. Resulting in: Code: Text [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ncurses 6.2.20210508 ábcdéfghíÁ̊ÅÁǺÁwowe̊́!é á: 2 b: 1 c: 1 d: 1 é: 2 f: 1 g: 1 h: 1 í: 2 Á̊: 5 Å: 3 Á: 3 Ǻ: 5 Á: 2 w: 1 o: 1 w: 1 e̊́: 5 !: 1 é: 3 Press any key to exit (Also see attached) ---------------------------------------------------------------------- dbannon · 2023-02-18 05:54 Quote from: Bogen85 on February 18, 2023, 04:53:14 am The lengths of those "characters" in that string range from 1 to 5 bytes. I find that rather surprising, by definition, a UTF8 char is made up of 1 to 4 bytes. What you are seeing there is a sequence of characters, one overwriting the other. Thats not, as I understand it, UTF8 Try pasting one of those 5 byte characters and then backspacing, you remove the overtype with the first backspace ! (and no one is more surprised at that than me, maybe an artifact of the forum's means of displaying it ?) On the other hand, the real UTF8 Å is a single character, two bytes, $C385. Or the one with the acute, Ǻ, $C7BA Davo ---------------------------------------------------------------------- Bogen85 · 2023-02-18 13:09 Quote from: dbannon on February 18, 2023, 06:54:31 amQuote from: Bogen85 on February 18, 2023, 04:53:14 am The lengths of those "characters" in that string range from 1 to 5 bytes. I find that rather surprising, by definition, a UTF8 char is made up of 1 to 4 bytes. What you are seeing there is a sequence of characters, one overwriting the other. Thats not, as I understand it, UTF8 Try pasting one of those 5 byte characters and then backspacing, you remove the overtype with the first backspace ! (and no one is more surprised at that than me, maybe an artifact of the forum's means of displaying it ?) On the other hand, the real UTF8 Å is a single character, two bytes, $C385. Or the one with the acute, Ǻ, $C7BA Some combinations go even higher than 5 bytes, as you can have any number of diacritical marks. Yes, this was surprising to me to. But this caused problems for the OP of https://forum.lazarus.freepascal.org/index.php/topic,62150.msg47003 1.html until I showed the OP about the diacritical marks. Take these 3 character combinations: A ́ ̊ A ̊ ́ Put them next to each and remove the spaces and display them. I'm not doing anything special for my terminal to display those as a single combined character. A proper UTF-8 display application will combine them all into one display character. If you iterate a UTF-8 string and simply break it apart by 1-4 byte code points you will separate the diacritical markers from the characters they are being combined with, and you will then have problems displaying them correctly. See https://en.wikipedia.org/wiki/Combining_Diacritical_Marks You can have a base character followed by some (any?) number of combining diacritical marks. So while it is true that a UTF-8 endpoint (incorrect term?) is 1-4 bytes, a single displayed character can be more. Therefore it is better to use UTF-8 strings to store single displayed characters, because a single displayed character can be 1 or more bytes, and is not limited to a 4 byte maximum. ---------------------------------------------------------------------- Bogen85 · 2023-02-18 13:37 changing the constant string of characters to: Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i const boo: string = 'oauo̥̊ḁ̊ů̥öäüö̥̊ḁ̈̊ü̥̊'; Results in (:changed to |): Code: Text [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i oauo̥̊ḁ̊ů̥öäüö̥̊ḁ̈̊ü̥̊o | 1a | 1u | 1o̥̊ | 5ḁ̊ | 5ů̥ | 5ö | 2ä | 2ü | 2ö̥̊ | 6ḁ̈̊ | 6ü̥̊ | 6 On my terminal the characters and marks were not separated. So that is an example with 6 byte "display characters" (which are multiple UTF-8 multi byte characters) I'm sure one could go higher than 6... Point being, a "display character" can be the combination of one or more multi-byte characters). It is the "or more" which for me means I will never assume some fixed byte size for displayable characters, and will treat displayable characters as UTF-8 strings containing one or more multi-byte unicode endpoints. Which works with the ncurses example as well. See attached screenshots for both. ---------------------------------------------------------------------- Bogen85 · 2023-02-18 14:13 I'll break it down even further... Since the forum webpage is not always combining correctly, I've also included a screen shot. Code: Text [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i oauo̥̊ḁ̊ů̥öäüö̥̊ḁ̈̊ü̥̊ 1:1: o <-- first length is number of bytes, second length is number of unicode characters6F - o 1:1: a61 - a 1:1: u75 - u 5:3: o̥̊ <-- 5 bytes, made up of 3 Unicode characters6F CC A5 CC 8A - o.... 5:3: ḁ̊ <-- 5 bytes, made up of 3 Unicode characters61 CC A5 CC 8A - a.... 5:3: ů̥75 CC A5 CC 8A - u.... 2:1: öC3 B6 - .. 2:1: äC3 A4 - .. 2:1: üC3 BC - .. 6:3: ö̥̊C3 B6 CC A5 CC 8A - ...... 6:3: ḁ̈̊C3 A4 CC A5 CC 8A - ...... 6:3: ü̥̊ <-- 6 bytes, made up of 3 Unicode charactersC3 BC CC A5 CC 8A - ...... ---------------------------------------------------------------------- Bogen85 · 2023-02-18 15:19 Quote from: dbannon on February 18, 2023, 06:54:31 amQuote from: Bogen85 on February 18, 2023, 04:53:14 am The lengths of those "characters" in that string range from 1 to 5 bytes. I find that rather surprising, by definition, a UTF8 char is made up of 1 to 4 bytes. What you are seeing there is a sequence of characters, one overwriting the other. Thats not, as I understand it, UTF8 And the overwriting (combining) is part of the standard. See: https://www.freepascal.org/docs- html/rtl/system/utf8codepointlen.html Which I already provided the link for: https://en.wikipedia.org/wiki/Combining_Diacritical_Marks And for utf8codepointlen: See the following: https://gitlab.com/freepascal.org/fpc/source/- /blob/ffa14ee4485dbb452fe4a89b9c7a6340ea359c7f/rtl/inc/generic.inc#L11 47 https://gitlab.com/freepascal.org/fpc/source/- /blob/ffa14ee4485dbb452fe4a89b9c7a6340ea359c7f/rtl/inc/generic.inc#L10 89 ---------------------------------------------------------------------- Bogen85 · 2023-02-18 15:50 From https://www.freepascal.org/docs- html/rtl/system/utf8codepointlen.html : QuoteDescription Utf8CodePointLen returns the length of the UTF-8 codepoint starting at the beginning of P. It will look at at most MaxLookAhead bytes to do create this codepoint. If IncludeCombiningDiacriticalMarks is true, combining diacritical marks trailing the first codepoint (which itself can also be such a mark) will be considered to be part of the codepoint. While technically that might not be part of the standard (considering the combined codepoints as a single codepoint) it is part of the of the standard that multiple codepoints can be combined to form a single displayed character with marks on it. ---------------------------------------------------------------------- Bogen85 · 2023-02-18 15:55 All of this may have seemed like a deviation from the topic "use special character in ncurses mvaddchar". However, the point of the deviation is to point out that displaying a special character with a some "add character" function might not work for a lot of unicode combinations (that are following the standard) of characters. This is not a problem specific to ncurses. It is a problem whenever a unicode string is broken up into subparts, where those subparts are to be kept intact, as far as the displayed characters they represent. ---------------------------------------------------------------------- dbannon · 2023-02-19 11:56 While not disputing anything you say here Bogen85, the point is that you are NOT displaying a UTF8 character under these circumstances. (Its decomposed Unicode ?). While it might look similar to a UTF8 character, the bytes stored in a string has no relationship to the bytes stored when a real UTF8 character is involved. So, if you know that some component, some display system, something can display UTF8, you cannot assume that it can display the overwritten method as well. And visa versa. As you noted, the forum does a less than perfect job of displaying overwritten characters but it does a perfect job of displaying UTF8 characters. My guess is that what we see with overwritten characters may be browser (or webkit) dependent. I tend to use https://www.utf8-chartable.de as my reference to UTF8 characters, there you will find all you can imagine (and a few more) without overwriting anything. Davo edit : typo ---------------------------------------------------------------------- Bogen85 · 2023-02-19 12:21 @dbannon, and I'm not disputing what you are saying. I included all I said in the context of displaying unicode characters from a string, especially if one is to take apart said string and display the characters individually. That can cause problems. The point of what I'm trying to saying to say, is if you break up a string into into its displayable characters, you should into account the composite parts (which may be more than one unicode character). Therefore, single character representation (2 bytes, 4 bytes, 8 bytes) is not going to work, the composite parts will need to be strings. If your display system can't display those composite parts correct (not every unicode aware display system is totally compliant), then yes, there is a problem, and it is not always solvable (the exact equivalent character will not always be found). I've worked with individuals who work with language scripts where those languages are not in the unicode standard. As such, they rely on diacritical marker combining that is supported by the unicode standard. While I only recently discovered this "feature" for myself in unicode I'd seen others use it before. As was already mentioned in this topic/thread, string output should be used for individual display characters (which may be a composite of multiple unicode characters), not only for the composite issues, but because what encoding are you to chose for individual characters? utf-8? utf-16? full 32 bit? Quote from: dbannon on February 19, 2023, 12:56:12 pmI tend to use https://www.utf8-chartable.de as my reference to UTF8 characters, there you will find all you can imagine (and a few more) without overwriting anything. That is an extremely small set of unicode characters and displayable combinations. Sure, it is fine for many who are just working with Western European languages, but not much beyond that. ---------------------------------------------------------------------- dbannon · 2023-02-19 22:59 QuoteAs was already mentioned in this topic/thread, string output should be used for individual display characters (which may be a composite of multiple unicode characters), not only for the composite issues, but because what encoding are you to chose for individual characters? utf-8? utf-16? full 32 bit? Definitly agree. If I know I are dealing with only UTF8 (and as a Linux user, a Lazarus user thats an easy choice) one character at a time, then it is Code: Pascal [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i type Tutf8Char = string[4]; Otherwise, its ansistring and the FPC/Lazarus developers are looking after me. QuoteQuote from: dbannon on February 19, 2023, 12:56:12 pmI tend to use https://www.utf8-chartable.de as my reference to UTF8 characters, there you will find all you can imagine (and a few more) without overwriting anything. That is an extremely small set of unicode characters and displayable combinations. Sure, it is fine for many who are just working with Western European languages, but not much beyond that. MONGOLIAN, PHOENICIAN, MEROITIC HIEROGLYPHIC, CHEROKEE, MYANMAR, THAI, ..... In the "control panel", top of each page, third control down, "try other block". Its actually a very large set. And, yes, of course in includes $f09382ba, much loved by first year archeology students. Davo ---------------------------------------------------------------------- Bogen85 · 2023-02-20 01:04 Quote from: dbannon on February 19, 2023, 11:59:50 pmMONGOLIAN, PHOENICIAN, MEROITIC HIEROGLYPHIC, CHEROKEE, MYANMAR, THAI, ..... In the "control panel", top of each page, third control down, "try other block". Its actually a very large set. And, yes, of course in includes $f09382ba, much loved by first year archeology students. I see. Thanks, I over looked that. There are many languages (many is relative) that have less than a few thousand speakers. For the most part they can use existing scripts where the all the needed character combinations are covered by existing Unicode blocks. For some of the languages that are not covered they somethings need to use combining diacritical markers. For instance, the Navajo language (170K speakers?, more than a few thousand) is not in the control blocks on the site you mentioned (at least I did not find one). https://unicode.org/faq/char_combmark.html#12 QuoteThe Navajo-specific question below is also applicable to a wide variety of similar cases. Q: Unicode doesn't contain some of the precomposed characters needed for Navajo and other indigenous languages of the Americas. Will you add them? The way to encode the various Navajo letters with diacritics is with the use of combining marks. For example, Navajo high-toned nasalized vowels: a + ogonek + acute = ( ǫ́ ) and so on for the other vowels.That is also mentioned here: https://itch.io/t/503554/adding-a-navajo-ogonek-to-combining- diacritical-marks-extended-unicode-block ---------------------------------------------------------------------- dbannon · 2023-02-20 07:15 Quote from: Bogen85 on February 20, 2023, 02:04:55 am There are many languages (many is relative) that have less than a few thousand speakers. Sadly, in my country there are many languages spoken by as little as half a dozen people. The Australian Aboriginal langues are both diverse and very, very endangered. I was asked to look at a problem some time ago where a project to record several then endangered languages had its data on a propriety portable disk system that no one had a working drive for any more. Nothing could be done. So, keeping records is important, even if it has to use decomposed characters ! Davo