SelLength incorrect value for text containing characters > $FFFF lazarus / 31-LCL ---------------------------------------------------------------------- fedkad · 2017-11-23 14:06 I think this is a bug. For TEdit or TMemo the SelLength property returns wrong information for selections containing UTF8 characters with codes larger than $FFFF. For example, create a TMemo object and during run time paste the character 𝛁 (code: $1D6C1) to it; then do a select all (Ctrl+A) and run the code: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i showmessage( inttostr(utf8length(Memo1.Text)) + ' ' +     inttostr(Memo1.SelLength) + ' ' +     inttostr(utf8length(Memo1.SelText))); It will display 1 2 1. Strangely enough, if you add Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i Memo1.SelectAll;before the code above, it will display 1 1 1. Manually selecting text causes SelLength to return a wrong value; programmatically selecting the text returns the correct value. Environment: Windows 7 + Lazarus 1.6.4 (64 bit). ---------------------------------------------------------------------- Bart · 2017-11-23 14:26 Maybe manually you also select a linefeed? Bart ---------------------------------------------------------------------- fedkad · 2017-11-23 15:00 Quote from: Bart on November 23, 2017, 03:26:52 pmMaybe manually you also select a linefeed? No. I was careful not to do that. Can you test it? And also, shouldn't Memo1.SelLength = utf8length(Memo1.SelText)? Obviously the code displays different values. ---------------------------------------------------------------------- ASerge · 2017-11-23 22:56 Quote from: fedkad on November 23, 2017, 04:00:10 pmAnd also, shouldn't Memo1.SelLength = utf8length(Memo1.SelText)? Obviously the code displays different values. Detail: string: Length=4, Bytes=$F09D9B81 UnicodeString: Length=2, Bytes=$35D8C1DE CodePoint: UTF8Length=1, Bytes=$C1D601 The system (my OS is Windows) uses UnicodeChar (2 byte characters). Because UnicodeLength = 2, then Ctrl + A will select two unicode characters (one codepont). Lazarus Memo1.SelectAll does not use the system message (EM_SETSEL, 0, -1), but explicitly calls SetSelLength(UTF8Length(Text)), which select only the first unicode character (half of the codepoint). ---------------------------------------------------------------------- fedkad · 2017-11-24 08:50 So, is this a bug or a feature? How do we code to avoid such inconsistencies? Should we avoid the functions/properties that return wrong (unexpected and inconsistent) values? Please forgive me. I am just confused. ---------------------------------------------------------------------- tomitomy · 2017-11-25 08:40 I did the test in Linux GTK2, and the results were all 111. Maybe it's just a BUG in Windows. Maybe you should submit a BUG report. ---------------------------------------------------------------------- fedkad · 2017-11-26 11:31 I prepared a small demo application. Please, find it in the attached ZIP file. I tested it in Linux (1.8.0~rc4) and it did not have the problem I described. However, Lazarus 1.6.4 (64bit) in Windows does have the problem. To test it, run the application and try to select from the beginning of the Editbox 1, 2, and 3 characters. The relevant values will be displayed on the right. I will try to submit a BUG report. I attached some screenshots too. ---------------------------------------------------------------------- Bart · 2017-11-26 13:31 Don't forget to attach the sample application and the screenshots to the bugreport. Bart ---------------------------------------------------------------------- fedkad · 2017-12-05 14:31 I DID submit a bug report. However, I am not very happy of its treatment!  https://bugs.freepascal.org/view.php?id=32731 It was "related" to bug #0030478, whose platform and other information does NOT look to be related to my bug platform, etc. Note: I DID mention that this problem is not encountered in Linux platform. It's a Windows only bug. ---------------------------------------------------------------------- tomitomy · 2017-12-05 14:41 I also have some reports that have not been dealt with, I think it may be too complicated, no one has time to deal with, or it's not a matter of top priority. ---------------------------------------------------------------------- wp · 2017-12-05 15:26 Quote from: fedkad on December 05, 2017, 03:31:58 pmI DID submit a bug report. However, I am not very happy of its treatment!  https://bugs.freepascal.org/view.php?id=32731 It was "related" to bug #0030478, whose platform and other information does NOT look to be related to my bug platform, etc. Note: I DID mention that this problem is not encountered in Linux platform. It's a Windows only bug. You also wrote "I tested the same code in Linux + Lazarus (1.8.0~rc4) and it did not have the problem described above." This means: this bug is fixed. I doubt if anybody will take care of your report and go through the hassle of backporting the fix to an old version. There will not be any 1.6.6 release since v1.8 is about to appear shortly. ---------------------------------------------------------------------- Thaddy · 2017-12-05 15:28 Yup. Like a patch for Windows 1.0 that makes it Windows 10.1 ---------------------------------------------------------------------- wp · 2017-12-05 15:57 Quote from: tomitomy on December 05, 2017, 03:41:21 pmI also have some reports that have not been dealt with. If the reports are important to you change something in them, maybe add a simple comment "ping" or so to bring it up to the top of the report list. If a report has not been assigned to a developer yet it could easily be unnoticed if the developer who normally would take care of it does not see it within the first days. Because later it will be buried by many other reports. And most important: Reports with poor description of the issue, referring to an old version, missing a demo project. missing exact steps how to reproduce, etc, have a high change of being forgotten. ---------------------------------------------------------------------- Thaddy · 2017-12-05 16:23 Quote from: ASerge on November 23, 2017, 11:56:13 pmThe system (my OS is Windows) uses UnicodeChar (2 byte characters). Because UnicodeLength = 2, No, it is not! Unicode16 can have a length of more than 2. It is explicitly a variable length format with a minimum of 2 bytes per character. You confuse UTF16 with UCS, the latter indeed being limited to exactly two bytes. Windows is - since XP+  - UTF16 and not UCS anymore. So you can't rely on a two byte format. http://www.differencebetween.net/technology/software- technology/difference-between-ucs-2-and-utf-16/ ---------------------------------------------------------------------- JuhaManninen · 2017-12-05 18:47 Quote from: fedkad on December 05, 2017, 03:31:58 pmI DID submit a bug report. However, I am not very happy of its treatment!  https://bugs.freepascal.org/view.php?id=32731 It was "related" to bug #0030478, whose platform and other information does NOT look to be related to my bug platform, etc. I had added a relation to a wrong issue. The right one is #32101. I changed that now. Please take a look. Both issues are about wrong surrogate pair treatment on Windows. Who will fix them? It may not be trivial. Somebody must fearlessly dive into the LCL-Win32 code. ---------------------------------------------------------------------- Thaddy · 2017-12-05 20:28 Fix what? Fix to UTF16? Drop UCS? See my remark on the bug report. Note e.g. Win2000 is still UCS and widely used in industrial and banking applications. We can't just drop that. ---------------------------------------------------------------------- JuhaManninen · 2017-12-05 22:09 Fixing UTF-16 should be enough. If the OS does not support it then it will not work whatever LCL does. LCL however should try to pass valid UTF-16 for Windows based on user input. ---------------------------------------------------------------------- tomitomy · 2017-12-06 05:47 Quote from: wp on December 05, 2017, 04:57:11 pmQuote from: tomitomy on December 05, 2017, 03:41:21 pmI also have some reports that have not been dealt with. If the reports are important to you change something in them, maybe add a simple comment "ping" or so to bring it up to the top of the report list. If a report has not been assigned to a developer yet it could easily be unnoticed if the developer who normally would take care of it does not see it within the first days. Because later it will be buried by many other reports. And most important: Reports with poor description of the issue, referring to an old version, missing a demo project. missing exact steps how to reproduce, etc, have a high change of being forgotten. Maybe it's because my description is too bad, I'm not good at using English. I don't know much about Lazarus, I don't even know what categories to choose when I submit a report. I don't know how to modify the report, I didn't see the "Edit" button. I don't know how to bring my report up to the top of the report list. All I can do is add note. ---------------------------------------------------------------------- molly · 2017-12-06 05:56 Quote from: tomitomy on December 06, 2017, 06:47:42 amI don't know how to bring my report up to the top of the report list. All I can do is add note. And adding a note is what wp was reffering to. Just add the word "ping" to that note (or preferably use the words: "any news ? is there anything i am able to do to speed up progress ?") and the report will be listed at the top of the list again ---------------------------------------------------------------------- tomitomy · 2017-12-06 06:06 Quote from: molly on December 06, 2017, 06:56:17 amQuote from: tomitomy on December 06, 2017, 06:47:42 amI don't know how to bring my report up to the top of the report list. All I can do is add note. And adding a note is what wp was reffering to. Just add the word "ping" to that note (or preferably use the words: "any news ? is there anything i am able to do to speed up progress ?") and the report will be listed at the top of the list again  Thank you, molly, I'll try it. ---------------------------------------------------------------------- wp · 2017-12-06 09:23 Quote from: tomitomy on December 06, 2017, 06:47:42 amI'm not good at using English. I can assure you, it certainly is not errors in English writing which makes up a poor bug report. Usually people do not provide enough information, they do not describe what happens or how a bug is triggered. Always attach a little demo which shows the bug - this is useful also later if another bug is fixed having side-effects on the current one. But please don't attach your current project in which you see the bug - nobody will work through a foreign program. Just pust yourself into the position of the developer: What you I need to know to understand the issue? As for the other points that you mention: Yes, I never understood why a normal user does not have the permission to edit his own report. So, you only can add a new comment or add/remove yourself to the list of users monitoring this issue etc to bring a report back up  to the top again. Categorization is less important, developers usually fix it if not correct - one point though: At least the selection between FPC or Lazarus should be correct because these are the two main projects and a developer usually does not work in both, but even this is usually fixed internally. ---------------------------------------------------------------------- fedkad · 2017-12-06 09:32 Thank you, JuhaManninen! ---------------------------------------------------------------------- JuhaManninen · 2017-12-06 11:10 Quote from: tomitomy on December 06, 2017, 06:47:42 amI don't know how to bring my report up to the top of the report list. All I can do is add note. You can also upload a patch that fixes the issue. It is the best way to draw attention. Remember, this is a FOSS project done by volunteers. There is no guarantee that a particular issue gets fixed even if you report it well. ---------------------------------------------------------------------- tomitomy · 2017-12-06 11:43 Quote from: wp on December 06, 2017, 10:23:45 amQuote from: tomitomy on December 06, 2017, 06:47:42 amI'm not good at using English. I can assure you, it certainly is not errors in English writing which makes up a poor bug report. Usually people do not provide enough information, they do not describe what happens or how a bug is triggered. Always attach a little demo which shows the bug - this is useful also later if another bug is fixed having side-effects on the current one. But please don't attach your current project in which you see the bug - nobody will work through a foreign program. Just pust yourself into the position of the developer: What you I need to know to understand the issue? As for the other points that you mention: Yes, I never understood why a normal user does not have the permission to edit his own report. So, you only can add a new comment or add/remove yourself to the list of users monitoring this issue etc to bring a report back up  to the top again. Categorization is less important, developers usually fix it if not correct - one point though: At least the selection between FPC or Lazarus should be correct because these are the two main projects and a developer usually does not work in both, but even this is usually fixed internally. Thank you, wp, I have added test file to my reports. Quote from: JuhaManninen on December 06, 2017, 12:10:55 pmQuote from: tomitomy on December 06, 2017, 06:47:42 amI don't know how to bring my report up to the top of the report list. All I can do is add note. You can also upload a patch that fixes the issue. It is the best way to draw attention. Remember, this is a FOSS project done by volunteers. There is no guarantee that a particular issue gets fixed even if you report it well. Thank you, JuhaManninen, If I find a way to fix the BUG, I will upload it. ---------------------------------------------------------------------- ASerge · 2017-12-06 16:59 Quote from: Thaddy on December 05, 2017, 05:23:45 pmYou confuse UTF16 with UCS, the latter indeed being limited to exactly two bytes. What does "confuse UTF16 with UCS"? I said that the UnicodeChar size is 2 bytes. If in doubt, run Writeln(SizeOf(UnicodeChar)). These double-byte characters are used in the Windows API. But logical symbols (CodePoint), of course, can consist of more than one character. You yourself have come up with an error that is not there, exposed it, but why do you blame me for this? ---------------------------------------------------------------------- fedkad · 2017-12-06 17:39 I am sorry ASerge and Thaddy. But, I cannot understand the thing you are arguing about. As far as I know, in Lazarus or Free Pascal we are using UTF-8 characters ( https://en.wikipedia.org/wiki/UTF-8 ) which means that whatever the number of bytes (i.e., 1, 2, 3, or 4) used to represent the Unicode "code point" for that characters, the utf8length function should return the number of "code points" (charactes). The utf8length function works correctly in this respect. However, the problem we are discussing here is that SelLength, SelText, and SelStart work incorrectly when there are characters with Unicode value > $FFFF in the string containing or preceding the selection. And a single character such as 𝛁 (code: $1D6C1) cannot be selected properly by using the keyboard. ---------------------------------------------------------------------- ASerge · 2017-12-06 17:56 Quote from: fedkad on December 06, 2017, 06:39:43 pmHowever, the problem we are discussing here is that SelLength, SelText, and SelStart work incorrectly when there are characters with Unicode value > $FFFF in the string containing or preceding the selection. Because they rely on the Windows API, which does not count in the "code point", but in double-byte characters. The SelectAll does not access the Windows API, but does the operation itself, in the "code point". Hence the difference. ---------------------------------------------------------------------- fedkad · 2017-12-07 08:25 After the new release I tested the code; and saw that the problem still persists on Lazarus 1.8.0. I will test the code in Windows 10 + Lazarus 1.8.0 (64-bit) also as soon as possible. ---------------------------------------------------------------------- JuhaManninen · 2017-12-07 09:39 The problem now is that people use different words for the same thing, and the same word (character) for different things. ASerge used "character" for Pascal Char type. fedkad used it for .. I am not sure what. In Unicode a "character" can mean about 7 different things. The word must be used very carefully and its meaning should be clarified then. Quote from: fedkad on December 06, 2017, 06:39:43 pmAs far as I know, in Lazarus or Free Pascal we are using UTF-8 characters ( https://en.wikipedia.org/wiki/UTF-8 ) Lazarus uses UTF-8 encoding in AnsiString. Don't use "characters" here, it is wrong. FPC aims for the Delphi compatible UTF-16 strings with mode DelphiUnicode. The default encoding of AnsiString is not Unicode at all. Quote... which means that whatever the number of bytes (i.e., 1, 2, 3, or 4) used to represent the Unicode "code point" for that characters, the utf8length function should return the number of "code points" (charactes). "Character" should not be used in the meaning of "codepoint". Unicode is confusing enough without such extra confusion. How do you call combining codepoints then? What means: the Unicode "code point" for that characters? I think the best meanings for "character" are: 1. Pascal Char, for historical reasons. In Unicode terms it represents a codeunit and is useful in many situations. 2. User perceived character. This involves combining codepoints, glyphs, ligatures and whatever. Quote from: ASerge on December 06, 2017, 06:56:30 pmBecause they rely on the Windows API, which does not count in the "code point", but in double-byte characters. The SelectAll does not access the Windows API, but does the operation itself, in the "code point". Hence the difference. Ok, "double-byte character" means here codeunit or Pascal WideChar. The problem came with codepoints outside Unicode BMP which means surrogate pairs in UTF-16. The LCL-Win32 binding code should take care of it and pass the values to WinAPI. It does not, somebody should fix it. ---------------------------------------------------------------------- munair · 2017-12-08 09:01 Quote from: JuhaManninen on December 07, 2017, 10:39:58 amThe default encoding of AnsiString is not Unicode at all.Indeed. Unicode has been a problem with Windows from the beginning. UTF-8 is still not supported by the Win API. Why Microsoft chose UTF-16 over UTF-8 (apart from the fact that it comes closest to UCS2) is still beyond me. And don't come with the crap that UTF-8 wasn't around at the time. Microsoft simply made a choice, and an expensive one if you ask me. Regarding UTF-16, we can read: "UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 valid code points of Unicode. The encoding is variable-length, as code points are encoded with one or two 16-bit code units." That's two or four bytes. UTF-8 has two main advantages: it is ASCII compatible and it eliminates the endianess problem. ---------------------------------------------------------------------- fedkad · 2017-12-08 10:01 Quote from: JuhaManninen on December 07, 2017, 10:39:58 amfedkad used it for .. I am not sure what. ... I think the best meanings for "character" are: 1. Pascal Char, for historical reasons. In Unicode terms it represents a codeunit and is useful in many situations. 2. User perceived character. This involves combining codepoints, glyphs, ligatures and whatever. By character I mean every possible of 1,112,064 valid code points in Unicode that can be represented in 1, 2, 3, or 4 bytes in UTF-8. In other words the "character" that I am talking about is something that is counted as "1" unit by the UTF8Length function or is returned by the UTF8Copy function when we give 1 as its third argument. The Lazarus documentation itself is using the same terminology: function UTF8Copy(   const s: string;   StartCharIndex: PtrInt;   CharCount: PtrInt ):string; See also: http://wiki.freepascal.org/UTF8_strings_and_characters#Acces sing_the_Nth_UTF8_character ---------------------------------------------------------------------- JuhaManninen · 2017-12-08 10:46 Quote from: fedkad on December 08, 2017, 11:01:31 amBy character I mean every possible of 1,112,064 valid code points in Unicode that can be represented in 1, 2, 3, or 4 bytes in UTF-8. In other words the "character" that I am talking about is something that is counted as "1" unit by the UTF8Length function or is returned by the UTF8Copy function when we give 1 as its third argument. Ok, that is codepoint. BTW, the valid codepoints in Unicode can be represented by any encoding. Conversions between Unicode encodings are always lossless. Only codepoints are encoded. People should remember it when arguing about the supremacy of a certain encoding. The true complexity of Unicode lies beyond codepoints and encodings. QuoteThe Lazarus documentation itself is using the same terminology: ... Good point. Actually it bothered also me when I edited the pages. I think they must all be changed to "codepoint" to prevent extra confusion. ---------------------------------------------------------------------- munair · 2017-12-08 13:04 Quote from: JuhaManninen on December 08, 2017, 11:46:24 amOnly codepoints are encoded. People should remember it when arguing about the supremacy of a certain encoding. The true complexity of Unicode lies beyond codepoints and encodings.True, but it's the codepoints we work with when representing or manipulating text. It is not just happenstance that UTF-8 has become the predominant encoding scheme on the internet and with Unix-like operating systems. ---------------------------------------------------------------------- JuhaManninen · 2017-12-08 14:42 Quote from: Munair on December 08, 2017, 02:04:31 pmTrue, but it's the codepoints we work with when representing or manipulating text. No, we also work with codeunits, combining codepoints, normalization of "characters" using decomposed or precomposed codepoint forms, locale dependent rules for upper-/lower-casing codepoints and "characters", locale dependent rules for sorting them ... etc. A programmer of an advanced text layout application must also work with glyphs and graphemes etc. which I don't know well. I only know it is complicated! QuoteIt is not just happenstance that UTF-8 has become the predominant encoding scheme on the internet and with Unix-like operating systems. Yes, UTF-8 has nice features but it is not a holy grail that makes the Unicode's complexity go away. No encoding can do that because the complexity is not in encodings or codepoints in the first place. Some people here and in mailing lists have suggested turning all string data into a fixed width array of "characters", I guess something like what UTF-32 does. In their opinion all problems would be solved then. Unfortunately it is not so easy. Clearly those people do not understand the complexity of Unicode, and don't know what the word "character" can mean there. ---------------------------------------------------------------------- tomitomy · 2017-12-08 15:08 Is this right: Code: [Select]❤(Character) = #E2#9D#A4(UTF8 code) = #E2(Byte) & #9D(Byte) & #A4(Byte) = 10084(Code Point) ---------------------------------------------------------------------- JuhaManninen · 2017-12-08 17:40 Quote from: tomitomy on December 08, 2017, 04:08:22 pmCode: [Select]❤(Character) = #E2#9D#A4(UTF8 code) = #E2(Byte) & #9D(Byte) & #A4(Byte) = 10084(Code Point)Maybe "UTF8 code" could be "UTF-8 encoded codepoint" or something. Anyway you picked an easy example consisting of one codepoint only. A visible character can consist of many codepoints. ---------------------------------------------------------------------- munair · 2017-12-08 17:51 Quote from: JuhaManninen on December 08, 2017, 03:42:14 pmNo, we also work with codeunits, combining codepoints, normalization of "characters" using decomposed or precomposed codepoint forms, locale dependent rules for upper-/lower-casing codepoints and "characters", locale dependent rules for sorting them ... etc.Yes, it is a complicated subject. My understanding is that a code unit is a particular sequence of bits. In ASCII a code unit is 7 bits, in UTF-8 a code unit is 8 bits and in UTF-16 a code unit is 16 bits. A codepoint may consist of one or more code units. In ASCII a code unit and a codepoint are the same. In UTF-8 a codepoint can be one, two or three code units. When speaking of characters, most people refer to the glyphs that we see, like 'A' or '€'. In the days of ASCII a code unit, codepoint and character referred to the same 7 bits, or simply a byte. Today a character or codepoint may occupy one or more bytes (code units) depending on the encoding. It would have been simpler if the world could agree on a single encoding covering all glyphs we use for communication. ---------------------------------------------------------------------- munair · 2017-12-08 17:52 Quote from: JuhaManninen on December 08, 2017, 06:40:03 pmQuote from: tomitomy on December 08, 2017, 04:08:22 pmCode: [Select]❤(Character) = #E2#9D#A4(UTF8 code) = #E2(Byte) & #9D(Byte) & #A4(Byte) = 10084(Code Point)Maybe "UTF8 code" could be "UTF-8 encoded codepoint" or something. Anyway you picked an easy example consisting of one codepoint only. A visible character can consist of many codepoints. I think you mistake codepoints for code units. This example shows one codepoint but three code units. ---------------------------------------------------------------------- JuhaManninen · 2017-12-08 20:09 Quote from: Munair on December 08, 2017, 06:51:32 pmYes, it is a complicated subject. My understanding is that a code unit is a particular sequence of bits. In ASCII a code unit is 7 bits, in UTF-8 a code unit is 8 bits and in UTF-16 a code unit is 16 bits. A codepoint may consist of one or more code units. In ASCII a code unit and a codepoint are the same. In UTF-8 a codepoint can be one, two or three code units. ... or even four codeunits. Yes except that terms "codeunit" and "codepoint" are used only with Unicode AFAIK. They were not needed with ASCII. QuoteWhen speaking of characters, most people refer to the glyphs that we see, like 'A' or '€' which would be identical to a codepoint. Those 2 happen to have only one codepoint, yes. QuoteIn the days of ASCII a code unit, codepoint and character referred to the same 7 bits, or simply a byte. Words codeunit and codepoint were not used then. A "character" meant a byte. Yes, everything was simple in the old days.  QuoteToday a character or codepoint may occupy one or more bytes (code units) depending on the encoding. It would have been simpler if the world could agree on a single encoding covering all glyphs we use for communication. Clearly you still don't understand the concepts of Unicode. A codepoint occupies one or more codeunits depending on encoding, yes. A "character" occupies one or more codepoints. I put the word "character" in quotes because its meaning is so ambiguous. See these 2 characters: ÓÓ They look the same, don't they? However: Ó has 2 codepoints and 3 codeunits. Ó has 1 codepoint and 2 codeunits. Amazing, ha! QuoteI think you mistake codepoints for code units. This example shows one codepoint but three code units. No mistake. That example had one codepoint but some other example could have more. Note also that encodings make no difference for that. Only codepoints are encoded! More examples (codepoints are encoded with UTF-8): ch=Ở has 2 codepoints and 4 codeunits. ch=Ở has 1 codepoint and 3 codeunits. ch=Ć̲ has 3 codepoints and 5 codeunits. See also:  http://www.alanwood.net/unicode/combining_diacritical_marks.html ---------------------------------------------------------------------- munair · 2017-12-08 23:09 What you describe as a character "ch=Ở has 2 codepoints and 4 codeunits" are in fact two characters: "Ơ" and "  ̉". What you describe as a character ch=Ć̲ has 3 codepoints and 5 codeunits are in fact three characters: 43h - CCh 81h - CCh B2h. When feeding that sequence to my UTF-8 library likeCode: FreeBasic  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i print UMid(text, 2, 1)the output will be CCh 81h, which is one character or codepoint. Has nothing to do with the glyphs we recognize a a single character. So I believe we understand each other. But keep in mind that even the Linux console has difficulty displaying these glyphs properly. For example, when trying to build the first glyph of two code points chr($C6)+chr($A0)+chr($CC)+chr($89), a rectangle is displayed instead. Likewise, when decomposing the two codepoints into two strings and concatenating them back will also display the rectangle even though the byte sequence is exactly the same and the UTF-8 library decodes the string correctly. ---------------------------------------------------------------------- JuhaManninen · 2017-12-09 00:04 Quote from: Munair on December 09, 2017, 12:09:21 amWhat you describe as a character "ch=Ở has 2 codepoints and 4 codeunits" are in fact two characters: "Ơ" and "  ̉". What you describe as a character ch=Ć̲ has 3 codepoints and 5 codeunits are in fact three characters: 43h - CCh 81h - CCh B2h. Well, they are codepoints which are combined to form a "user perceived character". In my opinion calling "codepoint" as "character" is very confusing. How do you call the visual combined thingy then? QuoteKeep in mind that even the Linux console has difficulty displaying these glyphs properly. ... Yes, both SW applications and fonts have issues displaying complex and rare Unicode "characters". Still a long way to go... I guess you were right about the terms (codeunit, codepoint) being used already before Unicode. There were many encoding systems before the global Unicode standard was created. Anyway I am happy it was created. It is complex, yes, but at least it is a global standard. BTW, I changed "character" into "codepoint" where appropriate in this page:  http://wiki.lazarus.freepascal.org/UTF8_strings_and_characters The wrong "character" term is still used in some function names. I am not sure if it is worth the trouble to change them. The current names should be deprecated and kept as aliases for some time. ---------------------------------------------------------------------- howardpc · 2017-12-09 08:10 Quote from: JuhaManninen on December 09, 2017, 01:04:12 amThe wrong "character" term is still used in some function names. I am not sure if it is worth the trouble to change them. The current names should be deprecated and kept as aliases for some time. I am convinced it is worth the trouble to change them. A comprehensively consistent naming convention for Lazarus functions is nowhere more applicable than in the confusing world of unicode and text encoding functionality. Some of this confusion arises (apart from the inherent complexities of the topic) because people in discussion use the same terms such as "character" with different or overlapping or imprecise meanings, often without being aware of the ambiguities, or different interpretations other readers may assume. Surely the LCL can embrace informed and intelligent consistency and abandon well-meant but (with hindsight) flawed naming. ---------------------------------------------------------------------- munair · 2017-12-09 08:59 Quote from: JuhaManninen on December 09, 2017, 01:04:12 amWell, they are codepoints which are combined to form a "user perceived character". In my opinion calling "codepoint" as "character" is very confusing. How do you call the visual combined thingy then?The visual combined thingy or glyph is nothing because one cannot do string operations on it without breaking it. For example, if it would be the first glyph in a string consisting of five code units, then a ULeft(text, 1) should isolate all five code units, but that's not happening because there's no indication in the first byte that it should take the next four too. This is confirmed in a text editor supporting UTF-8. When going over the glyph with the cursor, it sort of sticks to it three times before moving to the next. In short, it isn't a real glyph or character because it isn't recognized as such. One can do a lot of funny things in Unicode. It's like a palette of over a million colors, but it doesn't guarantee proper handing of all glyphs that could be constructed. ---------------------------------------------------------------------- munair · 2017-12-09 09:08 Quote from: howardpc on December 09, 2017, 09:10:42 amQuote from: JuhaManninen on December 09, 2017, 01:04:12 amThe wrong "character" term is still used in some function names. I am not sure if it is worth the trouble to change them. The current names should be deprecated and kept as aliases for some time. I am convinced it is worth the trouble to change them. A comprehensively consistent naming convention for Lazarus functions is nowhere more applicable than in the confusing world of unicode and text encoding functionality. Some of this confusion arises (apart from the inherent complexities of the topic) because people in discussion use the same terms such as "character" with different or overlapping or imprecise meanings, often without being aware of the ambiguities, or different interpretations other readers may assume. Surely the LCL can embrace informed and intelligent consistency and abandon well-meant but (with hindsight) flawed naming. The trouble goes back to the 70s and 80s when programming languages were developed using improper names. For example, the function ASC should only allow bytes in the range 0h - 7Fh strictly speaking. The name should be ASCE (extended ASCII) to allow all unsigned byte values. Same for CHR -> CHRE. Especially since the range 80h to FFh had different characters for each code page. ---------------------------------------------------------------------- tomitomy · 2017-12-09 11:40 I think it's a bad idea to represent a single character with more than one codepoint, it breaks the unitarity of the encoding (using the same rule to encode all the characters), and I do not know why there are multiple-codepoint characters, and who are using these characters? Do they have no trouble? No one object to this rule? ---------------------------------------------------------------------- JuhaManninen · 2017-12-09 13:09 Quote from: tomitomy on December 09, 2017, 12:40:49 pmI think it's a bad idea to represent a single character with more than one codepoint, it breaks the unitarity of the encoding (using the same rule to encode all the characters), No, it does not break encoding in any way because only codepoints are encoded. This seems to be a difficult concept to understand. People argue about which encoding is better as if encoding was a major complication. Codepoints and encodings are the easy part of Unicode. The true complexity lies beyond them. Quote and I do not know why there are multiple-codepoint characters, and who are using these characters? The common case for multiple codepoints are accent or umlaut marks, also called diacritical marks or diacritical signs. A codepoint for a diacritical mark is a special combining codepoint which is always combined to its preceding codepoint. For historical reasons some of those characters also have a single codepoint form. For example 'ä' has its own codepoint but it can also be 'a' + umlaut. Obviously comparing and searching text becomes difficult when there are different codes for one character. Text can then be normalized which means the combining codepoint form (single alphabet codepoint + a diacritical mark) is used always. Exotic languages have more complex rules for combining codepoints. The UTF8tools by Theo implements some complex rules which I don't quite understand. I hope I will some day. QuoteDo they have no trouble? No one object to this rule? They have trouble if your code is buggy and does not respect the rules. You can object it freely but it remains a fundamental part of the Unicode standard. ---------------------------------------------------------------------- JuhaManninen · 2017-12-09 13:41 Quote from: Munair on December 09, 2017, 09:59:34 amThe visual combined thingy or glyph is nothing because one cannot do string operations on it without breaking it. For example, if it would be the first glyph in a string consisting of five code units, then a ULeft(text, 1) should isolate all five code units, but that's not happening because there's no indication in the first byte that it should take the next four too. Not in the first byte obviously. The information is in a combining codepoint. They are special codepoints which are always combined to a preceding codepoint. Claiming they are just "normal" codepoints is wrong. It is false information. If you coded your function ULeft() to support only codepoints, it is your own fault. Don't blame the Unicode standard. Combining codepoints have clear rules. Look at my unit LazUnicode. It has: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function UTF8IsCombining(const AChar: PChar): Boolean;begin  Result :=   ( (AChar[0] = #$CC) ) or                   // Combining Diacritical Marks (belongs to previos char) 0300-036F   ( (AChar[0] = #$CD) and (AChar[1] in [#$80..#$AF]) ) or                        // Combining Diacritical Marks   ( (AChar[0] = #$D8) and (AChar[1] in [#$90..#$9A]) ) or                        // Arabic 0610 (d890)..061A (d89a)   ( (AChar[0] = #$D9) and (AChar[1] in [#$8b..#$9f, #$B0]) ) or                // Arabic 064B (d98b)..065F (d99f) // 0670 (d9b0)    ( (AChar[0] = #$DB) and (AChar[1] in [#$96..#$9C, #$9F..#$A4, #$A7..#$A8, #$AA..#$AD]) ) or // Arabic 06D6 (db96)..  .. ..06ED (dbad)   ( (AChar[0] = #$E0) and (AChar[1] = #$A3) and (AChar[2] in [#$A4..#$BE]) ) or  // Arabic 08E4 (e0a3a4) ..08FE (e0a3be)    ( (AChar[0] = #$E1) and (AChar[1] = #$B7) ) or                                 // Combining Diacritical Marks Supplement 1DC0-1DFF (e1b780)   ( (AChar[0] = #$E2) and (AChar[1] = #$83) and (AChar[2] in [#$90..#$FF]) ) or  // Combining Diacritical Marks for Symbols 20D0-20FF   ( (AChar[0] = #$EF) and (AChar[1] = #$B8) and (AChar[2] in [#$A0..#$AF]) );    // Combining half Marks FE20-FE2Fend;and a similar function for UTF-16 which then allows encoding agnostic code. It is used in TUnicodeCharacterEnumerator which indeed can iterate through Unicode characters, including their diacritical marks. Now it is even used for the "for .. in" loop. I am not sure if it is the right thing but nobody objected. Let's see how it works... QuoteThis is confirmed in a text editor supporting UTF-8. When going over the glyph with the cursor, it sort of sticks to it three times before moving to the next. In short, it isn't a real glyph or character because it isn't recognized as such. Depends on editor of course. Some program code supports Unicode better than some other code. QuoteOne can do a lot of funny things in Unicode. It's like a palette of over a million colors, but it doesn't guarantee proper handing of all glyphs that could be constructed. Yes, there are different levels of Unicode support. Getting codepoints right should be considered the minimum requirement. Drawing a glyph containing many diacritical marks both above and below a "character" is challenging. Text layout apps like browsers and WYSIWYG editors can show quite different results. The goal is to not have the diacritical marks on top of each other blurring the view. ---------------------------------------------------------------------- munair · 2017-12-09 19:00 Quote from: JuhaManninen on December 09, 2017, 02:41:32 pmIf you coded your function ULeft() to support only codepoints, it is your own fault. Don't blame the Unicode standard. Actually, I translated much of my UTF-8 library for FreeBASIC from the lazutf8.pas, a key procedure of which is:Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function UTF8CharacterLengthFull(p: PChar): integer;begin  case p^ of  #0..#191: // %11000000    // regular single byte character (#0 is a character, this is Pascal ;)    Result:=1;  #192..#223: // p^ and %11100000 = %11000000    begin      // could be 2 byte character      if (ord(p[1]) and %11000000) = %10000000 then        Result:=2      else        Result:=1;    end;  #224..#239: // p^ and %11110000 = %11100000    begin      // could be 3 byte character      if ((ord(p[1]) and %11000000) = %10000000)      and ((ord(p[2]) and %11000000) = %10000000) then        Result:=3      else        Result:=1;    end;  #240..#247: // p^ and %11111000 = %11110000    begin      // could be 4 byte character      if ((ord(p[1]) and %11000000) = %10000000)      and ((ord(p[2]) and %11000000) = %10000000)      and ((ord(p[3]) and %11000000) = %10000000) then      Result:=4      else        Result:=1;    end;  else    Result:=1;  end;end;Sure, my work isn't finished. BUT... the documentation already goes wrong by suggesting to iterate through codepoints as if they were characters. That will definitely go wrong with combining codepoints, as can be seen with synedit. So it is not my ULeft() that's wrong. When using UTF8Length it should give 1 for a complex glyph exisiting of two or more codepoints, but it doesn't. Likewise fot UTF8Copy. They will break complex characters made of multiple codepoints, which I thibk is the core problem of the issue the OP encountered. ---------------------------------------------------------------------- JuhaManninen · 2017-12-09 23:04 Quote from: Munair on December 09, 2017, 08:00:07 pmBUT... the documentation already goes wrong by suggesting to iterate through codepoints as if they were characters. Which documentation? I know unit LazUTF8 has functions with "character" in their names while it should be "codepoint". I will improve it tomorrow as howardpc suggested. QuoteThat will definitely go wrong with combining codepoints, as can be seen with synedit. So it is not my ULeft() that's wrong. When using UTF8Length it should give 1 for a complex glyph exisiting of two or more codepoints, but it doesn't. Likewise fot UTF8Copy. Now you are getting more and more confused. Those functions are meant to work at codepoint resolution. It is often enough and it can be done 100% correctly because the rules are simple. The rules for stitching codepoints together are more complex. My enumerator in LazUnicode only handles combining codepoints which covers most cases in western languages. LazUTF8 currently supports processing individual codepoints, nothing more. QuoteThey will break complex characters made of multiple codepoints, which I thibk is the core problem of the issue the OP encountered. No, AFAIK the OP encountered a single codepoint outside Unicode BMP. Unfortunately lots of code dealing with UTF-16 has trouble with it, this forum SW included, because the encoded value does not fit in one WideChar but needs a surrogate pair. It is equivalent to a multi-byte codepoint in UTF-8. About half of codepoints are already outside BMP and it is counting when Unicode expands. I have used the wineglass emoji as an example case :  https://emojiterra.com/wine-glass/ ---------------------------------------------------------------------- tomitomy · 2017-12-10 08:24 Is there a function to get the count of characters in a string (not the count of codepoints). And is there a function to get the character at the specified position in the string (not the codepoint)? I just know this enumerator: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i for string in string do ... With the enumerator to achieve the above function is inefficient? ---------------------------------------------------------------------- munair · 2017-12-10 09:33 Quote from: tomitomy on December 10, 2017, 09:24:10 amIs there a function to get the count of characters in a string (not the count of codepoints). And is there a function to get the character at the specified position in the string (not the codepoint)? I just know this enumerator: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i for string in string do ... With the enumerator to achieve the above function is inefficient? That would require indexing of grapheme clusters, which is quite complicated. Here is aninteresting read on the subject: https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning- to-unicode-code-points/ ---------------------------------------------------------------------- munair · 2017-12-10 09:41 It would be easiest BY FAR if the whole world would switch to English (at least with communication and computing), not because English should be the predominant language, but because it is most tightly integrated to the computer industry from the beginning. Try to calculate the current costs to get Unicode right on every OS and program, including browsers; they also don't seem to get it right. ---------------------------------------------------------------------- Thaddy · 2017-12-10 10:31 Yeah, but English is poor man's Dutch, just like German. Change everything to Dutch. Dutch is a convergence between two of the most widely used languages and the Brits call it Chinese, so that's also covered ---------------------------------------------------------------------- tomitomy · 2017-12-10 12:51 Quote from: Munair on December 10, 2017, 10:33:04 amThat would require indexing of grapheme clusters, which is quite complicated. That is to say, It is very difficult to be compatible with all languages in the software, Can only compatible with some languages. In this case, it is only enough to consider the single-codepoint character. Quote from: Munair on December 10, 2017, 10:41:01 amIt would be easiest BY FAR if the whole world would switch to English (at least with communication and computing), not because English should be the predominant language, but because it is most tightly integrated to the computer industry from the beginning. Try to calculate the current costs to get Unicode right on every OS and program, including browsers; they also don't seem to get it right.  Every language has profound connotations behind it. It is like losing the charm after translating Chinese poetry into English. Similarly, the translation of English poetry into Chinese also loses its charm. characters are the carrier of poetry. ---------------------------------------------------------------------- munair · 2017-12-10 15:38 Quote from: tomitomy on December 10, 2017, 01:51:23 pmEvery language has profound connotations behind it. It is like losing the charm after translating Chinese poetry into English. Similarly, the translation of English poetry into Chinese also loses its charm. characters are the carrier of poetry. We lost charm the moment we invented computers to write and speak for us. Take a look at the monstrous attempt and result  of the ICU (http://site.icu-project.org/). Unicode is not "keep it simple." A better approach may have been to create variable length, multi-byte character sets in a database with a language id for each set. If a document can carry an encoding, it could also carry a language ID. No mistake, no misformed or missing characters. But no, as usual, humans tend to keep things not simple. ---------------------------------------------------------------------- Thaddy · 2017-12-10 15:48 Quote from: Munair on December 10, 2017, 04:38:39 pmA better approach may have been to create variable length, multi-byte character sets Oh, you mean UTF8 (1 to 4 byte) ? or UTF16 (2 to 4 byte) ? These are variable length character sets by definition, not to be confused with UCS(2) (fixed length two byte) or UTF32 (Fixed length 4 byte).... ---------------------------------------------------------------------- munair · 2017-12-10 16:06 @Thaddy: NO, current encodings depend on unicode and they go wrong because of combining codepoints (they will never get that right). I mean a pre-defined set for every alphabet stored in one database. Even Chinese with some 240+ character variations wouldn't make it difficult to load and use. Pages, paragraphs, even single lines can be easily tagged to be displayed in the right language. The reason things are so complicated is because humans never agree on everything. Unicode is a disaster because it is overly complicated. Most software fail to interpret everything correctly, regardless of encoding. If I install different software packages on Windows, I usually get hundreds of MBs worth of runtimes installed with them. A single database carrying all languages (every country would be responsible for their own set and default font) would be nothing compared to all those runtimes needed to interpret every program on my computer. Disagreement = Inefficient = Complicated = Costly bla bla bla .... ---------------------------------------------------------------------- munair · 2017-12-10 16:15 For everyone who thinks Unicode is great. Even if you only care about English text, there’s more than one Latin alphabet in Unicode! Is “x” equivalent to “𝗑” or “𝘅” or “𝘹” or “𝙭” or “𝚡” or “x” or “𝐱”? What about “×” or “х” or “⨯” or “ⅹ”? Ah, sorry, those last four are actually the multiplication sign, a Cyrillic letter, the symbol for cross product, and the Roman numberal for ten. This is a particularly aggravating problem because most programming languages have facilities for comparing and changing the case of text built in, and most of them are extremely naïve about it. You can’t even correctly change the case of English-looking text without knowing what locale it came from — the title-case of “istanbul” may actually be “İstanbul” depending on language, because of Turkish’s dotted “i”. ---------------------------------------------------------------------- Thaddy · 2017-12-10 16:27 Use UTF32 everywhere. Problem solved... ---------------------------------------------------------------------- JuhaManninen · 2017-12-10 16:50 @Munair, your idea for DB with a language specific ID sounds desperate. It would be more complex and error prone than Unicode. We are just getting rid of locale specific ANSI codepages, let's no bring language specific IDs here. Besides, Unicode is reality. Let's learn it. Most of its complexity is needed for the various human languages. Quote from: tomitomy on December 10, 2017, 01:51:23 pmThat is to say, It is very difficult to be compatible with all languages in the software, Can only compatible with some languages. In this case, it is only enough to consider the single-codepoint character. Depends very much on what you are doing. Sometimes working with codepoint or even codeunit resolution is enough to support the full Unicode. Only if you implement an advanced text layout application of some sort then you need all the details of Unicode. Examples: 1. XML parser. All tags are in lower ASCII. A parser can iterate over codeunits and check for '<' and '>' ASCII characters. Data between tags is copied somewhere and it can include combining codepoints and grapheme clusters and whatever, no problem. 2. Searching, moving and copying verbatim text. (example from wiki) Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function SplitInHalf(Txt, Separator: string; out Half1, Half2: string): Boolean;var  i: Integer;begin  i := Pos(Separator, Txt);  Result := i > 0;  if Result then  begin    Half1 := Copy(Txt, 1, i-1);    Half2 := Copy(Txt, i+Length(Separator), Length(Txt));  end;end;Both the Txt and the Separator can contain complex Unicode, including combining codepoints and grapheme clusters, yet the code has nothing to deal with Unicode specifically. Both examples operate with codeunit (not codepoint) resolution and work regardless of encoding. ---------------------------------------------------------------------- JuhaManninen · 2017-12-10 16:53 Quote from: Thaddy on December 10, 2017, 05:27:43 pmUse UTF32 everywhere. Problem solved... Thaddy, you clearly did not read the thread. The realization was that codepoints and their encodings are not the complicated part of Unicode. Using a fixed width codepoint encoding does not really solve anything. Munair found a good article about it:  https://manishearth.github.io/blog/2017/01/14/stop-ascribing-meaning- to-unicode-code-points/ ---------------------------------------------------------------------- engkin · 2017-12-10 17:29 QuoteIs “x” equivalent to “𝗑” or “𝘅” or “𝘹” or “𝙭” or “𝚡” or “x” or “𝐱”? 𝗑 MATHEMATICAL SANS-SERIF SMALL X 𝘅 MATHEMATICAL SANS-SERIF BOLD SMALL X 𝘹 MATHEMATICAL SANS-SERIF ITALIC SMALL X 𝙭 MATHEMATICAL SANS-SERIF BOLD ITALIC SMALL X 𝚡 MATHEMATICAL MONOSPACE SMALL X x FULLWIDTH LATIN SMALL LETTER X 𝐱 MATHEMATICAL BOLD SMALL X Let's apply PUCUUTF8UpperCase and see the result: QuoteIS “X” EQUIVALENT TO “𝗑” OR “𝘅” OR “𝘹” OR “𝙭” OR “𝚡” OR “X” OR “𝐱”? ---------------------------------------------------------------------- munair · 2017-12-10 18:25 Quote from: JuhaManninen on December 10, 2017, 05:50:15 pm@Munair, your idea for DB with a language specific ID sounds desperate. It would be more complex and error prone than Unicode. We are just getting rid of locale specific ANSI codepages, let's no bring language specific IDs here. Besides, Unicode is reality. Let's learn it. Most of its complexity is needed for the various human languages. In essence, code pages or character sets are not at all bad. Programming becomes simpler when things are predefined. Most languages in the world have a fixed alphabet. Let's take advantage of that and store every language as a character set (together with at least one default font) by means of one universal encoding scheme. Simply do a database call "en" or "cn" to pull out the request character set at any time. The set could then be applied to anything that has the same language ID. No combining codepoints, no multiples of the same character. Errors could only arise from not tagging texts, documents or interfaces. It would be the same as defining a documents encoding. No one has any problem with that either. Programming would be a thousand times simpler. Unicode is considered a success because it heavily relies on the BMP. Outside of that things become more complicated. Another advantage of character sets is that documents in any language would have more or less the same size. That would also be something to aim for. Go to China with this idea and you'll be popular in no-time. Most programming languages are not capable of indexing grapheme clusters and are limited to indexing codepoints - which doesn't mean anything. How many programmers feel (un)comfortable with that? ---------------------------------------------------------------------- JuhaManninen · 2017-12-10 21:46 Quote from: Munair on December 10, 2017, 07:25:55 pmUnicode is considered a success because it heavily relies on the BMP. Outside of that things become more complicated. That is about codepoints and their "planes". It makes a difference with UTF-16 encoding only. For other encodings it is rather irrelevent. Not a big deal ... QuoteMost programming languages are not capable of indexing grapheme clusters and are limited to indexing codepoints - which doesn't mean anything. How many programmers feel (un)comfortable with that? Why would you index grapheme clusters? Even indexing codepoints is seldom useful as we can read from the article you found. ---------------------------------------------------------------------- munair · 2017-12-10 22:06 Quote from: JuhaManninen on December 10, 2017, 10:46:51 pmWhy would you index grapheme clusters? Even indexing codepoints is seldom useful as we can read from the article you found. Sure, do away with any form of... well whatever operations one would need to do on text, cursor movement, selection etc...  Although I believe you know this, I will spell it out here anyway (for those interested)  Take this Tai character ที่, which exists of three codepoints. I say 'character' as it is perceived as such by the Tai. Now do a UTF-8 string operation on Tai text where this characters happens to be near some split point. Without scanning for grapheme clusters, this character may be cut in pieces (which is what most software does): Code: FreeBasic  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i s = "ที่"print st = uleft(s, 2)s = t + uright(s, 1)print sThe character is being put back together again, but may still not be rendered correctly anymore. So the correct way to handle this is indexing and splitting by grapheme clusters... also explained in the good read. ---------------------------------------------------------------------- JuhaManninen · 2017-12-11 20:01 Quote from: howardpc on December 09, 2017, 09:10:42 amI am convinced it is worth the trouble to change them. A comprehensively consistent naming convention for Lazarus functions is nowhere more applicable than in the confusing world of unicode and text encoding functionality. Well, I changed "Character" to "Codepoint" in LazUTF8 function names in r56692. Please take a look. Some existing code may now throw many "deprecated" warnings. Let's see how many complaints we get. ---------------------------------------------------------------------- munair · 2017-12-14 13:52 Quote from: JuhaManninen on December 11, 2017, 09:01:20 pmQuote from: howardpc on December 09, 2017, 09:10:42 amI am convinced it is worth the trouble to change them. A comprehensively consistent naming convention for Lazarus functions is nowhere more applicable than in the confusing world of unicode and text encoding functionality. Well, I changed "Character" to "Codepoint" in LazUTF8 function names in r56692. Please take a look. Some existing code may now throw many "deprecated" warnings. Let's see how many complaints we get. Obviously, programmers had to get used to the idea of codepoints during the last several years, given the heavy use of CHLEN instead of CPLEN as identifier throughout the lazutf8.pas file. ---------------------------------------------------------------------- munair · 2017-12-14 13:59 BTW, how does UTF8IsCombining work, I mean, during what process, because I do not see it anywhere in the lazutf8.pas file, which means that the standard utf8 routines do not combine codepoints with separated diacritical marks. UPDATE: OK, found it in lazunicode.pas ---------------------------------------------------------------------- SymbolicFrank · 2018-01-11 14:07 A long time ago, WordPerfect came to DOS. And you could use it to write German and French characters. If you pressed F3, you could see that they were composed of multiple bytes. And it depended on your graphics adapter if they were visible on the screen. The secretaries had these fancy IBM ball typewriters that could be connected to function as printers, but the problem was that you had to change the letter ball if you wanted different characters. Fortunately, WordPerfect had this fancy new thing called printer drivers, where you could define the translations and control sequences. And so, one of the first things I have done as a (junior) programmer was to make diacritic chars by moving the print position back and printing the ampersands and dots on top of the previous char. Which was extra difficult because those printers knew about proportional printing and variable spacing. But it looked much better than letters printed on a dot matrix printer.