Multilanguage UpCase and LowerCase lazarus / 21-General ---------------------------------------------------------------------- LemonParty · 2025-06-09 12:35 If I need UpCase and LowerCase for other languages than English. Then I should implement separate function for each language or is there some ready solutions for that? ---------------------------------------------------------------------- CM630 · 2025-06-09 12:43 Are there any languages, except Turkish, for which the inbuilt Uppercase and Lowercase are not working properly? ---------------------------------------------------------------------- paweld · 2025-06-09 13:03 UTF8UpperCase and UTF8LowerCase from LazUTF8 unit. ---------------------------------------------------------------------- gues1 · 2025-06-09 13:10 There is no rules about Upper or Lower case in Unicode. This have no sense in many languages (look chinese for example). Do not blindly rely on generic conversions, but you should perform the verification on all languages ​​if supported. ---------------------------------------------------------------------- CM630 · 2025-06-09 13:52 Quote from: gues1 on June 09, 2025, 03:10:38 pm...This have no sense in many languages (look chinese for example)... It makes no sense in Chinese, but does it cause problems? The code below seems to be executed fine. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i procedure TForm1.Button1Click(Sender: TObject);...uses ...LazUTF8......begin  ShowMessage ('汉语'+#13#10+UTF8UpperCase('汉语')+#13#10+UTF8LowerCase('汉语'));  ShowMessage ('юЯ汉语'+#13#10+UTF8UpperCase('юЯ汉语')+#13#10+UTF8LowerCase( 'юЯ汉语'));end; ---------------------------------------------------------------------- gues1 · 2025-06-09 14:18 Quote from: CM630 on June 09, 2025, 03:52:58 pmIt makes no sense in Chinese, but does it cause problems? The code below seems to be executed fine. Of course, he doesn't make any issue. But you don't have what you need. And if in Chinese, like old Greek or Jewish languages or other languages, is not an issue, for other languages may be. Take care that in arabic and other languages the lower o upper case has a particular meaning. Failed to convert can change the meaning of the phrase. Try to convert those in uppercase (uppercase exists of course and is defined as uppercase): 'ӄӥҗ' This is the lower case code of one of them (unicode point): #$04E5 ---------------------------------------------------------------------- gues1 · 2025-06-09 16:15 I found in the PascalScript these functions that in Windows do excelent work: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i function WideUpperCase(const S: WideString): WideString;var  Len: Integer;begin  // CharUpperBuffW is stubbed out on Win9x platofmrs  if Win32Platform = VER_PLATFORM_WIN32_NT then  begin    Len := Length(S);    SetString(Result, PWideChar(S), Len);    if Len > 0 then CharUpperBuffW(Pointer(Result), Len);  end  else    Result := AnsiUpperCase(S);end; function WideLowerCase(const S: WideString): WideString;var  Len: Integer;begin  // CharLowerBuffW is stubbed out on Win9x platofmrs  if Win32Platform = VER_PLATFORM_WIN32_NT then  begin    Len := Length(S);    SetString(Result, PWideChar(S), Len);  if Len > 0 then CharLowerBuffW(Pointer(Result), Len);  end  else    Result := AnsiLowerCase(S);end;  If you use those instead of "UTF8UpperCase", you can view the characters that i show on the last post with uppercase. It works only in Windows, but it works. ---------------------------------------------------------------------- paweld · 2025-06-09 16:47 The only problem I associate with UTF8(Lower/Upper)Case functions is the error that occurs when a lowercase letter (code point) has a different length (number of bytes) than an uppercase letter (code point). ---------------------------------------------------------------------- dsiders · 2025-06-09 16:57 Quote from: paweld on June 09, 2025, 06:47:30 pmThe only problem I associate with UTF8(Lower/Upper)Case functions is the error that occurs when a lowercase letter (code point) has a different length (number of bytes) than an uppercase letter (code point). To my knowledge, differing code point lengths has been addressed in the trunk version (main). There are more than likely still mapping issues for upper- and lowercase conversions... but you thank the unicode standard for that over-engineered nonsense. ---------------------------------------------------------------------- gues1 · 2025-06-09 17:34 Quote from: paweld on June 09, 2025, 06:47:30 pmThe only problem I associate with UTF8(Lower/Upper)Case functions is the error that occurs when a lowercase letter (code point) has a different length (number of bytes) than an uppercase letter (code point). I dont' use very often UpperCase in languages, but I used them (it's for this reason that I know Lazarus has difficulty performing such operations on the characters shown). I worked with other tool in Windows, but I never saw different code lenght in UPPERCASE or LOWERCASE with Windows (I use UTF16). I saw totally different code between them, I think 'cause use of extend and supplemental use of BMP. Quote from: dsiders on June 09, 2025, 06:57:44 pmTo my knowledge, differing code point lengths has been addressed in the trunk version (main). There are more than likely still mapping issues for upper- and lowercase conversions... but you thank the unicode standard for that over-engineered nonsense. May be someone think that unicode is not a good instruments, but I think is the better that we have now. Look at web, only with unicode is possible to saw it in the correct way with every language. Without unicode we still talk about CP ... ---------------------------------------------------------------------- Warfley · 2025-06-09 17:36 Quote from: gues1 on June 09, 2025, 03:10:38 pmThere is no rules about Upper or Lower case in Unicode. This have no sense in many languages (look chinese for example). Do not blindly rely on generic conversions, but you should perform the verification on all languages ​​if supported. Thats not true, the Unicode Consortium defines character properties, including the properties UpperCase and LowrCase. They also provide the Unicode Character Database, which is a collection of text files that contain all that information in machine readable form, containing lists (or more sets of ranges) for all those characters. All the information is already collected and specified there, because no one can be expected to know all the ins and outs of all those languages, it's all done by the unicode consortium for you. How well the Lazarus UTF8 functions follow the unicode standard I don't know though, it could very well be that they are not fully implemented ---------------------------------------------------------------------- Thaddy · 2025-06-09 17:46 Yes, that answer is correct: it is defined in the standard(s). I believe FPC adheres to them, not 100% sure about Lazarus, though, but probably also yes. ---------------------------------------------------------------------- gues1 · 2025-06-09 18:11 Quote from: Warfley on June 09, 2025, 07:36:39 pmThats not true, the Unicode Consortium defines character properties, including the properties UpperCase and LowrCase. They also provide the Unicode Character Database, which is a collection of text files that contain all that information in machine readable form, containing lists (or more sets of ranges) for all those characters. All the information is already collected and specified there, because no one can be expected to know all the ins and outs of all those languages, it's all done by the unicode consortium for you. How well the Lazarus UTF8 functions follow the unicode standard I don't know though, it could very well be that they are not fully implemented You are right, it was lot of time I didn't look at Unicode standards. These is the reference:https://www.unicode.org/faq/casemap_charprop.html I'll give a look to the news (especially arabian lang). Quote from: Thaddy on June 09, 2025, 07:46:22 pmI believe FPC adheres to them, not 100% sure about Lazarus, though, but probably also yes. I already post some characters that aren't converted by Lazarus / FPC. But sure may be improved in the future. EDIT: repost the link, it didn't work. ---------------------------------------------------------------------- CM630 · 2025-06-09 18:37 Without bug reports things might never get fixed. ---------------------------------------------------------------------- gues1 · 2025-06-09 18:59 Quote from: CM630 on June 09, 2025, 08:37:19 pmWithout bug reports things might never get fixed. A bug report should be filled out by someone who knows the topic very well, in order to provide the right input and information. In addition, you should also provide a tip or some additional suggestions for solving the problem. And only a person who knows the development environment and the topic of the problem well can provide this. I'm OT now, so it's better to go on. ---------------------------------------------------------------------- Thaddy · 2025-06-10 05:11 I am not really familiar with the Lazarus provided UTF8 functions but I have a unicode rtl installed and there Uppercase and Lowercase seem to work correctly. That is Unicode-16, though. I do not have a Lazarus installed that makes use of the unicode16 rtl, so your mileage may vary. Also note that misprinted characters may be due to wrong terminal settings: the terminal/console must also support unicode (8 or 16 or 32) so you should always check that first (with e.g. chcp, on linux you may want to install nilfs-tools for that). ---------------------------------------------------------------------- gues1 · 2025-06-10 07:24 I'll start by saying that I know less than zero about Unicode in Lazarus and FPC. I use other tools that don't have these problems (that's why I wasn't updated on Unicode regulations) What I said and wrote concerns Windows, with Lazarus 4.0 stable and FPC 3.2.2 and therefore CP and localization (except for some elements such as punctuation marks) are fortunately a distant memory and fall within historical culture. The functions used are those indicated in the posts, which I believe we use FPC as regards the use of unicode. The malfunction, if you can call it that, perhaps derives from a single simple update not present in the stable version of FPC. However, for those who want to create their own function, they can rely on the attached file (original link at bottom) that matches the code points of all uppercase / lowercase characters. Exceptions such as changes in string length are also reported (I had never encountered any). A simple matching table would solve the UPPERCASE and LOWERCASE, A second file indicates the special cases for those who want to further refine everything. https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt https://www.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt ---------------------------------------------------------------------- Thaddy · 2025-06-10 07:33 I do not see the problem. My fifth language - skipping classics - is Lithuanian which is notoriously difficult with its 32 character alphabet and that simply works with the UTF functions from Lazarus. A post by me about a year or so ago handles simplified Chinese and others. Plz check your terminal, I do not think fpc/lazarus are at fault. Otherwise provide a simple full example on what you are missing and "does not work." Only then I can provide you with an example that solves your coding issue and display issue. I suspect it is just the display issue. ---------------------------------------------------------------------- gues1 · 2025-06-10 08:02 Quote from: Thaddy on June 10, 2025, 09:33:17 amOtherwise provide a simple full example on what you are missing and "does not work." Only then I can provide you with an example that solves your coding issue and display issue. I suspect it is just the display issue. Maybe I didn't explain myself well. But it's not a problem. I have already posted the solution (for Windows), for the community's use, and it is extracted from "Pascalscript". The problem that I had reported here anyway is that with the standard functions present as also posted in this topic, those characters are not converted: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     uses LazUTF8;    procedure TForm1.Button1Click(Sender: TObject);    begin    ShowMessage ('ӄӥҗ'+#13#10+UTF8UpperCase('ӄӥҗ')+#13#10+UTF8LowerCase('ӄӥҗ'));    end; Dont' tell that is a Windows problem, terminal or anything else, use "WideUpperCase" and "WideLowerCase" ... they works. But I don't want to create anything that might cause controversy. For me the question is closed. Bye ---------------------------------------------------------------------- Thaddy · 2025-06-10 08:14 Use the predefined LineEnding instead of #13#10 ---------------------------------------------------------------------- munair · 2025-06-10 09:49 Years ago I ported Lazarus' utf8 upper/lower case conversion routines to a FreeBASIC library and found that not all languages/characters are covered. There are currently 290,000+ unicode characters with data points supporting 160+ languages/scripts. Libraries will probably never be complete. ---------------------------------------------------------------------- CM630 · 2025-06-10 10:01 Quote from: gues1 on June 10, 2025, 10:02:53 amThe problem that I had reported here anyway is that with the standard functions present as also posted in this topic, those characters are not converted: ... Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i     uses LazUTF8;    procedure TForm1.Button1Click(Sender: TObject);    begin    ShowMessage ('ӄӥҗ'+#13#10+UTF8UpperCase('ӄӥҗ')+#13#10+UTF8LowerCase('ӄӥҗ'));    end; ... But I don't want to create anything that might cause controversy. ... Controversy? This seems like nothing but a bug to me. But the better place for reporting is not “here”, but the bugtracker, have you created a report? But here is something that might be controversial: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ShowMessage ('ӄӥҗѝьß'+#13#10+UTF8UpperCase('ӄӥҗѝьß')+#13#10+UTF8LowerCase('ӄӥҗѝьß' ));  AFAIK, there is already a capital eszett, but UTF8UpperCase('ß') converts it to SS (the case is similar to the cyrillic letter ь, which is never used in the beginning of the word, it is even named small “er”.) ---------------------------------------------------------------------- gues1 · 2025-06-10 10:36 Quote from: CM630 on June 10, 2025, 12:01:21 pmControversy? This seems like nothing but a bug to me. If one tell that all is perfect, that its' working 100% and it'is my issue ... what I must say? In this forum when you write something that can help (may be this is the thinking), talking about bug or improvements seems that if one doesn't have more then 1000 posts you want only create problems. So, better stop the conversation. Quote from: munair on June 10, 2025, 11:49:07 amYears ago I ported Lazarus' utf8 upper/lower case conversion routines to a FreeBASIC library and found that not all languages/characters are covered. There are currently 290,000+ unicode characters with data points supporting 160+ languages/scripts. Libraries will probably never be complete. That is right, of course, like always. But me be some better things one can do (or propose). I don't meaning to say "core developers let work" but we can do something, a little bit. Excpecially if "outside" those thing works better. In this case, under Windows the things should works better (actually no sorry, they were already working well), without call "core developers". Quote from: CM630 on June 10, 2025, 12:01:21 pmCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i ShowMessage ('ӄӥҗѝьß'+#13#10+UTF8UpperCase('ӄӥҗѝьß')+#13#10+UTF8LowerCase('ӄӥҗѝьß' ));  AFAIK, there is already a capital eszett, but UTF8UpperCase('ß') converts it to SS (the case is similar to the cyrillic letter ь, which is never used in the beginning of the word, it is even named small “er”.) 'SS' is the right uppercase from standard (codepoint $00DF -> $0073 $0073). And it is one of the case that the lenght grows. Other forms are (the graphics come form a PDF and are not rights): Lowercase letters Quote00DF ß LATIN SMALL LETTER SHARP S = Eszett • German • not used in Swiss High German • uppercase is “SS” (standard case mapping), alternatively 1E9E ẞ • typographically the glyph for this character can be based on a ligature of 017F ſ with either 0073 s or with an old-style glyph for 007A z (the latter similar in appearance to 0292 ʒ ). Both forms exist interchangeably today. → 017F ſ latin small letter long s → 0292 ʒ latin small letter ezh → 03B2 β greek small letter beta → 1E9E ẞ latin capital letter sharp s → A7B5 ꞵ latin small letter beta → A7D7 ꟗ latin small letter middle scots s ---------------------------------------------------------------------- CM630 · 2025-06-10 11:03 Quote from: gues1 on June 10, 2025, 12:36:47 pm... If one tell that all is perfect, that its' working 100% and it'is my issue ... what I must say? In this forum when you write something that can help (may be this is the thinking), talking about bug or improvements seems that if one doesn't have more then 1000 posts you want only create problems. So, better stop the conversation. ... I have 1385 pots in this forum, in approx 1300 I have asked questions, many of them were stupid. 90 % of my questions were answered (with the remaining 85 posts I have tried to help someone else). If you are restrained for some reason to create bug reports yourself, give me the list of the inhandled characters, I will create the report. It might be rejected, but it might be accepted, fully or partially. Indeed, if it is not, I would rather stop using the inbuilt *case routines, as long as a working 3th party solution is present. ---------------------------------------------------------------------- Thaddy · 2025-06-10 11:14 Quote from: munair on June 10, 2025, 11:49:07 amYears ago I ported Lazarus' utf8 upper/lower case conversion routines to a FreeBASIC library and found that not all languages/characters are covered. There are currently 290,000+ unicode characters with data points supporting 160+ languages/scripts. Libraries will probably never be complete. Lazarus and Fpc use the official tables ad verbatum so that is not the case, even false information. If it does not work it is your fault, ---------------------------------------------------------------------- Warfley · 2025-06-10 11:14 Quote from: gues1 on June 10, 2025, 12:36:47 pmIf one tell that all is perfect, that its' working 100% and it'is my issue ... what I must say? In this forum when you write something that can help (may be this is the thinking), talking about bug or improvements seems that if one doesn't have more then 1000 posts you want only create problems. So, better stop the conversation. This Forum is also not the place to report bugs, first it will spark discussions and will not end up on any todo list of the developers until it's posted to the gitlab issues. So the best way is just to skip that and just make a bug report in the gitlab. Just give a short explanation, an example of what goes wrong and what the expected result should be and you should be fine. Here's the link for Lazarus or LCL: https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues And for FPC, RTL and FCL: https://gitlab.com/freepascal.org/fpc/source/-/issues ---------------------------------------------------------------------- CM630 · 2025-06-10 11:49 Quote from: Warfley on June 10, 2025, 01:14:53 pm... + 1 Quote from: Thaddy on June 10, 2025, 01:14:37 pm... If it does not work it is your fault, It does not work and it is not his/her fault. If the standard says indeed that it shall not work, the fault is in (those who made) the standard. Whether FPC/Lazarus shall follow a faulty standard is another issue. ---------------------------------------------------------------------- gues1 · 2025-06-10 12:28 Quote from: Warfley on June 10, 2025, 01:14:53 pmQuote from: gues1 on June 10, 2025, 12:36:47 pmIf one tell that all is perfect, that its' working 100% and it'is my issue ... what I must say? In this forum when you write something that can help (may be this is the thinking), talking about bug or improvements seems that if one doesn't have more then 1000 posts you want only create problems. So, better stop the conversation. This Forum is also not the place to report bugs, first it will spark discussions and will not end up on any todo list of the developers until it's posted to the gitlab issues. So the best way is just to skip that and just make a bug report in the gitlab. Just give a short explanation, an example of what goes wrong and what the expected result should be and you should be fine. Here's the link for Lazarus or LCL: https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues And for FPC, RTL and FCL: https://gitlab.com/freepascal.org/fpc/source/-/issues May be I don't speak a comprensible English: I already express my point, no one except @CM630 and me (I think is an incomplete work) say that this is a bug, one the most evaluable member of this forum said that all is OK and it's my fault, what report I should make ? I'm the last of the world that should open a bug report. I don't know nothing about Lazarus / FPC, so from me no report is going to be open. Like you told this is a forum where to talk about this things and I never meant (I wrote about this) to call the "core developers" here. At the end, for me is that is not an issue and works as expected (bad but as expected). ---------------------------------------------------------------------- CM630 · 2025-06-10 12:39 Some other people also told you: forum ≠ bugtracker Just give me the list of what is wrong, and I will write the bug report. Meanwhile in Python: ---------------------------------------------------------------------- paweld · 2025-06-10 12:40 Quote from: gues1 on June 10, 2025, 02:28:38 pmI already express my point, no one except @CM630 and me (I think is an incomplete work) say that this is a bug, one the most evaluable member of this forum said that all is OK and it's my fault, what report I should make ? I'm the last of the world that should open a bug report. I don't know nothing about Lazarus / FPC, so from me no report is going to be open. Like you told this is a forum where to talk about this things and I never meant (I wrote about this) to call the "core developers" here. At the end, for me is that is not an issue and works as expected (bad but as expected). No one said it's not a bug (except maybe @Thaddy), it's just that not everyone uses all unicode code points to be able to observe such a problem. According to what you wrote that you encountered problems with Cyrillic and Arabic and Chinese characters, you are the best person to report the error. In the report, you should state in what cases the error manifests itself (sample characters), and it is good to state what the correct behavior should be - the more information the better. ---------------------------------------------------------------------- munair · 2025-06-10 13:06 Quote from: Thaddy on June 10, 2025, 01:14:37 pmLazarus and Fpc use the official tables ad verbatum so that is not the case, even false information. If it does not work it is your fault, No. ---------------------------------------------------------------------- Thaddy · 2025-06-10 14:46 Yes. Examine the sources....Which you did not do.... (amateur) If you have troubles it is your fault.... I am confining this to FPC, not Lazarus, although I expect no difference. When you get stuck, look at the sourcecode and then you can make bold remarks: The tables are there and - to my knowledge - correct. Otherwise they are deferred to the OS. The tables are machine generated from the standard of course. There is no such thing as a FreePascal or Lazarus specific implementation. ---------------------------------------------------------------------- etrusco · 2025-06-11 04:31 Quote from: Thaddy on June 10, 2025, 04:46:14 pm(...) I am confining this to FPC, not Lazarus, although I expect no difference. (...) The tables are machine generated from the standard of course. There is no such thing as a FreePascal or Lazarus specific implementation. FPC's implementation indeed seems to be generated from CLDR data (in unicodedata.pas and related includes) but the version is not documented and it's not clear how it was generated (except that the author is Inoussa Ouedraogo). I expected AnsiUpperCase() to work, but only UnicodeUpperCase(UTF8Decode()) had an effect. Lazarus' LazUTF8 unit is an adaptation with a single case/switch (and an extra check for 'I' if you pass 'tr' in Language parameter). ---------------------------------------------------------------------- etrusco · 2025-06-11 04:43 Quote from: LemonParty on June 09, 2025, 02:35:36 pmIf I need UpCase and LowerCase for other languages than English. Then I should implement separate function for each language or is there some ready solutions for that? UpperCase() and LowerCase() only convert chars from A to Z. This is documented in both Delphi and FPC. ---------------------------------------------------------------------- Thaddy · 2025-06-11 06:26 Use AnsiUppercase/AnsiLowercase instead. https://www.freepascal.org/docs-html/rtl/sysutils/ansiuppercase.html ---------------------------------------------------------------------- CM630 · 2025-06-11 08:33 . ---------------------------------------------------------------------- munair · 2025-06-11 09:04 Quote from: Thaddy on June 10, 2025, 04:46:14 pmIf you have troubles it is your fault.... I am confining this to FPC, not Lazarus, although I expect no difference. When you get stuck, look at the sourcecode and then you can make bold remarks: You're assuming too much. I was referring specifically to LazUTF8. I ported the entire upper- /lowercase functions and I didn't "get stuck" in doing that. They simply didn't cover every language (some only partially) and my ported functions had exactly the same functionality. ---------------------------------------------------------------------- CM630 · 2025-06-11 09:35 Well, the issue occurrs to be in the bugtracker: https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/41444 Also: https://forum.lazarus.freepascal.org/index.php/topic,69704.0.html