UpperCase, ToUpper, LowerCase and ToLower bug or my misunderstanding?
lazarus / 64-FPC development

----------------------------------------------------------------------
avra · 2021-07-15 10:21

This program:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
testcase;{$H+} uses  {$IFDEF WINDOWS}windows,{$ENDIF} sysutils; var 
PreviousValue: uint;  MyStr: string;begin  {$IFDEF WINDOWS}   
PreviousValue := GetConsoleOutputCP;    DefaultSystemCodePage :=
CP_UTF8;    SetConsoleOutputCP(CP_UTF8);  {$ENDIF}   MyStr :=
'Укључено';  WriteLn('PascalCase Жељко'.ToUpper); 
WriteLn(MyStr.ToUpper);  WriteLn(UpperCase(MyStr)); 
WriteLn(AnsiUpperCase(MyStr)); 
WriteLn(UTF8String('Укључено').ToUpper); 
WriteLn(UpperCase(UnicodeString(Utf8ToString('Укључено')))); 
WriteLn(UpperCase(Utf8ToString('Укључено')));   {$IFDEF WINDOWS}   
SetConsoleOutputCP(PreviousValue);  {$ENDIF}end. 
produces this output:
Quotem:\Lazarus\testing>testcase
PASCALCASE Жељко
Укључено
Укључено
Укључено
Укључено
Укључено
Укључено

As you can see, whatever I tried output is upper cased only for ASCII
characters in a string. If 'Укључено' can be shown properly in output,
then I think that upper cased  'УКЉУЧЕНО' should be shown on at least
one output line above. Is this a bug or my misunderstanding?

Tested output on Win10 x64 console with trunk 32-bit FPC.

----------------------------------------------------------------------
Kays · 2021-07-15 10:42

Quote from: avra on July 15, 2021, 12:21:33 pm[…] As you can see,
whatever I tried output is upper cased only for ASCII characters in a
string. […]And this is “correct” because only this functionality is
implemented. The great thing about open source is you can investigate
it it on your own: toUpper calls upperCaseCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TStringHelper.ToUpper: string;begin  Result:=UpperCase(Self);end;which
callls sysUtils.upperCaseCode: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i class function TStringHelper.UpperCase(const S: string):
string;begin  Result:=sysutils.Uppercase(S);end;which only looks at
the range 'a'..'z'. The description of upperCase even says so. It is
understandable though that just looking at the documentation of
toUpper you might expect all Unicode code points to be treated
correctly. The behavior of ANSIUpperCase depends on the utilized wide
string manager, so this may be different.

----------------------------------------------------------------------
lucamar · 2021-07-15 11:00

Untested, but try with UTF8UpperCase() or UpperCaseString() from
LazUTF8 (LazUtils package); both should work (the last is the inined
version of the former).

----------------------------------------------------------------------
avra · 2021-07-15 11:23

@Kays: I must admit that I didn't expect to see that FPC does not have
any Upper derivate that goes beyond ASCII. Maybe there is a way to let
OS handle it?

Quote from: lucamar on July 15, 2021, 01:00:22 pmUntested, but try
with UTF8UpperCase() or UpperCaseString() from LazUTF8 (LazUtils
package); both should work (the last is the inined version of the
former).Thanks, but these are not available from just RTL SysUtils and
console where I need them at the moment.

The main reason why I consider it as a potential bug is this
behaviour:

AnsiUpperCase() works in Lazarus (look at bithelpers demo from my
signature):
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Memo1.Append(AnsiUpperCase('Укључено')); // output is 'УКЉУЧЕНО' as it
should be
but does not work in FPC:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
WriteLn(AnsiUpperCase('Укључено')); // output is 'Укључено' which is
wrong since it's not upper cased
Strange, right?

----------------------------------------------------------------------
ASerge · 2021-07-15 15:20

Quote from: avra on July 15, 2021, 01:23:29 pmAnsiUpperCase() works in
Lazarus (look at bithelpers demo from my signature):
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
Memo1.Append(AnsiUpperCase('Укључено')); // output is 'УКЉУЧЕНО' as it
should be
but does not work in FPC:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
WriteLn(AnsiUpperCase('Укључено')); // output is 'Укључено' which is
wrong since it's not upper casedOn Windows, SysUtils.AnsiUpperCase
calls the native CharUpperBuffA, which only works with ANSI, but the
string contains UTF8.
When you add LazUtils, it replaces the AnsiUpperCase call with
UTF8UpperCase, which understands UTF8.

----------------------------------------------------------------------
marcov · 2021-07-15 15:28

Note in the icon/manifest project section there is a tick to use the
newer Windows 10 utf8 default option

----------------------------------------------------------------------
Kays · 2021-07-15 15:35

Quote from: avra on July 15, 2021, 01:23:29 pm@Kays: I must admit that
I didn't expect to see that FPC does not have any Upper derivate that
goes beyond ASCII. Maybe there is a way to let OS handle it?OS? No,
that ain’t no task of an OS.
Quote from: avra on July 15, 2021, 01:23:29 pm[…] Strange, right?No,
because, as mentioned above, the behavior of ANSIUpperCase depends on
the utilized wide string manager. The following works on a Linux
system:Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
cyrillicUpcase(input, output, stdErr);uses        {$ifDef UNIX}      
        CWString,        {$endIf} sysUtils;begin       
writeLn(ANSIUpperCase('Укључено'));end.

----------------------------------------------------------------------
avra · 2021-07-15 20:25

Quote from: ASerge on July 15, 2021, 05:20:59 pmWhen you add LazUtils,
it replaces the AnsiUpperCase call with UTF8UpperCase, which
understands UTF8.I did not use LazUtils anywhere in bithelpers demo,
but maybe some of the used units uses it. Thanks for the info!

Quote from: marcov on July 15, 2021, 05:28:49 pmNote in the
icon/manifest project section there is a tick to use the newer Windows
10 utf8 default optionI was not aware of that. Thanks! That can not be
applied to console applications, right?

Quote from: Kays on July 15, 2021, 05:35:46 pmQuote from: avra on July
15, 2021, 01:23:29 pm@Kays: I must admit that I didn't expect to see
that FPC does not have any Upper derivate that goes beyond ASCII.
Maybe there is a way to let OS handle it?OS? No, that ain’t no task of
an OS.So what are functions like CharUpperBuffW() or LCMapStringA()
doing in Windows API?

Quote from: Kays on July 15, 2021, 05:35:46 pmbehavior of
ANSIUpperCase depends on the utilized wide string manager. The
following works on a Linux systemThanks! It's more clear now.

So, it's not a bug after all...

----------------------------------------------------------------------
ttomas · 2021-07-15 20:32

@avra, look at LazUtils as FPC library, not as LCL dependency. Maybe
is good to move LazUtils to FPC or I'm wrong?

----------------------------------------------------------------------
marcov · 2021-07-15 21:01

Quote from: avra on July 15, 2021, 10:25:35 pmQuote from: marcov on
July 15, 2021, 05:28:49 pmNote in the icon/manifest project section
there is a tick to use the newer Windows 10 utf8 default optionI was
not aware of that. Thanks! That can not be applied to console
applications, right?
[/quote]

Afaik it can.