Questions of new Strings in FPC 3.0
lazarus / 67-Windows

----------------------------------------------------------------------
Michl · 2015-10-29 09:21

Hi,

I have read a lot about the new strings and understood most of it and
the ideas behind it. I've build a lot of test cases for my self and
test a lot of conversions of strings, but I still have some questions.

Shortstring: The code page of a shortstring is implicitly CP_ACP and
hence will always be equal to the current value of
DefaultSystemCodePage.  So I make a test:Code: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1; //{$codepage cp1252}//{$mode ObjFPC}{$H+}//{$modeswitch
systemcodepage} const  StrCP1252 = #$80#$C4#$D6#$8C#$A5;// CP1252    
€   Ä   Ö   Œ   ¥// CP437      Ç   ─   ╓   î   Ñ// 1. output  ?   Ä  
Ö   O   ¥// 2. output  Ç   ─   Í   î   Ñ var  s: String; begin 
writeln(DefaultSystemCodePage);  s := StrCP1252;  writeln(s);        
     // expected: €ÄÖŒ¥   get: ?ÄÖO¥  writeln(StrCP1252);       //
expected: €ÄÖŒ¥   get: Ç─ÍîÑ  writeln(Char(#$80), Char(#$C4),
Char(#$D6), Char(#$8C), Char(#$A5));end.  
The output is:Quote1252
?ÄÖO¥
Ç─ÍîÑ
Ç─ÍîÑ
My Codepage is 1252. The output of ShortStrings is more similar to the
codepage 437. With the assigning of the ShortString to a String, I got
nearly a CP1252 string.

It doesn´t matter if I define/not define:Code: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
//{$codepage cp1252}//{$mode ObjFPC}{$H+}//{$modeswitch
systemcodepage}
I expected, if I define copepoints (Bytes > 127) of a CP1252 string, I
got the characters shown here:
https://en.wikipedia.org/wiki/Windows-1252

Why are in the output other characters then explained in the wiki (I
got ? instead € and O instead Œ)?
Why is WriteLn(SomeShortString) not the same as WriteLn(SomeString)?

My system: Windows 7, 64bit, FPC 3.1.1 32bit r32092
Compileroptions: -MObjFPC -Scghi -O1 -g -gl -l -vewnhibq
-Filib\i386-win32 -Fu. -FUlib\i386-win32

----------------------------------------------------------------------
Cyrax · 2015-10-29 12:46

Quote from: Michl on October 29, 2015, 10:21:52 am...
Compileroptions: -MObjFPC -Scghi -O1 -g -gl -l -vewnhibq
-Filib\i386-win32 -Fu. -FUlib\i386-win32

You are compiling your test project via Lazarus. That is why there is
no difference if you undefine those compiler conditional settings. You
should try compiling your test program via command line.

----------------------------------------------------------------------
Michl · 2015-10-29 13:46

You are right, I can't deactivate such FPC conditional settings. If I
compile from command line the result of enabled/disabled {$mode
ObjFPC}{$H+} is a other, cause in one case "s" is a ShortString on the
other "s" is a String. Thank you for that hint.

But my questions are still here. Can anyone give me a hint for:
Why are in the output other characters then explained in the wiki (I
got ? instead € and O instead Œ)?
Why is WriteLn(SomeShortString) not the same as WriteLn(SomeString)
with {$mode ObjFPC}{$H+}?

----------------------------------------------------------------------
Cyrax · 2015-10-29 14:05

WriteLn is compiler intrinsic subroutine which will be replaced during
compile time to calls real subroutine. In case of
WriteLn(SomeShortString) there will be type specific call and call to
subroutine which will print EOL to standard output. You will see this
yourself by examining disassembler output via  View->Debug
Windows->Assembler in Lazarus.

----------------------------------------------------------------------
Michl · 2015-10-29 14:28

OK I understand.

I make a new test:Code: Pascal  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i program project1; //{$codepage cp1252}{$mode ObjFPC}{$H+} var 
s: String;  s2: ShortString;  f: file of byte;  ftext:
TextFile; begin  AssignFile(f, 'test.txt');  Rewrite(f);  write(f,
$80);  write(f, $C4);  write(f, $D6);  write(f, $8C);  write(f, $A5); 
CloseFile(f);   AssignFile(ftext, 'test.txt');  Reset(ftext); 
Read(ftext, s);  Reset(ftext);  Read(ftext, s2);  CloseFile(ftext); 
writeln(s);   // expected: €ÄÖŒ¥   Console Output: ?ÄÖO¥  writeln(s2);
 // expected: €ÄÖŒ¥   Console Output: Ç─ÍîÑ   AssignFile(ftext,
'teststring.txt');  Rewrite(ftext);  write(ftext, s); 
CloseFile(ftext);   AssignFile(ftext, 'testshortstring.txt'); 
Rewrite(ftext);  write(ftext, s2);  CloseFile(ftext);end.
If I now inspect the three files with Windows Notepad. They all have
the same and correct content (€ÄÖŒ¥). Now it is seems so, that there
is no difference between Write(SomeShortString) and Write(SomeString)
also for WriteLn.

If I compile the project "project1.exe > testconsole.txt" there are
the wrong chars in that file.

So only the Console output is wrong (the output for the ShortString
and/or the String). Should I report that to the bugtracker or is it a
known problem?

----------------------------------------------------------------------
Cyrax · 2015-10-29 14:44

You might find attached source file educating in this matter. It will
demonstrate how to change codepage of standard output in console
application (also its font) so characters will show correctly.

Unfortunately it is Windows only.

----------------------------------------------------------------------
JuhaManninen · 2015-10-29 15:22

Quote from: Cyrax on October 29, 2015, 03:44:19 pmYou might find
attached source file educating in this matter. It will demonstrate how
to change codepage of standard output in console application (also its
font) so characters will show correctly.

I guess Michl wanted to find a solution using system codepage, without
using UTF-8.

----------------------------------------------------------------------
Michl · 2015-10-29 15:44

Quote from: Cyrax on October 29, 2015, 03:44:19 pmYou might find
attached source file educating in this matter. It will demonstrate how
to change codepage of standard output in console application (also its
font) so characters will show correctly.

Unfortunately it is Windows only.
Thank you very much - it is realy interesting!!! With this unit, the
output for Strings works now, but it is a UTF8 string, not a CP1252.
Maybe the answer near, I'll play with it a little bit

Quote from: JuhaManninen on October 29, 2015, 04:22:19 pmI guess Michl
wanted to find a solution using system codepage, without using
UTF-8.You are right. A solution is, to write all data into a file, the
bytes there I can trust, the shown output in the console not.

And hey, that is just the first question, and I have created a lot of
test cases

----------------------------------------------------------------------
Cyrax · 2015-10-29 15:45

System code page is different than console code page. That is why
Michl demo project fails.

If I execute command chcp in command prompt, it says that Active code
page: 850.

----------------------------------------------------------------------
marcov · 2015-10-29 15:51

Console is in OEMSTRING:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
OemString = type AnsiString(CP_OEMCP);
if not already predefined.

----------------------------------------------------------------------
Cyrax · 2015-10-29 15:59

Check out attached demo project. It should shine some light for this
matter.

----------------------------------------------------------------------
Cyrax · 2015-10-29 17:07

And I think I managed to solve Michl's problem. Downside is that you
need to change console font every time when you execute your console
program and maybe restore to default font.

See attached project for more info.

----------------------------------------------------------------------
Michl · 2015-10-29 17:08

Quote from: Cyrax on October 29, 2015, 04:59:00 pmCheck out attached
demo project. It should shine some light for this matter.
You make the goal! The shown codepage in the console is cp850 for the
files cp1252. 

If I change the project to something likeCode: Pascal
 [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i program
project1; {$codepage cp1252}{$mode ObjFPC}{$H+} uses windows; var  s:
String;  s2: ShortString; begin  System.SetTextCodePage(Output,
1252);  s := #$80#$C4#$D6#$8C#$A5;  s2 := #$80#$C4#$D6#$8C#$A5; 
writeln(s);   // get: Ç─ÍîÑ  writeln(s2);  // get: Ç─ÍîÑend.
I now got the same result for both, ShortString and String. It is a
string represented by chars of codepage 850.

Quite confusing. I must first digest it.



Quote from: marcov on October 29, 2015, 04:51:23 pmConsole is in
OEMSTRING:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i type 
OemString = type AnsiString(CP_OEMCP);
if not already predefined.
A direct assigning of a constant to that new defined string doesn't
work, only from a other string. But this is a new topic.


For me it was important to understand, if the statement "Shortstring:
The code page of a shortstring is implicitly CP_ACP and hence will
always be equal to the current value of DefaultSystemCodePage." is
correct.
For files it is.
For the console too, but you can't get the displayed codepage with
DefaultSystemCodePage, but with System.GetTextCodePage(Output).


Thank you all! For me this topic is solved!

I think, I'll create a new thread for a new topic.

----------------------------------------------------------------------
Michl · 2015-10-29 17:23

Quote from: Cyrax on October 29, 2015, 06:07:10 pmAnd I think I
managed to solve Michl's problem. Downside is that you need to change
console font every time when you execute your console program and
maybe restore to default font.

See attached project for more info.
Very nice!!! The name of the unit isn't correct any more

Maybe it is good for a feature request, that the codepage for the
console is the same as for files, the DefaultSystemCodePage?

----------------------------------------------------------------------
Michl · 2015-10-30 08:30

A new day, new tests.

Your project compiles fine on command line. If I compile your project
with Lazarus I got a Runerror 103 on line "  WriteLn(StrCP1252);    ".
I understand also why, cause your project file is saved with CP1252 -
the DefaultSystemCodePage.

If I create an new program with Lazarus and copy the content of your
project file in the new project file and compile it, it works but now
the String "  WriteLn('€ÄÖŒ¥'); " is a UTF8-String. If I compile the
new created project on command line there is the same, cause Lazarus
saves the project file as a UTF8-File.

So here we have the issue, that with Lazarus it is never possible to
build clean CP_APC projects, cause Lazarus saves the project file as
UTF8, so the compiler interprets all the time the saved strings as
UTF8. It is also not possible to compile such a UTF8 project from
command line, the result is the same. I'm right???

----------------------------------------------------------------------
JuhaManninen · 2015-10-30 08:50

Quote from: Michl on October 30, 2015, 09:30:27 amSo here we have the
issue, that with Lazarus it is never possible to build clean CP_APC
projects, cause Lazarus saves the project file as UTF8,

You can change the encoding in editor File settings -> Encoding.

Quoteso the compiler interprets all the time the saved strings as
UTF8.

Without {$codepage utf8} / -FcUTF8 it should follow the file's
excoding.

----------------------------------------------------------------------
Michl · 2015-10-30 09:02

Thanky you! I didn't know that (I have seen that though but forget
again ).

But now the own created project isn't compileable and the project from
cyrax breaks with a runerror. I will try further and if I identified
the problem, I'll make a minimal example and post it in the
bugtracker, if it is necessary.

----------------------------------------------------------------------
Michl · 2015-10-30 12:26

Quote from: Michl on October 30, 2015, 10:02:59 am I'll make a minimal
example and post it in the bugtracker, if it is necessary.Here it is
http://bugs.freepascal.org/view.php?id=28934

----------------------------------------------------------------------
pascalpopper · 2015-12-08 04:07

I found it helpful to put my UTF8 and UTF16 test data into files on
disk, which I could check with multiple editors, to make sure they
were in particular formats and encodings, before involving FPC (or
Delphi).  For me, that obviates the need to declare string constants
with complex encodings and worry whether Lazarus etc is modifying the
contents as intended.  Please see attached -- a selection of 1-2 kbyte
files.   The file extension is .utx so you can connect the files with
the editor of your choice.  Note that tests where the file name
includes non-Latin letters can prove more difficult, which is why
there are a couple file *names* in Russian and Arabic.

The downside to this approach is that the file i/o has to be rock
solid, and I am stuck there with my FPC 3.0 tests, which is why I
wanted to jump in here.  This in particular seem "wrong" compared to
Delphi's idea:

* TStringList.Load  of a file, where the file contains a UTF16 Byte
Order Mark ("BOM") does not seem to be able to load that data directly
into a string (UnicodeString).  The length of the data is roughly 2x
larger than I would expect.  Measured with Length(), Delphi gives a
length of 21 and FPC gives a length of 43.  This is for the attached
chinese.16.utx file. 

A quote from the Delphi docwiki: "If the Encoding parameter is not
given, then the strings are loaded using the appropriate encoding. The
value of the encoding is obtained by calling the GetBufferEncoding
routine of the TEncoding class."  ( http://docs.embarcadero.com/produc
ts/rad_studio/delphiAndcpp2009/HelpUpdate2/EN/html/delphivclwin32/Clas
ses_TStrings_LoadFromFile@string.html )

In terms of sanity checking the contents of files, I can highly
recommend two editors: TedNPad (for very quick display of encoding and
BOM on the status bar) and Unipad (for excellent display of details
about individual chars).

This is my function for loading via TStringList - confirmed working in
Unicode Delphi for quite a few years:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i function
TTest_ucLogFil.TStringList_File_To_String(  const InFilespec: string):
string;var  y: TStringList;begin  y := nil;  Result := '';   try    y
:= TStringList.Create;    y.LoadFromFile(InFilespec);    // strip
trailing CRLF, which was not in the disk file    Result :=
Copy(y.Text, 1, Length(y.Text) - 2);  finally    FreeAndNil(y); 
end;end; 
To summarize, FPC 3 is not loading the chinese.16.utx file the way I
think it should.