[Solved] Unicode Guidance
lazarus / 31-LCL

----------------------------------------------------------------------
Mike.Cornflake · 2018-08-29 20:30

My second ever unicode issue, and the first since Lazarus 0.9.  Things
have changed and I haven't kept up :-(  Knew it would bite me one
day. 

I have simplified an issue in my software down to the following code

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i Procedure
TForm1.Button1Click(Sender: TObject);Var  oTemp: TStringList;Begin 
oTemp := TStringList.Create;  oTemp.Add('-15µV/cm'); 
oTemp.SaveToFile('C:\TEMP\Test.html');  oTemp.Free;End;  

If I open Test.html in notepad, I see the correct text.
If I open Test.html in firefox (latest) or IE (ancient) I see
incorrect text: -15µV/cm

I know nothing about file or character encoding.   Is there some flag
I need to set on TStringList to get this working?

I'm also not sure what info I need to provide.  Lazarus SVN 58055, fpc
3.0.4.  Windows 7 set to Australia settings.  Anything else?

So yeah.  Help :-)

Many thanks

Mike

----------------------------------------------------------------------
taazz · 2018-08-29 20:41

try adding a utf8 BOM as the first line of the file and see if firefox
changed behavior. If it does then its firefox (correctly) trying to
convert from ansi to utf8 an already utf8 string.

----------------------------------------------------------------------
Mike.Cornflake · 2018-08-29 20:49

Err, how do I do that then?  I can change encoding using Notepad++,
but that doesn't seem to persist.    And I'm not seeing an option in
TStringList either. 

Thanks

----------------------------------------------------------------------
Mike.Cornflake · 2018-08-29 20:52

Ah, found it :-)

Code: [Select]Procedure TForm1.Button1Click(Sender: TObject);
Var
  oTemp: TStringList;
Begin
  oTemp := TStringList.Create;
  oTemp.Add(Chr($EF) + Chr($BB) + Chr($BF));
  oTemp.Add('-15µV/cm');
  oTemp.SaveToFile('C:\TEMP\Test.html');
  oTemp.Free;
End; 

Thanks, this works.

Update:  I've got this in my app for now, so that takes that monkey
off my back.  But this feels like a hack.  I'm busy now, but will try
and revisit this in a few weeks.  This feels more like a patch against
TStringList.SaveToFile is required (with sort of optional encoding?)

----------------------------------------------------------------------
taazz · 2018-08-29 21:05

glad to be of service and to complete the info, here is a link for the
byte order mark (BOM) https://en.wikipedia.org/wiki/Byte_order_mark .

Quote from: Mike.Cornflake on August 29, 2018, 10:52:16 pmUpdate: 
I've got this in my app for now, so that takes that monkey off my
back.  But this feels like a hack.  I'm busy now, but will try and
revisit this in a few weeks.  This feels more like a patch against
TStringList.SaveToFile is required (with sort of optional encoding?)
based on the delphi TTstringlist I bet there are already plans to add
an overloaded savetofile that has an encoding parameter but I have no
idea what they plan to do with BOM in my part of the world it is not
used at all it is a waste of space the encoding is part of the file
metadata that saves time and guessing.

----------------------------------------------------------------------
JuhaManninen · 2018-08-30 00:01

The browser interprets it as Windows codepage although UTF-8 is most
often used in HTML files.
You should have a <meta>-tag there:
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
A HTML file should also have <html>, <head> and <body> tags.

Your Pascal code is OK. UTF-8 is used automatically also in
TStringList.
I remember copying this link to you earlier but here comes again:
 http://wiki.freepascal.org/Unicode_Support_in_Lazarus

----------------------------------------------------------------------
PascalDragon · 2018-08-31 19:30

Quote from: taazz on August 29, 2018, 11:05:44 pmQuote from:
Mike.Cornflake on August 29, 2018, 10:52:16 pmUpdate:  I've got this
in my app for now, so that takes that monkey off my back.  But this
feels like a hack.  I'm busy now, but will try and revisit this in a
few weeks.  This feels more like a patch against
TStringList.SaveToFile is required (with sort of optional encoding?)
based on the delphi TTstringlist I bet there are already plans to add
an overloaded savetofile that has an encoding parameter but I have no
idea what they plan to do with BOM in my part of the world it is not
used at all it is a waste of space the encoding is part of the file
metadata that saves time and guessing.
That was already added in 3.1.1, thus will be part of 3.2.0.

----------------------------------------------------------------------
lucamar · 2018-08-31 20:21

An even better solution is to configure Firefox to interpret unknowns
as UTF-8. I can't remember where the option is exactly*, but it's
there---I have it set to ISO-8859-15


-----
* Launching FF in this machine is a process that takes more than 10
minutes, so excuse me if I don't do it just to look this.