UTF-8 and Laz 2.0.10/fpc 3.2.0  difference?
lazarus / 16-Databases

----------------------------------------------------------------------
OH1KH · 2021-02-10 09:32

HI!

PC has Fedora 32 with env setting as LANG=fi_FI.UTF-8

I have a program compiled with Laz2.0.8/fpc3.0.4 where I write to
TEdit a string that contains some special characters like ÖÄÅöäå.
Content is then saved to mysql varchar column.

If I look column content with console mysql it is "garbage" I.E every
special char is different and doubled in database column.
How ever  when it is taken back to TEdit all looks same ÖÄÅöäå as it
was written.

Without any change I compile same source with Laz 2.0.10/fpc3.2.0 and
with that write to TEdit a string that contains same special
characters like ÖÄÅöäå, and put them to database.
The console mysql view shows same "garbage" content in database column
as before.

If column is taken back to TEdit it shows out now same garbage, not
proper text as with Laz2.0.8/fpc3.0.4 compiled version.

What is changed?
I have tried to find a glue. No luck.
It looks like reading from database does not do the same conversion as
with older laz/fpc

----------------------------------------------------------------------
trev · 2021-02-10 10:16

Did you check the FPC 3.2.0 release notes: User Changes and New
Features?

This  User Changes breaking change looks like it may be relevant.

[Edit: Fix URL]

----------------------------------------------------------------------
OH1KH · 2021-02-10 13:46

Yes I did.
Several times and also this part, but I did not understand what to do
to resume old way functionality.

By my testing that has somehow "one direction" effect.
Strings written to database with Laz2.0.10/fpc3.2.0  return completely
readable if they are read back to TEdit with Laz2.0.8/fpc3.0.4
compiled version.

https://github.com/ok2cqr/cqrlog/issues/323

----------------------------------------------------------------------
trev · 2021-02-11 06:30

So now you have your answer to what has changed which was documented
in the release notes. Those notes are always useful reading.

----------------------------------------------------------------------
OH1KH · 2021-02-11 14:49

Hi !
There is no answer how to turn on compatibility to old way!

Is it really so that I have to change nearly 900 lines of code from
XXXXX.AsString to XXXXX.AsBinary and then pass result to function loop
that creates string without conversion ??

I test that and it works, but it is a huge job to do for source that
is ok with old fpc.
And very stupid requirement just like regexp that does not allow empty
strings.

----------------------------------------------------------------------
JuhaManninen · 2021-02-11 15:44

Quote from: trev on February 11, 2021, 07:30:14 amSo now you have your
answer to what has changed which was documented in the release notes.
Those notes are always useful reading.
trev, your answer was not very helpful. You should tell how to solve
the problem if you really know.
I remember other people asking about the same issue. I don't remember
how it was solved though. Is it something that got fixed in FPC 3.2.1
fixes branch? I hope they will release 3.2.2 ASAP.
I don't do programming with DBs currently. This bug did not hit me.
In any case a breaking change like this is bad. A realistic workaround
is to stick with FPC 3.0.4 until it gets fixed. You can build the
latest Lazarus, including Lazarus trunk, with FPC 3.0.4.

----------------------------------------------------------------------
JuhaManninen · 2021-02-11 17:15

Quote from: OH1KH on February 11, 2021, 03:49:51 pmIs it really so
that I have to change nearly 900 lines of code from XXXXX.AsString to
XXXXX.AsBinary and then pass result to function loop that creates
string without conversion ??
Typecasting the binary value to RawByteString should prevent
conversion. But yes, it must be added everywhere. Not an easy
solution.

----------------------------------------------------------------------
Hartmut · 2021-02-12 08:02

Seems that OH1KH has the same problem that I have (on Windows) since I
tried FPC 3.2.0:

When I read data from a SQLite-DB, for so many years (at least since
FPC 2.6.4 and until 3.0.4) this data was always in UTF8 - for console
and GUI programs - regardless whether Unit LazUTF8 was used or not.
But now (if Unit LazUTF8 is not used) this data is instead returned in
Windows-charset (ANSI 1252). But the SELECT-Statements, if they
include characters like Ä Ö Ü ä ö ü ß, have still to be in UTF8.
Crazy...

Of course I read above release notes multiple times. But I do not
understand too much of what is written there - especially not, how a
solution for existing code could look - and I'm not a bloody beginner
(started with Turbo Pascal 3 nearly 35 years ago).

I wasted 2 days with useless tries to find a global/central solution,
which does not require to change endless code lines in every program
and in multiple common units but I failed. Then I had to stop this
because I have enough other todo's with higher prio and I decided for
now not to use FPC 3.2.0 (but this is not the sense of a new release).
And this was not the 1st time, where breaking code changes in a new
release in combination with insufficient release notes caused to me a
lot of time consuming and frustrating troubles :-((

I appreciate the continuous work of the FPC developers to improve and
enhance FPC. Thanks a lot to them. But if they break existing code, I
think it is important, to write in the release notes not only what
they have changed and why. If things are not trivial - as here -
additionally there should be clear instructions, how to adapt existing
code:
 - step 1: do this
 - step 2: do that
 - step 3: if you have ... then do ... else do ...
If not, FPC developers do a disservice to a lot of FPC users who have
existing code: of course the developer, who changes something like
this, is an expert in this area and is the one, who can at easiest
overlook how is the best/easiest way to adapt existing code. A lot of
"normal FPC users" / hobbyists are far away from that and are thrown
into a lot of trouble plus the risk of adding unneccessary bugs.

How could / should a solution for this problem look like?
 - it should not be neccessary that every FPC user has to change
"hundreds" of code lines
 - there should be something like a global/central setting in one of
the classes, which are involved to access a database (e.g.
TSQLConnection?) so that you "only" have to adapt 1 code line for each
whole database
 - and the effort should be reasonable, that common units could be
adapted in a way (e.g. via not too much $IFDEF), that they can be
compiled both with the new FPC release and with some older FPC
versions.

@JuhaManninen:
I see your name since a long time and assume that you have some
influence in the FPC community. Would you please be so kind to forward
the blue and green parts to the FPC developers? I hope that this could
help
 - that release notes in the future become more expressive and helpful
- especially for breaking code changes
 - that this current issue will be fixed in a wise way.
Thanks a lot to you.

----------------------------------------------------------------------
JuhaManninen · 2021-02-12 09:06

Quote from: Hartmut on February 12, 2021, 09:02:29 am@JuhaManninen:
I see your name since a long time and assume that you have some
influence in the FPC community. Would you please be so kind to forward
the blue and green parts to the FPC developers? I hope that this could
help
 - that release notes in the future become more expressive and helpful
- especially for breaking code changes
 - that this current issue will be fixed in a wise way.
Thanks a lot to you.
I am a Lazarus developer but not really involved in FPC development. I
haven't even followed fpc-dev and fpc-pascal mailing lists diligently
recently.
Some FPC developers do follow this forum.
Does nobody really know a solution for this problem? Will the next FPC
3.2.2 fix it?

----------------------------------------------------------------------
Lutz Mändle · 2021-02-12 09:18

@OH1KH and @Hartmut,
have you set the charset property of the TSQLConnection to utf8?
I'm working with mysql and sqlite on windows and some linux flavors
and have no utf8 related problems.

----------------------------------------------------------------------
Hartmut · 2021-02-12 10:42

Quote from: Lutz Mändle on February 12, 2021, 10:18:57 am@OH1KH and
@Hartmut,
have you set the charset property of the TSQLConnection to utf8?
I'm working with mysql and sqlite on windows and some linux flavors
and have no utf8 related problems.
Thanks Lutz for your reply. Here is how I did the DB initialization:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i var
DataSourceX: TDataSource;    DBConnectionX: TSQLite3Connection;   
SQLQueryX: TSQLQuery;    SQLTransactionX: TSQLTransaction;... 
 DataSourceX:=TDataSource.Create(nil);           {create all vars: } 
 DBConnectionX:=TSQLite3Connection.Create(nil); 
 DBConnectionX.CharSet:='UTF8'; {tried other spellings, see below} 
 SQLQueryX:=TSQLQuery.Create(nil); 
 SQLTransactionX:=TSQLTransaction.Create(nil);  
 DBConnectionX.Transaction:=SQLTransactionX;     {connect all vars: } 
 SQLQueryX.Database:=DBConnectionX;//
SQLQueryX.Transaction:=SQLTransactionX; {happens automatically} 
 DataSourceX.Dataset:=SQLQueryX;  
 DBConnectionX.Name:='DBConnection'; 
 DBConnectionX.DatabaseName:='mydb.sqlite';
I tried some other spellings like 'cpUTF8', 'utf8', 'CPUTF8',
'CP_UTF8' because, as often, I found no documentation about the
allowed names with reasonable effort (even the Objectinspector does
not have a ComboBox for that property where you could see the
spellings).
But all this changed nothing.

----------------------------------------------------------------------
LacaK · 2021-02-12 10:56

As it was already mentioned TSQLConnection.CharSet property plays here
major role.
Try set it to 'utf8mb4' for MySQL
https://dev.mysql.com/doc/refman/8.0/en/charset-connection.html

----------------------------------------------------------------------
Lutz Mändle · 2021-02-12 12:11

I'm using a TSQLConnector and set the property ConnectorType to
'SQLite3', DatabaseName is the filename of the sqlitedb, Hostname,
UserName and Password leave empty.
According to the sources the spellings 'utf8', 'utf-8' and 'utf8mb4'
for the charset property should have the same effect.

In TSQLite3Connection the method GetConnectionCharSet is overridden
and always returns 'utf8' no matter how the charset property is set.

----------------------------------------------------------------------
Hartmut · 2021-02-12 13:31

Quote from: LacaK on February 12, 2021, 11:56:50 amAs it was already
mentioned TSQLConnection.CharSet property plays here major role. Try
set it to 'utf8mb4' for MySQL
Thanks. I tested this in a console program with FPC 3.2.0 on Windows 7
32-bit, but unfortunately nothing changed.

Quote from: Lutz Mändle on February 12, 2021, 01:11:38 pmI'm using a
TSQLConnector and set the property ConnectorType to 'SQLite3',
DatabaseName is the filename of the sqlitedb, Hostname, UserName and
Password leave empty. According to the sources the spellings 'utf8',
'utf-8' and 'utf8mb4' for the charset property should have the same
effect.
Thanks for this suggestion. Until now I've never heard of
TSQLConnector and must 1st find out how this thing has to be
interlaced in the already existing combination of the 4 above classes
which I normally use for DB's which is not trivial to me (in some
programs I used the Objectinspector to set up static classes while in
others I needed dynamic classes as shown above). Unfortunately these
days I don't have more time to dive into this, but I will try it
later. How sure are you that it will solve the problem?

QuoteIn TSQLite3Connection the method GetConnectionCharSet is
overridden and always returns 'utf8' no matter how the charset
property is set.
If I understand this correctly, all my tries to set
'TSQLite3Connection.CharSet' some month ago and now again were
foredoomed and wasted time. I had tried it, because I understood above
release notes in this way which say:
QuoteNew behaviour: When those fields are created there can be
specified CodePage (if none specified CP_ACP is assumed), which
defines encoding of character data presented by this field. In case of
sqlDB TSQLConnection.CharSet is usualy used.

Again I am very unhappy how release notes in such a non trivial case
were realized :-((

----------------------------------------------------------------------
ASBzone · 2021-02-12 13:43

Quote from: OH1KH on February 11, 2021, 03:49:51 pmHi !
There is no answer how to turn on compatibility to old way!

Is it really so that I have to change nearly 900 lines of code from
XXXXX.AsString to XXXXX.AsBinary and then pass result to function loop
that creates string without conversion ??

I test that and it works, but it is a huge job to do for source that
is ok with old fpc.
And very stupid requirement just like regexp that does not allow empty
strings.


Are you able to explicitly set the code page in your application so
that it returns it to that code page when reading from SQL?

----------------------------------------------------------------------
Lutz Mändle · 2021-02-12 15:17

Replace TSQLite3Connection in your example with TSQLConnector, that
should work.

Then after the creation of DBConnectionX have a line like this:

Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i ....   
DBConnectionX.ConnectorType:='SQLite3';   
DBConnectionX.CharSet:='utf8'; ....

----------------------------------------------------------------------
OH1KH · 2021-02-12 17:00

I have tried:

MainCon.CharSet:='CP_NONE';   (in my case MariaDB is used)

This passes ok when compiled, but when running it causes error:
TMySQL57COnnection: Failed to set connection character set: Can't
initialize character set CP_NONE (path: compiled_in)

As the problem persist when reading varchar column from database and
placing to to Tedit (.Text) that now with fpc3.2.0 causes unneeded
character conversion I assume that I could get proper view of TEdit if
charset would be CP_NONE, I.E. that no conversion would happen.

This is quite funny. Reading from TEdit(.Text) and placing it to
database column works like before.
But to other direction it makes unneeded charcter conversion.
If I write values to databse with fpc320 compiled version and then
open fpc304 compiled version for reading database everything is ok,
like before.

Why the database string reading is twisted with unneeded conversion
when writing has left untouched?
Well, lucky in a way. Then we would get double trouble!

----------------------------------------------------------------------
OH1KH · 2021-02-12 17:21

Quote from: Lutz Mändle on February 12, 2021, 10:18:57 am@OH1KH and
@Hartmut,
have you set the charset property of the TSQLConnection to utf8?
I'm working with mysql and sqlite on windows and some linux flavors
and have no utf8 related problems.

Tried that also:

 MainCon.CharSet:='UTF8';  

Compiles and runs OK.
When I set TEdit.text :='ÖÄÅöäå' and push it to database and then pull
it back it is still 'ÖÄÅöäå' in Tedit.Text

But it arises an new problem when I start fpc304 compiled version.
Then TEdit.Text shows out as '???'
And if I run fpc320 compiled version all old database entries have
garbage view, while ones inserted with this compile version have
special characters ok.
My database has > 10000 entries, but there are thousands of users
running this program and they are facing same problem then.

It must be backward compatible in all ways !

----------------------------------------------------------------------
Hartmut · 2021-02-12 18:40

Quote from: Lutz Mändle on February 12, 2021, 04:17:21 pmReplace
TSQLite3Connection in your example with TSQLConnector, that should
work.
Then after the creation of DBConnectionX have a line like this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i    
DBConnectionX.ConnectorType:='SQLite3';   
DBConnectionX.CharSet:='utf8';  Thank you very much for this helpful
example. That sounds not difficult. I will test it when I have the
time, that in case that it solves my problem, I can directly adapt it
in all programs and common units with static and dynamic DB classes.

----------------------------------------------------------------------
Lutz Mändle · 2021-02-12 21:29

@OH1KH

the charset property should set to a value that matches the charset
used by the database. From what you wrote earlier in this thread I
assume that your tables are not configured for utf8. What shows the
status command of the command line client?

It should something like this:
Code: Text  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i MariaDB
[(none)]> status--------------mysql  Ver 15.1 Distrib 10.5.8-MariaDB,
for Linux (x86_64) using  EditLine wrapper Connection id:        
 7Current database:Current user:           dbuser@localhostSSL:      
            Not in useCurrent pager:          lessUsing outfile:     
   ''Using delimiter:        ;Server:                 MariaDBServer
version:         10.5.8-MariaDB MariaDB packageProtocol version:     
10Connection:             Localhost via UNIX socketServer
characterset:    utf8mb4Db     characterset:    utf8mb4Client
characterset:    utf8Conn.  characterset:    utf8UNIX socket:        
  /run/mysql/mysql.sockUptime:                 1 hour 43 min 29
sec Threads: 2  Questions: 12  Slow queries: 0  Opens: 16  Open
tables: 10  Queries per second avg: 0.001-------------- MariaDB
[(none)]>

----------------------------------------------------------------------
OH1KH · 2021-02-13 08:32

There are two databases in use:

1st choice is a database running in user's config folder (mysql_safe)

Connection:      Localhost via UNIX socket
Server characterset:   utf8
Db     characterset:   utf8
Client characterset:   utf8
Conn.  characterset:   utf8
UNIX socket:      /home/saku/.config/cqrlog/database/sock
Uptime:         1 min 6 sec

Second choice (normally used here) is in external mysql network
server:

Server characterset:   latin1
Db     characterset:   utf8
Client characterset:   utf8
Conn.  characterset:   utf8
UNIX socket:      /var/lib/mysql/mysql.sock
Uptime:         3 days 21 hours 38 min 21 sec


I both cases databases have UTF-8 charset.

From reading many wiki pages because of this problem I have understood
that if client  running program and database connected both use UTF-8
charset TMySQLConnection should automatic adopt that for use without
any specification in connect source code.

Setting connection parameter  XXXXXX.CharSet:='UTF8'; makes this work
ok with both fpc3.0.4 and fpc3.2.0 compiled versions.
Problem now is just old database contents that have been written with
fpc3.0.4 compiled version(s) that do not have  had
XXXXXX.CharSet:='UTF8'; line.

If continuing now with CharSet:='UTF8'; line. all future database
additions have special characters ok, but all past information shows
out as garbage.

Perhaps database update routine has to be added that goes through all
old entries and converts them to "new normal".

----------------------------------------------------------------------
LacaK · 2021-02-13 09:17

What I can comment:
1. for SQLite3 there is hardcoded UTF-8 as connection charset because
SQLite3 always assumed that character data are UTF-8 encoded. There is
no way to change charset/collation for database/table/column or
connection

2. As I understood if MySQL is used only with FPC3.2 then all is as
expected. Problem arrises when old data are read, which were written
in FPC3.0.4
If these data originated in Lazarus then they were UTF-8 encoded. So
what hapend is that they were translated during transmision to MySQL
database. Logical explaination is that default connection charset was
other than UTF-8 in scenario with FPC3.0.4

----------------------------------------------------------------------
JuhaManninen · 2021-02-13 09:19

OH1KH and Hartmut, just to be sure :
Do you use the Lazarus UTF-8 system? It is supported automatically in
LCL applications. In programs you need to use unit LazUTF8.
See
 https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus

----------------------------------------------------------------------
Hartmut · 2021-02-13 16:19

Quote from: JuhaManninen on February 13, 2021, 10:19:34 amOH1KH and
Hartmut, just to be sure :
Do you use the Lazarus UTF-8 system? It is supported automatically in
LCL applications. In programs you need to use unit LazUTF8.

I am mostly working on console programs and because I faced endless
problems when including unit LazUTF8 I try to avoid this unit whenever
possible, because it sets many used charsets upside down. If you are
interested in examples you can find some of them in reply #12 of
https://forum.lazarus.freepascal.org/index.php/topic,51558.0.html
where I had a discussion with wp about some of those problems. So unit
LazUTF8 would not be a solution for me. But I will check before long
if the suggestion of Lutz Mändle in reply #12 and #15 does solve the
problem which I hope.

----------------------------------------------------------------------
JuhaManninen · 2021-02-13 17:37

Quote from: Hartmut on February 13, 2021, 05:19:25 pmI am mostly
working on console programs and because I faced endless problems when
including unit LazUTF8 I try to avoid this unit whenever possible,
because it sets many used charsets upside down.
Ok, now we know what causes your Unicode related problems. You must
use the Unicode support through LazUTF8 or you are doomed ... as you
have noticed yourself.
You could have mentioned initially that you try to avoid our well
tested and supported Unicode system. People spent time searching a
problem elsewhere.
BTW, it does not set charsets upside down. It sets Unicode with UTF-8
encoding. I don't see any reason not to use Unicode. It is 2021
already. Unicode has been around for 25+ years.
Why did you set your DB for Unicode if you don't want to use Unicode?

QuoteIf you are interested in examples you can find some of them in
reply #12 of
https://forum.lazarus.freepascal.org/index.php/topic,51558.0.html
where I had a discussion with wp about some of those problems.
I am not interested. I already know what happens. No need to fight
against windmills.

QuoteSo unit LazUTF8 would not be a solution for me. But I will check
before long if the suggestion of Lutz Mändle in reply #12 and #15 does
solve the problem which I hope.
Good luck with that! 

OH1KH, I believe you have the same problem. Didn't you know about the
Unicode support thing?

----------------------------------------------------------------------
Hartmut · 2021-02-14 10:34

Quote from: JuhaManninen on February 13, 2021, 06:37:59 pmYou could
have mentioned initially that you try to avoid our well tested and
supported Unicode system. People spent time searching a problem
elsewhere.
Sorry, I had no idea that there could be an influence. I read and
understood the release notes that there would be a generally breaking
code problem with FPC 3.2.0 and not, that this problem occurs only to
people, who do not use unit LazUTF8. Again my petition for better and
more helpful release notes.

QuoteWhy did you set your DB for Unicode if you don't want to use
Unicode?
I never set my SQLite DB's to UTF8. They always used UTF8
automatically for all those years since my 1st FPC version.

Currently unit LazUTF8 is not an option for me. I tried to point you
to some reasons why. If the suggestion of Lutz Mändle works I would
prefer this solution.

----------------------------------------------------------------------
JuhaManninen · 2021-02-14 15:32

Quote from: Hartmut on February 14, 2021, 11:34:56 amSorry, I had no
idea that there could be an influence. I read and understood the
release notes that there would be a generally breaking code problem
with FPC 3.2.0 and not, that this problem occurs only to people, who
do not use unit LazUTF8. Again my petition for better and more helpful
release notes.
From FPC's point of view the Lazarus UTF-8 solution is a hack.
Now I understand the change in FPC 3.2 was consistent with dynamic
string codepage assignments elsewhere.
The UTF-8 solution works well when all your data is UTF-8. When you
read / write data containing Windows codepage, you must convert to /
from Unicode ASAP.
Please read and understand this :
 https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus#Readin
g_.2F_writing_text_file_with_Windows_codepage

QuoteI never set my SQLite DB's to UTF8. They always used UTF8
automatically for all those years since my 1st FPC version.
Ok, I will ask in a different way: Why do you want to use Windows
codepages?
For decades they have been a source of pain and sorrow and texts with
question marks ('?????????') when transferred between locales.
Unicode solves those problems!
You must be a masochist to still use Windows codepages.

QuoteCurrently unit LazUTF8 is not an option for me. I tried to point
you to some reasons why. If the suggestion of Lutz Mändle works I
would prefer this solution.
Surely it is an option. Just wrap your mind around the issue. It may
be simpler than you realize.

----------------------------------------------------------------------
Hartmut · 2021-02-15 13:31

Quote from: JuhaManninen on February 14, 2021, 04:32:08 pmWhen you
read / write data containing Windows codepage, you must convert to /
from Unicode ASAP. Please read and understand this :
https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus#Reading
_.2F_writing_text_file_with_Windows_codepage
Thanks for that link. I looked into it and added it to my list where I
collect infos about UTF8 related things if I might need it one day.

QuoteOk, I will ask in a different way: Why do you want to use Windows
codepages?
It's not my declared goal to use Windows charsets everywhere. But
great parts of my sources rely on a specific charset (for which they
had been written and tested): all my sources for lots of programs
including many common units have grown up over nearly 35 years (I
started with Turbo Pascal 3). Great parts of this sources would not
work with UTF8.

For me it is important, that I do not loose enormous time by changing
things, which had already been finished including costly testing (!)
and which work perfectly, as long as there is no real need. Because
I'm only a hobbyist and unfortunately the time for each hobby is
limited :-((

QuoteFor decades they have been a source of pain and sorrow and texts
with question marks ('?????????') when transferred between locales.
I had some cases in the past, but I solved them long time ago. That
means, this is definitely not my problem. But this is funny: when I
*included* unit LazUTF8, one of the problems which you did not want to
read was, that in some cases '???' instead of 'äöü' etc. appeared. But
only *with* this unit and definitely *never* without this unit.

Please let us stop this discussion now. You did not want to read some
examples of the problems which I faced with unit LazUTF8 - which of
course is free to you - but then all its disadvantages for me might
not be comprehensible to you.

I highly appreciate this forum (including your part) for changing
technical informations e.g. what is possible or how something can made
working. But please allow, that I want to be the one who decides which
of 2 possible solutions I would prefer in my personal situation,
because I am the one who has to shoulder the resulting work.

----------------------------------------------------------------------
JuhaManninen · 2021-02-16 19:34

Quote from: Hartmut on February 15, 2021, 02:31:51 pmI had some cases
in the past, but I solved them long time ago. That means, this is
definitely not my problem. But this is funny: when I *included* unit
LazUTF8, one of the problems which you did not want to read was, that
in some cases '???' instead of 'äöü' etc. appeared. But only *with*
this unit and definitely *never* without this unit.
Yes because a wrong encoding was assumed.
How well does your code work with a phrase in Greek or Russian? ... Or
in Hindi or Chinese?

QuotePlease let us stop this discussion now. You did not want to read
some examples of the problems which I faced with unit LazUTF8 - which
of course is free to you - but then all its disadvantages for me might
not be comprehensible to you.
Actually I read your posts briefly. You had problems iterating UTF-8
codepoints. It could be solved easily.
Do you need to show text outside ASCII in Windows console? That may be
the only reason one would omit the Unicode solution.
Your data in DB is Unicode with UTF-8 encoding. It is a perfect match
for our UTF-8 system. It means you never need to convert string
encoding.
I still encourage you to create a test version of your code with
LazUTF8 where you strip all conversions. Just assign data and you are
good.
For specific codepoint iteration issues you get help from here. Wiki
has info, too :
 https://wiki.lazarus.freepascal.org/UTF8_strings_and_characters
There is even a nice "character" iterator supporting combining
codepoints :
 https://wiki.lazarus.freepascal.org/Unicode_Support_in_Lazarus#CodePo
int_functions_for_encoding_agnostic_code

QuoteI highly appreciate this forum (including your part) for changing
technical informations e.g. what is possible or how something can made
working. But please allow, that I want to be the one who decides which
of 2 possible solutions I would prefer in my personal situation,
because I am the one who has to shoulder the resulting work.
Yes of course. A valid solution also is to stick with a compiler
version that works for you. If it works now, it will continue to work
in the future.

----------------------------------------------------------------------
EgonHugeist · 2021-02-20 08:41

@LacaK

I do understand the users having trouble with the FPC3.2 and not using
LazUTF8.

The input/ouput data should be equal to the database encoding, except
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
T(String/Memo)Field.Transliterate is set to true. See:
https://www.freepascal.org/docs-html/fcl/db/tstringfield.html. That
than means each raw "string" should have the codepage of the field if
Transliterate is disabled. It doesn't matter if ithe property is
declared as "String" and the DefaultSystemCodePage <> TField.CodePage.
IIRC the fpc defaults is: Code:
[Select]T(String/Memo)Field.Transliterate := False;. Did you change
that to the Delphi defaults inbetween? If not it's a bug from my POV.

Just what i'm thinking and how i implemented it on Zeos8. What do you
think?
I don't wanna comment the "LCL all utf8" design here.

Regards from ZeosDevTeam, Michael

----------------------------------------------------------------------
LacaK · 2021-03-01 08:37

May be that implementation of Transliterate property / Translate
method is incomplete.
I remembered that I studied it years ago, but I never understood it
fully.
So I can not comment on this ... byt I think, that most of users are
not aware of this Transliterate property ...

----------------------------------------------------------------------
EgonHugeist · 2021-03-09 17:52

Laco, i just wanna help. Honestly it's a bug for me (and the
reporters) what you/others are doing. Dealing with DB's vs
String+StringCodePage doesn't mean you should convert "something by"
default just because the codepage-aware compiler is expecting a string
with defaultsystemcodepage. It's 100% accapltable if the output/input
string has the codepage of the database column. I'm trying to say: You
need an indicator todo a CP conversion, OTH it's an inacaptable
behavior nobody is happy about.

----------------------------------------------------------------------
jc99 · 2021-03-20 10:43

I have the same problem. I have a huge database filled with strings by
programms compiled with fpc 3.0.4 and earlier. I accepted the remedy
that to my oppinion fpc 3.0.4 did the encoding wrong, because other
programms like HeidiSQL or SQL-Workbench did not show the special-
chars in the right way.
Starting with fpc 3.2.0 the strings are also displayed wrongly.So I
needed the old behavior.... but when I try to compile the code with
Laz2.0.8& fpc 3.0.4 the strings are also displayed wrongly,
and now I have a problem.
btw: No Console Program, Data goes TMySQL57Connection -> TSQLQuery ->
TDataset -> TDBGrid
So no manual fiddeling with data.

I tried LazUTF8 -> no change.
I tried Charset to whatever -> no change.
Any help is welcome.
I am on Laz 2.0.8 fpc 3.0.4

----------------------------------------------------------------------
dseligo · 2021-03-20 15:03

Quote from: jc99 on March 20, 2021, 11:43:48 amI have the same
problem. I have a huge database filled with strings by programms
compiled with fpc 3.0.4 and earlier. I accepted the remedy that to my
oppinion fpc 3.0.4 did the encoding wrong, because other programms
like HeidiSQL or SQL-Workbench did not show the special-chars in the
right way.
Starting with fpc 3.2.0 the strings are also displayed wrongly.So I
needed the old behavior.... but when I try to compile the code with
Laz2.0.8& fpc 3.0.4 the strings are also displayed wrongly,
and now I have a problem.
btw: No Console Program, Data goes TMySQL57Connection -> TSQLQuery ->
TDataset -> TDBGrid
So no manual fiddeling with data.

I tried LazUTF8 -> no change.
I tried Charset to whatever -> no change.
Any help is welcome.
I am on Laz 2.0.8 fpc 3.0.4
So, you say that with 3.0.4 and 3.2.0 your data isn't displayed
correct? And previously with 3.0.4 it was displayed correct? Then you
probably changed something in your project.
What is character encoding in your database?
Can you show some example how your data is shown now and how it should
be shown (incorrect data - correct data)?

----------------------------------------------------------------------
engkin · 2021-03-20 15:29

Lazy people like me, would be motivated to try to help if you post a
simple project showing the problem.

----------------------------------------------------------------------
dseligo · 2021-03-20 16:25

Quote from: engkin on March 20, 2021, 04:29:50 pmLazy people like me,
would be motivated to try to help if you post a simple project showing
the problem.
It would be best for me if he would prepare test data. Create a copy
of a table, one row with sample data and exports it with mysqldump.
And he need to say how data should look like.
Then we could analyze how is string written in his database (and
propose a solution).

----------------------------------------------------------------------
engkin · 2021-03-20 16:37

Quote from: dseligo on March 20, 2021, 05:25:41 pmQuote from: engkin
on March 20, 2021, 04:29:50 pmLazy people like me, would be motivated
to try to help if you post a simple project showing the problem.
It would be best for me if he would prepare test data. Create a copy
of a table, one row with sample data and exports it with mysqldump.
And he need to say how data should look like.
Then we could analyze how is string written in his database (and
propose a solution).

Test project without data is not a test project, and I will not be
motivated without data, I am that lazy. And I was/still that lazy not
to mention it in my previous post.

----------------------------------------------------------------------
jc99 · 2021-03-26 07:45

Sorry I was busy for some Days ...
A test-procect is hard, cause you need a DB filled in old manner, my
actual DB is ~4Gb ~50k Main-Entries, lot's of (very personal) data.
I'll try, maybe share a GitHub-Link ...
(btw some of the code is published on GitHub)[BRI'll set up a VM with
Laz2.0.8 ... to test (and work as workaround)

----------------------------------------------------------------------
jcmontherock · 2021-03-26 09:35

What I am using is the sql script "set names=..." depending of the
encoding of the DB, just after the connection to server. This sql set
the encoding for everybody (server, client). All scripts used after
that should have the same encoding.

----------------------------------------------------------------------
dseligo · 2021-03-26 09:38

Quote from: jc99 on March 26, 2021, 08:45:24 amSorry I was busy for
some Days ...
A test-procect is hard, cause you need a DB filled in old manner, my
actual DB is ~4Gb ~50k Main-Entries, lot's of (very personal) data.
I didn't mean whole database. One word would possibly be enough. You
can create new table (create table new_table like your_table), insert
one row from existing table, delete data not relevant from that row
and export that table only.

----------------------------------------------------------------------
Seenkao · 2021-03-26 09:53

Как я понял, раньше происходила автоматическая кодировка в UTF-8? В
настоящее время вы вынуждены вручную переводить в UTF-8?

Eng: As I understand it, there was automatic encoding in UTF-8 before?
Are you currently forced to manually translate to UTF-8?

----------------------------------------------------------------------
jc99 · 2021-03-26 13:07

Quote from: dseligo on March 20, 2021, 04:03:59 pm[...]
So, you say that with 3.0.4 and 3.2.0 your data isn't displayed
correct? And previously with 3.0.4 it was displayed correct? Then you
probably changed something in your project.
I thought so too, ... (but I have found nothing yet)

QuoteWhat is character encoding in your database?
--> UTF8

QuoteCan you show some example how your data is shown now and how it
should be shown (incorrect data - correct data)?
Wälde -> Wälde
Günther -> Günther
Maaß -> Maaß
DB-Code (SQL)
Code: SQL  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i /*!40101
SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;/*!40101 SET
NAMES utf8 */;/*!50503 SET NAMES utf8mb4 */;/*!40014 SET
@OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0
*/;/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE,
SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; CREATE DATABASE IF NOT EXISTS
`rnz-traueranzeigen` /*!40100 DEFAULT CHARACTER SET utf8 */;USE `rnz-
traueranzeigen`; CREATE TABLE IF NOT EXISTS `anzeigen` (  `idAnzeige`
BIGINT(20) NOT NULL AUTO_INCREMENT COMMENT 'ID des Eintrags', 
`Auftrag` VARCHAR(20) NOT NULL COMMENT 'Auftragsnummer ',  `Stichwort`
VARCHAR(100) DEFAULT NULL COMMENT 'Ausgelesenes Stichwort', 
`Nachname` VARCHAR(100) DEFAULT NULL COMMENT 'Eingegebener Nachname
(bei Personen)',  `Vorname` VARCHAR(100) DEFAULT NULL COMMENT
'Eingegebener Vorname  (bei Personen)',--[...]  `Pfad` VARCHAR(100)
DEFAULT NULL COMMENT 'rel. Pfad zu den Daten',  `Bild` longblob
COMMENT 'Bild-Daten',  `LinkID` BIGINT(20) DEFAULT NULL COMMENT 'ID
der "Hauptanzeige" (bei Nachrufen, Danksagungen und Errinnerungen)', 
`ProfileImg` VARCHAR(50) DEFAULT NULL COMMENT 'Bild der Person
(Profilbild)',  `ProfImgCount` SMALLINT(6) NOT NULL DEFAULT '-1'
COMMENT 'Anzahl der PDF-Bilder',  `TimeStamp` TIMESTAMP NOT NULL
DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT
'Zeitstempel der letzten Änderung',  PRIMARY KEY (`idAnzeige`), 
UNIQUE KEY `Auftrag` (`Auftrag`),  KEY `Stichwort`
(`Stichwort`),--[...]  KEY `ixLinkID` (`LinkID`),  KEY `ixTimeStamp`
(`TimeStamp`)) ENGINE=InnoDB AUTO_INCREMENT=48862 DEFAULT CHARSET=utf8
COMMENT='RNZ-Anzeigen ab Mrz 2015 ';

----------------------------------------------------------------------
jc99 · 2021-03-26 13:21

The Places-Table would be:
Code: SQL  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i /*!40101
SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;/*!40101 SET
NAMES utf8 */;/*!50503 SET NAMES utf8mb4 */;/*!40014 SET
@OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0
*/;/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE,
SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; CREATE DATABASE IF NOT EXISTS
`rnz-traueranzeigen` /*!40100 DEFAULT CHARACTER SET utf8 */;USE `rnz-
traueranzeigen`; CREATE TABLE IF NOT EXISTS `orte` (  `idOrte` INT(11)
NOT NULL COMMENT 'ID',  `Ortname` VARCHAR(100) NOT NULL COMMENT
'Kurzname',  `LangName` VARCHAR(300) DEFAULT NULL COMMENT 'Voll
qualifizierter Name',  `idOrte_Lnk` VARCHAR(300) DEFAULT NULL COMMENT
'Bei Kopie Verweiss auf (haupt-)Ort',  `Longitude` VARCHAR(20) DEFAULT
NULL COMMENT 'Längengrad des Orts',  `Latitude` VARCHAR(20) DEFAULT
NULL COMMENT 'Breitengrad des Orts',  PRIMARY KEY (`idOrte`),  KEY
`Ortname` (`Ortname`)) ENGINE=InnoDB DEFAULT CHARSET=utf8; /*!40000
ALTER TABLE `orte` DISABLE KEYS */;INSERT INTO `orte` (`idOrte`,
`Ortname`, `LangName`, `idOrte_Lnk`, `Longitude`, `Latitude`) VALUES 
     (2, 'Schönbrunn', 'Schönbrunn, Rhein-Neckar-Kreis, Baden-
Württemberg, Germany', NULL, 'E8,9284', 'N49,4123');/*!40000 ALTER
TABLE `orte` ENABLE KEYS */; /*!40101 SET
SQL_MODE=IFNULL(@OLD_SQL_MODE, '') */;/*!40014 SET
FOREIGN_KEY_CHECKS=IF(@OLD_FOREIGN_KEY_CHECKS IS NULL, 1,
@OLD_FOREIGN_KEY_CHECKS) */;/*!40101 SET
CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; btw: the name of
the place is Schönbrunn !

----------------------------------------------------------------------
engkin · 2021-03-26 14:56

To get the following results:
Wälde -> Wälde
Günther -> Günther
Maaß -> Maaß

The string is UTF8, but the codepage is cp1252

For instance, try:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i s :=
"Maaß";memo1.Append(s);// gives correct
word SetCodePage(RawByteString(s), 1252, False);memo1.Append(s);//
gives the wrong word
Notice that, in this example, the data itself is correct, only the
codepage is wrong.

The more examples of wrong conversions you give the more we confirm
the real codepage (cp1252). What is the codepage of your system?

----------------------------------------------------------------------
dseligo · 2021-03-26 20:02

Try this:
Code: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i uses
lazutf8;...var s:String;...s:='Wälde'; // -> WäldeMemo1.Append(s+'
--> '+Utf8ToWinCP(s));s:='Günther'; // -> GüntherMemo1.Append(s+' -->
'+Utf8ToWinCP(s));s:='Maaß'; // -> MaaßMemo1.Append(s+' -->
'+Utf8ToWinCP(s));

----------------------------------------------------------------------
jc99 · 2021-03-27 08:18

TestForm:Code: Pascal  [Select][+][-]window.onload = function(){var x1
= document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i unit
frm_TestDBEncoding; {$mode objfpc}{$H+} interface uses  Classes,
SysUtils, Forms, Controls, Graphics, Dialogs, ExtCtrls,
StdCtrls; type   { TFrmTstDBEncodingMain }   TFrmTstDBEncodingMain =
class(TForm)    memo1: TMemo;    procedure FormShow(Sender: TObject); 
private   public   end; var  FrmTstDBEncodingMain:
TFrmTstDBEncodingMain; implementation {$R *.lfm} uses lazutf8;{
TFrmTstDBEncodingMain } procedure
TFrmTstDBEncodingMain.FormShow(Sender: TObject);var  s: String;begin 
 s := 'Maaß';    memo1.Append(s);// gives correct word    
SetCodePage(RawByteString(s), 1252, False);    memo1.Append(s);//
gives the wrong word     s:='Wälde'; // -> Wälde    memo1.Append(s+'
--> '+Utf8ToWinCP(s));    s:='Günther'; // -> Günther   
memo1.Append(s+' --> '+Utf8ToWinCP(s));    s:='Maaß'; // -> Maaß   
memo1.Append(s+' --> '+Utf8ToWinCP(s)); end; end. result:Maaß
Maaß
Wälde --> Wälde
Günther --> Günther
Maaß --> Maaß

OK, the data in the DB is double-UTF8-Encoded and can be decoded by
decoding them again ...
... with rawbytestring the data can be encoded to be consistent with
the DB.

So what is the next step ?
Is there somewhere an injection-point to get the displayed and written
right [correctly-wrong] in DBComponents i.e. a DBGrid ?
[edit]Of topic: the edit-button in the forum seems to open a broken
entry-editor only the modify-button works ...

----------------------------------------------------------------------
dseligo · 2021-03-27 10:05

Quote from: jc99 on March 27, 2021, 09:18:43 amOK, the data in the DB
is double-UTF8-Encoded and can be decoded by decoding them again ...
... with rawbytestring the data can be encoded to be consistent with
the DB.

So what is the next step ?
Is there somewhere an injection-point to get the displayed and written
right [correctly-wrong] in DBComponents i.e. a DBGrid ?
[edit]Of topic: the edit-button in the forum seems to open a broken
entry-editor only the modify-button works ...
I don't know about injection point, hopefully someone else will answer
you about that.

I would recommend that you correct data in database, and after that
you could use database components without decoding. Since you
mentioned you filled database with programs compiled with FPC 3.0.4
and older versions, I don't know if you still use them and if you
could replace them.

Benefit of that would also be that you could use HeidiSQL (and other
tools).

----------------------------------------------------------------------
jc99 · 2021-03-27 10:43

Quote from: dseligo on March 27, 2021, 11:05:30 am[...]I would
recommend that you correct data in database, and after that you could
use database components without decoding. Since you mentioned you
filled database with programs compiled with FPC 3.0.4 and older
versions, I don't know if you still use them and if you could replace
them.
Replacing the Programs is not the problem. The problem would be that I
then have inconsistent data because there are many of them doing
different tasks. I would have to switch all of them including the data
in a split-second.
... And have i mentioned the database is BIG.

I cannot shut down the programms for long because the gathered data is
very short lived, that is the reason why I started the project in the
first place.The end-goal is a database correcty encoded, but I need a
mid-term solution.
It's gonna be a step by step solution.

Quote from: dseligo on March 27, 2021, 11:05:30 amBenefit of that
would also be that you could use HeidiSQL (and other tools).
I already use HeidiSQL 11.2, I just have to be carefull with the
encodings.

----------------------------------------------------------------------
engkin · 2021-03-27 12:51

Can you produce a project showing the problem. I just tried with a
small project using SQLite, but failed to reproduce the same problem.
I used Laz1.8.0 and Laz2.0.10

I suspect the problem is caused by adding codepage to TStringField,
not sure. The change might be related to bug report #35796

Assuming this is the right track, here is the log of fields.inc.

All fields of string types are affected.

----------------------------------------------------------------------
dseligo · 2021-03-28 21:22

Quote from: jc99 on March 27, 2021, 11:43:42 amReplacing the Programs
is not the problem. The problem would be that I then have inconsistent
data because there are many of them doing different tasks. I would
have to switch all of them including the data in a split-second.
... And have i mentioned the database is BIG.

I cannot shut down the programms for long because the gathered data is
very short lived, that is the reason why I started the project in the
first place.The end-goal is a database correcty encoded, but I need a
mid-term solution.
It's gonna be a step by step solution.
I don't know how complex (not big) your database is or how complex
your programs are.
Depending if (and how long) your database could be offline I suggest
two approaches.

I.) If database could be offline for some time then you would prepare,
test and distribute new version of programs which correctly work with
UTF-8. After that you put
database offline (to this programs) and correct the encoding the data
in database. After that you start using new programs. You should
ensure that old programs aren't able to access database.

II.) If you can't take database offline then things get much more
complicated.
1.) Add column named 'version' or similar to your tables and set it to
be default '1'.
2.) Change your program to be able to decode "bad" UTF-8 and "good"
UTF-8 depending of 'version' value in row you are reading ("bad" UTF-8
is version 1, "good" UTF-8 is version 2).
3.) Start updating your programs. Programs still should use "bad"
UTF-8 for writing, but they should be able to read both versions.
After all programs are updated with they can read both versions of
table rows.
4.) You can start switching programs to use version '2' for writing
into the database.
5.) After all programs writing "good" UTF-8 to database (and can read
both versions) you start to correct encoding of strings in database
(and update 'version' value to 2).
6.) When you are done with that you deploy new version of program
where it doesn't need to take care of row 'version' value - only uses
correct encoding.
7.) Remove 'version' columns from your tables.

While I think number II. is doable, number I. is definitely much
easier to accomplish.

----------------------------------------------------------------------
jc99 · 2021-03-29 08:59

Quote from: engkin on March 27, 2021, 01:51:42 pmCan you produce a
project showing the problem. I just tried with a small project using
SQLite, but failed to reproduce the same problem. I used Laz1.8.0 and
Laz2.0.10

I suspect the problem is caused by adding codepage to TStringField,
not sure. The change might be related to bug report #35796

Assuming this is the right track, here is the log of fields.inc.

All fields of string types are affected.
I use Maria-DB the free fork of mySQL.

I suspected that, there was a breaking-change reported in the version-
changes relating to TStringField.
... in my opinion the DB is irrelevant now. Just use my provided SQL-
Creation script for places and make a program display and write the
data correct (two-way).

----------------------------------------------------------------------
jc99 · 2021-03-29 09:02

Quote from: dseligo on March 28, 2021, 11:22:48 pmI don't know how
complex (not big) your database is or how complex your programs are.
Depending if (and how long) your database could be offline I suggest
two approaches.

I.) If database could be offline for some time then you would prepare,
test and distribute new version of programs which correctly work with
UTF-8. After that you put
database offline (to this programs) and correct the encoding the data
in database. After that you start using new programs. You should
ensure that old programs aren't able to access database.

II.) If you can't take database offline then things get much more
complicated.
1.) Add column named 'version' or similar to your tables and set it to
be default '1'.
2.) Change your program to be able to decode "bad" UTF-8 and "good"
UTF-8 depending of 'version' value in row you are reading ("bad" UTF-8
is version 1, "good" UTF-8 is version 2).
3.) Start updating your programs. Programs still should use "bad"
UTF-8 for writing, but they should be able to read both versions.
After all programs are updated with they can read both versions of
table rows.
4.) You can start switching programs to use version '2' for writing
into the database.
5.) After all programs writing "good" UTF-8 to database (and can read
both versions) you start to correct encoding of strings in database
(and update 'version' value to 2).
6.) When you are done with that you deploy new version of program
where it doesn't need to take care of row 'version' value - only uses
correct encoding.
7.) Remove 'version' columns from your tables.

While I think number II. is doable, number I. is definitely much
easier to accomplish.
I have to go for II but first I need a solution to handle the
incorrect old data. Then the other steps fall in place.

----------------------------------------------------------------------
jc99 · 2021-03-29 20:22

I found a solution (for me)Code: SQL  [Select][+][-]window.onload =
function(){var x1 = document.getElementById("main_content_section");
if (x1) { var x = document.getElementsByClassName("geshi");for (var i
= 0; i /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT
*/;/*!40101 SET NAMES utf8 */;/*!50503 SET NAMES utf8mb4 */;/*!40014
SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0
*/;/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE,
SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;/*!40111 SET
@OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; CREATE DATABASE IF NOT
EXISTS `test` /*!40100 DEFAULT CHARACTER SET utf8 */;USE
`test`; CREATE TABLE IF NOT EXISTS `daten` (  `idDaten` INT(11) NOT
NULL AUTO_INCREMENT,  `Description` VARCHAR(50) DEFAULT NULL, 
`Nummer` VARCHAR(50) DEFAULT NULL,  `Name` VARCHAR(50) DEFAULT NULL, 
PRIMARY KEY (`idDaten`)) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT
CHARSET=utf8; /*!40000 ALTER TABLE `daten` DISABLE KEYS */;INSERT INTO
`daten` (`idDaten`, `Description`, `Nummer`, `Name`) VALUES    (1,
'Hand', '6', 'Hand Maaß'),    (2, 'Fritz', '3', 'Fritz Ãœhlein'),   
(3, 'Peter', '9', 'Peter Öhringer'),    (4, 'Otto', '2', 'Otto
Ährenmal'),    (5, 'Franz', '11', 'Franz Währentaz'),    (6,
'Ludwig', '8', 'Ludwig Sören Schütze');/*!40000 ALTER TABLE `daten`
ENABLE KEYS */; /*!40101 SET SQL_MODE=IFNULL(@OLD_SQL_MODE, '')
*/;/*!40014 SET FOREIGN_KEY_CHECKS=IFNULL(@OLD_FOREIGN_KEY_CHECKS, 1)
*/;/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT
*/;/*!40111 SET SQL_NOTES=IFNULL(@OLD_SQL_NOTES, 1) */;and the test-
project uses a custom TStringField2.
the injection is done with redefining the DefaultFieldClasses of
DBCode: Pascal  [Select][+][-]window.onload = function(){var x1 =
document.getElementById("main_content_section"); if (x1) { var x =
document.getElementsByClassName("geshi");for (var i = 0; i
initialization  DefaultFieldClasses[ftString]:=TStringField2; 
DefaultFieldClasses[ftFixedChar]:=TStringField2;    in the attached
project.