String and Database problem lazarus / 16-Databases ---------------------------------------------------------------------- thehidden · 2020-05-28 11:49 Hi, first Application with german language to enter. My problem I have a Memo Field or Edit Field. When - for example - an ä is entered, I see in the database an ä When I change it in the database back to ä I can get it back to the Memo or Edit Field as ä. But when I store it again, it is shown as ä Any idea what is going wrong? ---------------------------------------------------------------------- fred · 2020-05-28 12:07 I think windows code pages, Memo and Edit uses UTF-8 and your database uses a windows code page. Look for UTF8ToWinCP, WinCPToUTF8 ---------------------------------------------------------------------- af0815 · 2020-05-28 13:25 What kind of DB ? Which DB-drivers (Connection)? In Server based drivers you have often options to say, which charset they should use. ---------------------------------------------------------------------- thehidden · 2020-05-28 14:17 Quote from: af0815 on May 28, 2020, 03:25:05 pmWhat kind of DB ? Which DB-drivers (Connection)? In Server based drivers you have often options to say, which charset they should use. Desktop Database Elevate DB All Tables are set to UTF-8. I have tried to use {$codepage utf8} {$modeswith unicodestrings}  but without success. ---------------------------------------------------------------------- af0815 · 2020-05-28 14:56 Quote from: thehidden on May 28, 2020, 04:17:39 pmI have tried to use {$codepage utf8} {$modeswith unicodestrings}  but without success. This is not for the DB Connection, only for compiler. Do you use SQLdb or ZEOS ? ---------------------------------------------------------------------- thehidden · 2020-05-28 15:01 Quote from: af0815 on May 28, 2020, 04:56:26 pmQuote from: thehidden on May 28, 2020, 04:17:39 pmI have tried to use {$codepage utf8} {$modeswith unicodestrings}  but without success. This is not for the DB Connection, only for compiler. Do you use SQLdb or ZEOS ? Nothing from both. ElevateDB comes with his own connection API. The only requirement is, you need to make sure that you feed the connector with Unicode. ---------------------------------------------------------------------- af0815 · 2020-05-28 20:40 Quote from: thehidden on May 28, 2020, 05:01:32 pm ElevateDB comes with his own connection API. The only requirement is, you need to make sure that you feed the connector with Unicode. Which Unicode  UTF-8 (Lazarus uses) or UTF-16 (like IMHO Delphi uses) But i think the guys from this DB system can give you more help or samples. I see only a paywall, but no information on their HP. ---------------------------------------------------------------------- thehidden · 2020-05-28 21:34 Quote from: af0815 on May 28, 2020, 10:40:22 pmWhich Unicode  UTF-8 (Lazarus uses) or UTF-16 (like IMHO Delphi uses) But i think the guys from this DB system can give you more help or samples. I see only a paywall, but no information on their HP. I don't know. With Delphi I have no problems. But when I use the database manager to add a record with german umlauts, I can read them without problem to a edit or memo field. Only when I write back to the database I get an unwanted result. Any way to find out what string type the database connector is sending? Something like Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i if whatutf(dataset.field[0].asstring) = 'UTF8' then  showmessage('UTF-8 and not UTF-16') ---------------------------------------------------------------------- winni · 2020-05-28 21:53 Hi! Allways treasures inside of Lazarus: In the unit LConvEncoding you findCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  function GuessEncoding(const s: string): string; The result returns a string with the name of the codepage. Winni ---------------------------------------------------------------------- thehidden · 2020-05-29 07:22 Quote from: winni on May 28, 2020, 11:53:59 pmHi! Allways treasures inside of Lazarus: In the unit LConvEncoding you findCode: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i  function GuessEncoding(const s: string): string; The result returns a string with the name of the codepage. Winni Thank you for the help. Database returns as UTF-8 and I write as UTF-8. But I get a compiler message for the line with the SQL Execute and don't see why. Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var   temptxt1 : string;begintemptxt1 := 'UPDATE "Data" SET content1 = ''' + Memo1.text+''' where ID = ' + IDMarker; EDBDatabase1.StartTransaction(EmptyEDBStringsArray);  //prepare Transaction for all affected tables EDBDatabase1.Execute(temptxt1);   //Compiler warning for this line as below. Databaseform.EDBDatabase1.Commit(); Warning message: Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString" ---------------------------------------------------------------------- Zoran · 2020-05-29 07:53 Quote from: thehidden on May 29, 2020, 09:22:03 am Warning message: Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString" Is this directive: {$modeswith unicodestrings} still present in your unit? Remove it and try again. ---------------------------------------------------------------------- thehidden · 2020-05-29 08:12 Quote from: Zoran on May 29, 2020, 09:53:32 amQuote from: thehidden on May 29, 2020, 09:22:03 am Warning message: Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString" Is this directive: {$modeswith unicodestrings} still present in your unit? Remove it and try again. It is now removed. Warning is gone, but problem is not fixed. Anyway, thank you for clearing the Warning message. ---------------------------------------------------------------------- Zoran · 2020-05-29 10:21 Quote from: thehidden on May 29, 2020, 10:12:09 amQuote from: Zoran on May 29, 2020, 09:53:32 amQuote from: thehidden on May 29, 2020, 09:22:03 am Warning message: Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString" Is this directive: {$modeswith unicodestrings} still present in your unit? Remove it and try again. It is now removed. Warning is gone, but problem is not fixed. Anyway, thank you for clearing the Warning message. That directive tells to compiler to treat "String" type as UnicodeString (not AnsiString). Except in special cases, you should not use it in LCL applications. Now, make sure you do not use UTF8String anywhere, as LCL handles UTF8 encodings its own way, so, except in special cases, you should not normally let FPC make any conversions in LCL applicatons, it would lead to double conversions, thus corrupting string's content. Simply, in general, this should be enough: - make sure that your source files (units) are saved in utf8 encoding (I am not talking about using any compiler directive, just about how the files are saved). - declare your string variables as just "String", do not use UTF8String, UnicodeString or any specific code page string. - do not set any code page directive yourself. ---------------------------------------------------------------------- thehidden · 2020-05-29 11:29 Quote from: Zoran on May 29, 2020, 12:21:46 pmQuote from: thehidden on May 29, 2020, 10:12:09 amQuote from: Zoran on May 29, 2020, 09:53:32 amQuote from: thehidden on May 29, 2020, 09:22:03 am Warning message: Warning: Implicit string type conversion with potential data loss from "WideString" to "AnsiString" Is this directive: {$modeswith unicodestrings} still present in your unit? Remove it and try again. It is now removed. Warning is gone, but problem is not fixed. Anyway, thank you for clearing the Warning message. That directive tells to compiler to treat "String" type as UnicodeString (not AnsiString). Except in special cases, you should not use it in LCL applications. Now, make sure you do not use UTF8String anywhere, as LCL handles UTF8 encodings its own way, so, except in special cases, you should not normally let FPC make any conversions in LCL applicatons, it would lead to double conversions, thus corrupting string's content. Simply, in general, this should be enough: - make sure that your source files (units) are saved in utf8 encoding (I am not talking about using any compiler directive, just about how the files are saved). - declare your string variables as just "String", do not use UTF8String, UnicodeString or any specific code page string. - do not set any code page directive yourself. The directive was only added for testing. I use only string and not switching/selecting codepages. It should work, but it does not. If nothing else helps, I will store the information encoded as base64 in the database and transfer it back when I read them. ---------------------------------------------------------------------- Zoran · 2020-05-29 11:47 Perhaps ElevateDB connection tries to do string conversions based on passed string encoding (which could make problem in LCL). You might try this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var   temptxt1 : string;   temptxt2 : UTF8String;//...//...     // Instead of this:    // EDBDatabase1.Execute(temptxt1);    // just try:    temptxt2 := UTF8String(RawByteString(temptxt1));    EDBDatabase1.Execute(temptxt2); ---------------------------------------------------------------------- thehidden · 2020-05-29 12:00 Quote from: Zoran on May 29, 2020, 01:47:48 pmPerhaps ElevateDB connection tries to do string conversions based on passed string encoding (which could make problem in LCL). You might try this: Code: Pascal  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i var   temptxt1 : string;   temptxt2 : UTF8String;//...//... Thank you, but nothing changed. I have asked the manufacturer, because it works with Delphi. No answer so far.    // Instead of this:    // EDBDatabase1.Execute(temptxt1);    // just try:    temptxt2 := UTF8String(RawByteString(temptxt1));    EDBDatabase1.Execute(temptxt2); ---------------------------------------------------------------------- thehidden · 2020-06-03 07:15 UPDATE: Together with the forum from ElevateDB I got the confirmation that it is only a problem with Lazarus/Free Pascal. Lazarus uses UTF-8 , Delphi UTF-16 and ElevateDB uses UTF-16. Developer of ElevateDB must do some work on their code to make it work with Lazarus. In the meantime I will store text as base64 in the database. Makes some more work, but I have used this way in the past for other reasons.