ODBC dBASE cp852 - missing some characters lazarus / 16-Databases ---------------------------------------------------------------------- bambamns · 2013-05-13 23:07 Hi, I made an Lazarus application to exchange data from dBASE to Firebird. To access dBASE i use ODBC and it works, but it has some problems with non English letters from CP852. To convert data I use : Code: [Select]CP852ToUTF8(DM.DBFQ.FieldByName('GR_NAZ').AsString)It only convert some non English letters, but not all. I've found on the forum the similar problem on : http://www.lazarus.freepascal.org/index.php?topic=16993.0 so I tried to use : Code: [Select]UTF8Encode(DM.DBFQ.FieldByName('GR_NAZ').AsWideString)but no luck. I use latest Lazarus 1.1 and fpc 2.6.2, because I make some applications for WinCE  ( fpc 2.7.1 have problems with Glyphs on WinCE) Any help - Thank You. ---------------------------------------------------------------------- ludob · 2013-05-14 04:59 http://www.lazarus.freepascal.org/index.php?topic=16993.0 refers to a patch attached to a mantis bug report. Have you applied the patch? Without the patch, the utf8Encode won't work. The problem is that the sqldb ODBC code uses the Ansi ODBC interface and that not all characters can be converted in the driver manager. The patch should apply to 2.6.2 ---------------------------------------------------------------------- bambamns · 2013-05-14 08:01 ludob, I'm trying to apply patch but without success - with TortoiseSVN it report .... it's not a working copy. I've found GnuWin32Patch but it seams that my version of odbcsql.inc is newer than patch need. Can you help me how to apply patch ? Thank You. ---------------------------------------------------------------------- Lacak2 · 2013-05-14 12:18 @bambamns: do you have in your dBase files (encoded in CP852) some characters which can not be expressed in your Windows (I expect, that you are working under Windows here) non-unicode ansi system code page ? I expect, that you have under Windows Windows-1250 CP, if it is so, then IMO there can be translated all characters from CP852 to Win1250. Which ODBC driver do you use when accessing dBase files ? Did you set up correctly code page in driver configuration ? (AFAIK original DBase files does not support stored CP in header of file ... so you must somewhere explicitly tell what encoding is used) ---------------------------------------------------------------------- bambamns · 2013-05-14 13:01 Quotedo you have in your dBase files (encoded in CP852) - Yes QuoteI expect, that you are working under Windows here - Yes QuoteI expect, that you have under Windows Windows-1250 CP, if it is so, then IMO there can be translated all characters from CP852 to Win1250.- Yes QuoteWhich ODBC driver do you use when accessing dBase files ? - I'd tried     Microsoft dBase driver (Windows 7 32bit)     Microsoft Access dBase driver (Office 2010 32bit) QuoteDid you set up correctly code page in driver configuration ? - I hope I have - Code: [Select]   //Driver := 'Microsoft dBASE Driver (*.dbf)';     Driver := 'Microsoft Access dBASE Driver (*.dbf, *.ndx, *.mdx)';     Params.Add('DBQ='+FolderDBF);     Params.Add('EXCLUSIVE=1');     Params.Add('CHARSET=852'); I was using TDBF component to access to DBF files , but TDBF have bugs and sometimes it won't write data in database, so I switched to ODBC. With TDBF - the same files are converted with CP852TOUTF8 with no problems. ---------------------------------------------------------------------- BigChimp · 2013-05-14 13:27 You're welcome to try with the FPC trunk version of TDBF and report bugs. I'll have a look at them. ---------------------------------------------------------------------- bambamns · 2013-05-14 19:55 @BigChimp - I'll try , but I have spent a whole week to change my app. to use ODBC, so I need some time to bring things back. @ludob - I changed odbcsql.inc , rebuid Lazarus and my app, but I still have the same results.              - Do you have any other idea ? Thank you ---------------------------------------------------------------------- ludob · 2013-05-15 06:19 Assuming you build odbcsql.inc correctly, the most probable problem is, as Lacak2 indicated, a problem with the ODBC driver not recognizing correctly the code page and/or not succeeding to convert all characters correctly. Can you attach a stripped down version of your dbf file? Remove all data except for a few records that don't translate correctly. If not possible, make a hexdump of the first $400 bytes and somewhere in the middle showing a string that is not converted correctly. That would allow us (BigChimp for TDBF) to see where things go wrong. ---------------------------------------------------------------------- bambamns · 2013-05-17 04:03 Hi , Quote ....a problem with the ODBC driver not recognizing correctly the code page and/or not succeeding to convert all characters correctly. It seems the problem is something like that. As I wrote I had tried to use :     Microsoft dBase driver (Windows 7 32bit)     Microsoft Access dBase driver (Office 2010 32bit) with no success, but I found one more driver - Microsoft Visal Fox Pro (on Windows 7 it is from 1999) - I changed ODBC connection string and after that I had success. Unfortunately then come up some other problem - it has some strange behavior when I write to DBF file and close it - it is still locked ..... so I give up from ODBC - DBF combination - it is slow,  it has different drivers for each windows .... it's nightmare ..... Back to TDBF .... I've changed code back to use TDBF and set up some loggers to catch the problem......in mean time I'll test the new fpc trunk . @Big Chimp  - Can you confirm that spurce for dbf is on http://svn.freepascal.org/svn/fpc/trunk/packages/fcl-db/src/dbase/ ? Thank you ---------------------------------------------------------------------- BigChimp · 2013-05-17 06:11 Back to TDBF .... I've changed code back to use TDBF and set up some loggers to catch the problem......in mean time I'll test the new fpc trunk . @Big Chimp  - Can you confirm that spurce for dbf is on http://svn.freepascal.org/svn/fpc/trunk/packages/fcl-db/src/dbase/ ? [/quote] Yep, that's the FPC trunk repo with the newest tdbf source. Let me know (e.g in this thread) if you hit any problems.