UTF8 & Locale lazarus / 16-Databases ---------------------------------------------------------------------- JZS · 2015-04-27 06:03 Hello everyone, Until recently I was thinking that I understand UTF8 and system locale, but I discovered there is a missing link in my applications. Some of my clients use Arabic, so I use UTF8 collation in my DB tables, and when viewing the data from my application everything looks OK as long as the system locale is set to Arabic. But there is a scenario that I need to handle, apparently is annoying to miss. When a user with System Locale set to Arabic adds some arabic text, all other users, with Arabic Locale, are able to read, but non-arabic locale see ?????, instead of the real text, also the other way around, non-arabic locale users when adding arabic text all non-arabic locale users are able to read, but those who have system locale set to arabic see ?????. How to make the text appears in both configuration correctly? I know the difference about UTF8 and ASCII, in bytes and char representation,  but how can I enable both users to see what others enter regardless the system locale: 1. Does it involve collation of my DB Tables (cp1256 <=> UTF8)? 2. Or, I encode (SysToUTF8?) when adding arabic text to my UTF8 Collation Tables? Appreciate any input on this subject. I use Laz 1.4 & MariaDB 10. Regards, ---------------------------------------------------------------------- JuhaManninen · 2015-04-27 06:41 Quote from: JZS on April 27, 2015, 08:03:57 amUntil recently I was thinking that I understand UTF8 and system locale ... I know the feeling QuoteWhen a user with System Locale set to Arabic adds some arabic text, all other users, with Arabic Locale, are able to read, but non- arabic locale see ????? It means you are dealing with ASCII + codepages. If everything was in UTF-8, you would not see ????? QuoteHow to make the text appears in both configuration correctly? By using our improved Unicode system :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus Then if your DB fully supports UTF-8, you don't need to worry about encodings (not much anyway). ---------------------------------------------------------------------- LacaK · 2015-04-27 07:57 How do you have defined connection charset between your application and MariaDB ? ---------------------------------------------------------------------- JZS · 2015-04-27 07:58 Quote from: JuhaManninen on April 27, 2015, 08:41:24 amIt means you are dealing with ASCII + codepages. If everything was in UTF-8, you would not see ????? I knew UTF8 will complicate things Quote from: JuhaManninen on April 27, 2015, 08:41:24 amBy using our improved Unicode system :   http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus Then if your DB fully supports UTF-8, you don't need to worry about encodings (not much anyway). So if I understand correctly then I must use FPC 2.7.1+ to enable default usage of UTF8? If yes, can I use FPC 2.7.1 (3.x?) with 1.4? I will download the latest from trunk just in case. Thank you Juha. ---------------------------------------------------------------------- JZS · 2015-04-27 07:59 Quote from: LacaK on April 27, 2015, 09:57:18 amHow do you have defined connection charset between your application and MariaDB ? Always UTF8 ---------------------------------------------------------------------- JuhaManninen · 2015-04-27 08:20 Quote from: JZS on April 27, 2015, 09:58:38 amI knew UTF8 will complicate things No, quite the opposite. Once everything is UTF-8, it will simplify things. QuoteSo if I understand correctly then I must use FPC 2.7.1+ to enable default usage of UTF8? If yes, can I use FPC 2.7.1 (3.x?) with 1.4? FPC trunk is now 3.1.1. You can use it or the fixes_3.0 branch which will soon (?) become the FPC 3.0 release. FPC 2.7.1 is history. ---------------------------------------------------------------------- JZS · 2015-04-27 08:37 Quote from: JuhaManninen on April 27, 2015, 10:20:33 amNo, quite the opposite. Once everything is UTF-8, it will simplify things. I hope so. Quote from: JuhaManninen on April 27, 2015, 10:20:33 amFPC trunk is now 3.1.1. You can use it or the fixes_3.0 branch which will soon (?) become the FPC 3.0 release. FPC 2.7.1 is history. Do you mean this one (for win32): ftp://ftp.freepascal.org/pub/fpc/snapshot/fixes/i386-win32/ After downloading, shall I just set the compiler under Lazarus 1.4 to this new fixes, or I need to rebuild lazarus from source with this new compiler? ---------------------------------------------------------------------- JuhaManninen · 2015-04-27 09:49 Quote from: JZS on April 27, 2015, 10:37:21 amDo you mean this one (for win32): ftp://ftp.freepascal.org/pub/fpc/snapshot/fixes/i386-win32/ Yes it looks right. I installed with fpcup myself. QuoteAfter downloading, shall I just set the compiler under Lazarus 1.4 to this new fixes, or I need to rebuild lazarus from source with this new compiler? You need Lazarus trunk, not the 1.4 release. Set the compiler and its sources in Lazarus options, then rebuild Lazarus. Then you are ready to go. ---------------------------------------------------------------------- JZS · 2015-04-27 10:09 I am going to do that. Will get back if anything goes wrong. Many Thanks Juha, you are a great help, as always. Regards, J ---------------------------------------------------------------------- JZS · 2015-05-03 14:45 Hi Juha, I managed to install FPC and Lazarus from trunk. FPC 3.1.1 revision: 30773 Lazarus 1.5 revision: 48923 Good news: I got FPC 3.1.1 & Lazarus 1.5 both built successfully using fpc 2.6.4. Bad news is I can't run any program or rebuild lazarus, I guess there is something I am missing here. Steps of what I did (as I remember): 1. Built fpc & lazarus by using Martin's directions: Code: [Select]PATH=C:\lazarus\fpc\2.6.4\bin\i386-win32; SET FPC=C:\lazarus\fpc\2.6.4\bin\i386-win32\fpc.exe make.exe clean  distclean make.exe all  OPT="-gw -godwarfset -O-1" make.exe install  INSTALL_PREFIX=c:\FPC\trunk  COPYTREE=echo Code: [Select]C:\lazarus\fpc\2.6.4\bin\i386-win32\make clean bigide 2. Started the new Lazarus with> startlazarus.exe --pcp="New Config Path" 3. Ran Lazarus and filled in the Compiler/Make/Source/Debugger fields, as follows: - Compiler=The trunk fpc that I build -> OK (needed to copy the fpc.cfg over, from 2.6.4) - Make= from 2.6.4 because there is no make.exe in the new built fpc dir. - Source= from 2.6.4, again not found in the new fpc. - Debugger= copied "mingw" directory from Laz 1.4 installation, (I hope this one did not make a big difference). 4. Tried to run the default started project, I got an error message about Unit "Interfaces" then I did step 2 again, then it never showed up again. 5. Tried to rebuild Lazarus (from the menu -> Tools -> Build Lazarus...), got this error: registerfcl.pas(43,3) Fatal: Cannot find ServiceManager used by RegisterFCL of package FCL. So now am out of tricks, need some push please. Regards, ---------------------------------------------------------------------- zeljko · 2015-05-03 14:54 You're probably missing PPC_CONFIG_PATH if fpc 3.1.1 isn't in the path. EDIT: PPC_CONFIG_PATH means where fpc.cfg is. ---------------------------------------------------------------------- JZS · 2015-05-03 17:21 Quote from: zeljko on May 03, 2015, 04:54:23 pmYou're probably missing PPC_CONFIG_PATH if fpc 3.1.1 isn't in the path. EDIT: PPC_CONFIG_PATH means where fpc.cfg is. i can get Lazarus run & rebuild with fpc 2.6.4, but can not get it working with fpc 3.1.1. How can I get the missing files (fpc.cfg, make.exe, probably others?) for 3.1.1? ---------------------------------------------------------------------- JZS · 2015-05-03 17:46 I tried "fpcmkcfg.exe -o fpc.cfg" but that was not accepted (i.e could not run or rebuild Lazarus). I guess there is one step missing to link 1.5 with 3.1.1. ---------------------------------------------------------------------- totya · 2015-05-03 20:17 Fcpup sometimes working for me, under windows. You can get fpc and lazarus svn at once. https://github.com/LongDirtyAnimAlf/Reiniero- fpcup/tree/master/bin/i386-win32 I tired it and working in few times, but isn't perfect yet. ---------------------------------------------------------------------- JZS · 2015-05-04 13:05 Thanks totya. fpcup is refusing to work, for sometime now (although it did work for me before), that's why I am in this quest. I even installed Getlazarus, but same thing, did not rebuild, complaining about Interface unit. No matter what I did, the different combinations of 1.5 and 3.1.1 did not work out. The Interface unit error message when run the default project is: project1.lpr(9,3) Fatal: Cannot find Interfaces used by Project1. Make sure all ppu files of a package are in its output directory. ppu in wrong directory=C:\Development\Lazarus\lcl\units\i386- win32\win32\interfaces.ppu. Clean up package "LCL".. ---------------------------------------------------------------------- DonAlfredo · 2015-05-04 13:16 Quotefpcup is refusing to work, for sometime now (although it did work for me before) I am trying to maintain fpcup. Could you elaborate on the problems you encounter with the new fpcup ? Thanks ! ---------------------------------------------------------------------- totya · 2015-05-04 15:42 Quote from: DonAlfredo on May 04, 2015, 03:16:03 pmQuotefpcup is refusing to work, for sometime now (although it did work for me before) I am trying to maintain fpcup. Could you elaborate on the problems you encounter with the new fpcup ? Thanks ! Okay, this is not for me, but always in first time (now with rev23), on windows x64 with: fpcup.exe --fpcURL="trunk" --lazURL="trunk" --verbose I got: ppc386.exe appcrash and fpc.exe appcrash twice, under lazarus build. In this rev, lazarus icon doesn't created on desktop (this isn't a big problem). edit.: I see startlazarus.exe is missing Default project (after rescan fpc directory and Set utf8 in rlt pressed) compiled and run without any problem. ---------------------------------------------------------------------- DonAlfredo · 2015-05-04 16:41 Yep. I have seen those errors also (sometimes, not always). And also sometimes miss the shortcut. And I do not know what happens. Keep in mind that : trunk = risky ! (fpc 3.1.1; lazarus 1.5) default = ok ! (fpc 3.0 fixes; lazarus 1.4) stable = ok ! (fpc 2.6.4; lazarus 1.4) fpcup.exe --fpcURL="default" --lazURL="default" will give you a very good system ---------------------------------------------------------------------- totya · 2015-05-04 16:49 Quote from: DonAlfredo on May 04, 2015, 06:41:09 pmfpcup.exe --fpcURL="default" --lazURL="default" will give you a very good system Hi! Thanks for this information! http://forum.lazarus.freepascal.org/index.php/topic,28111.msg175152.ht ml#msg175152 QuoteYes, correct. It still requires development versions of both FPC and Lazarus, as Taazz and Felipe noted. FPC can be either trunk or the fixes_3_0 branch. Okay, I need only the fixes_3_0 branch of full UTF-8 support, not the svn... But your all (windows) bat file is "svn".. See: https://github.com/LongDirtyAnimAlf/Reiniero- fpcup/tree/master/bin/i386-win32 ...and I think the "PAUSE" command is good idea to the end of the (windows) batch file... ---------------------------------------------------------------------- DonAlfredo · 2015-05-04 17:00 You are right. I will change the scripts ! ---------------------------------------------------------------------- JZS · 2015-05-05 07:59 Quote from: DonAlfredo on May 04, 2015, 03:16:03 pmQuotefpcup is refusing to work, for sometime now (although it did work for me before) I am trying to maintain fpcup. Could you elaborate on the problems you encounter with the new fpcup ? Thanks ! Which new fpcup? I did not know there is a new one. I have been using the one of Reiniero's. Anyways the errors I got are: 1. It starts with unable to download the svn.exe through the Temp folder, but that one I overcame easily by downloading manually and putting in the bootstrap folder under svn. 2. One error I got, complaining about ppc386.exe but I did not do anything about and after sometime while I keep trying, it never happened again (probably redownload fixed it). Also one occurrence was about rm.exe but I do not remember what it was about. I deleted everything and started from zero, again. 3. Then it gets interrupted with code 64, every now and then, about no update in fpc source, or decompression error (ra_serf: An error occurred during decompression), but I guess this last one due to my connection, which requires cleanup but it did not continue after cleanup, until yesterday. This morning, surprisingly it did pass the yesterday's point that I was stuck at, and it's downloading now. Let's see if it completes. ---------------------------------------------------------------------- DonAlfredo · 2015-05-05 08:06 https://github.com/LongDirtyAnimAlf/Reiniero-fpcup ---------------------------------------------------------------------- JZS · 2015-05-05 08:18 Many thanks DonAlfredo for maintaining fpcup. I can spare sometime, and will be very glad to contribute to Reinier's project. So please, if I can help with anything, for BigChimp efforts be maintained, just let me know. I did not try that one, (I do not want to interrupt the running instance, it almost finished, it is compiling Lazarus source now), but if I may suggest things I noticed in the old one, probably some have been implemented already in the new fpcup: 1. Allow downloading the fpc or laz source as zipped files (with browse and locate option) and they get extracted in the fpcup development\fpc & development\lazarus folders as an extra option next to download through svn. In slow connections it makes a difference and saves time. 2. Have the download and all the exe3cution done in a secondary thread, giving the possibility to scroll, (pause, cancel, resume)?, without affecting the GUI. 3. No need to show the Dos window, if the shell is executing, then it is fine, it could be run as hidden. ---------------------------------------------------------------------- JZS · 2015-05-05 08:57 Continue on suggestions... 4. If downloading is interrupted no need to go through the entire compilation process. Just have milestone for progress. for instance if fpc is compiled successfully, and progress advanced no need to go back to it if restarted. Or just download everything first, (i.e process whatever most-prone-to-interruption first), and then compile everything at once. It just happened to me now, after compiling fpc and lazarus sources, while at downloading "Extras" the download was interrupted with checksum mismatch error at foot"something".png and when restarted it has gone through recompiling everything again. I apologize if this has already been figured out, I just can't check the new fpcup now, and I am afraid I'll forget what I've just gone through now, if I wait for later. ---------------------------------------------------------------------- JZS · 2015-05-05 09:27 Hooray, it did work. But I had to interrupt the fpcup svn download and put the fpc & lazarus sources that I downloaded separately in place where it (fpcup) expected to find them (sources). Otherwise it kept giving decompression error every minute, and I had to cleanup and continue, which is so annoying. Also for "extras" I had to continue the download with svn client (TortoiseSVN) before I resume fpcup process to avoid interruption and restart compiling before hitting another interruption to repeat and wast time. Finally I got Laz1.5 and fpc 3.1.1 (compiles and rebuilds). Now I can say, it's a success. Thanks DonAlfredo, I will try the new fpcup soon. Regards, J ---------------------------------------------------------------------- DonAlfredo · 2015-05-05 13:09 Please report back your findings with the new fpcup ! 1 previous) svn problem is solved 1) SVN only downloads the changes AFAIR. So little bandwidth after the initial download. But I will look into file download. 2) fpcup itself is command line ... no GUI ! Do you use the graphic shell ? 3) See 2 4) you can limit fpcup by using the --only or --skip commands ! Edit : additional As an example. After a vanilla (and very clean and limited) install with fpcup, you can do Code: [Select]fpcup.exe --only="indy" --verboseto install indy. Or Code: [Select]fpcup.exe --only="bgrabitmap,bgracontrols" --verboseto get the bgra framework. Or Code: [Select]fpcup.exe --only="wst" --verboseto get the Web Service Toolkit. This is valid for all included modules in fpcup.ini !! ---------------------------------------------------------------------- JZS · 2015-05-10 08:42 Quote from: DonAlfredo on May 05, 2015, 03:09:35 pmPlease report back your findings with the new fpcup ! 1 previous) svn problem is solved 1) SVN only downloads the changes AFAIR. So little bandwidth after the initial download. But I will look into file download. 2) fpcup itself is command line ... no GUI ! Do you use the graphic shell ? 3) See 2 4) you can limit fpcup by using the --only or --skip commands ! Edit : additional As an example. After a vanilla (and very clean and limited) install with fpcup, you can do Code: [Select]fpcup.exe --only="indy" --verboseto install indy. Or Code: [Select]fpcup.exe --only="bgrabitmap,bgracontrols" --verboseto get the bgra framework. Or Code: [Select]fpcup.exe --only="wst" --verboseto get the Web Service Toolkit. This is valid for all included modules in fpcup.ini !! For simplicity, I tend to use fpcupGUI. Upon compiling fpcup.lpi, and at unit "installerCore" line 984: Code: [Select]                {$IFDEF DARWIN}                 DiffFileSL.TextLineBreakStyle:=tlbsCRLF;                 {$ELSE}                 DiffFileSL.LineBreak:=#13#10;                 {$ENDIF} I got error at "LineBreak" as no member of "DiffFileSL" and I had to replace the five lines with the following line to be able to compile: Code: [Select]DiffFileSL.TextLineBreakStyle:=tlbsCRLF; It seems working OK, I started it and watched it downloading the fpcbootstrap, but because of my slow connection I can not keep it running to finish, it will take too long. I know that might not be a convenient option for others with high speed connection, and there is no any reason for you to follow, but am just saying my opinion here, you can cut the download time, as per my last time experience, way less than quarter the current download time, if it was possible to download zip files. i.e: 1. fpcbootstrap.zip 2. fpcup.zip 3. lazarus.zip 4. extras.zip Then run fpcup with the option to look for local files to install, that would be much faster than the overhead of the current downloading rate... Regards, J PS: Am on Win8 32Bit ---------------------------------------------------------------------- DonAlfredo · 2015-05-10 10:02 Thanks for reporting ! Changed the line-ending problem. Again, fpcup is designed to limit bandwidth !! Binutils and bootstrap are only downloaded once. Fpc source is only downloaded once. As is Lazarus. As are the extras. As are all other svn, git and hg sources !!. So absolutely no gain in using zips for source. The extras are user dependent. A zip would be huge if all packages are to be included. ---------------------------------------------------------------------- JZS · 2015-05-10 12:03 Quote from: DonAlfredo on May 10, 2015, 12:02:34 pmAgain, fpcup is designed to limit bandwidth !! Binutils and bootstrap are only downloaded once. Fpc source is only downloaded once. As is Lazarus. As are the extras. As are all other svn, git and hg sources !!. Sorry DonAlfredo, was not referring to change the mechanism, fpcup design or how the versions and revision control system works. It was all based on my personal experience during specific situation, so I was thinking of one additional option, for some other users who are having my same issue of slow connection, and when there will be no future revision updates. Kinda of thinking out loud. Quote from: DonAlfredo on May 10, 2015, 12:02:34 pmSo absolutely no gain in using zips for source. It's just a matter of quick process when utilizing zipped sources and compile them and get the final trunk version ready, instead of waiting for hours to download, and resume when connection gets interrupted, specially when no future updates needed, and on top of them all, sources are already available as zips in acceptable sizes @ ftp://ftp.freepascal.org/pub/ Please forget I suggested that, I was just expressing an idea, could be silly for experienced Lazarus guru with high speed connection  Regards, ---------------------------------------------------------------------- DonAlfredo · 2015-05-10 12:28 Yes ... 100Mb/s ... up/down ...  But I am willing to look at your situation ! Please tell me exactly. From : ftp://ftp.freepascal.org/pub/ What do you download. Because it has sources, binaries, .... To make it more easy: give me the links to the files you use ! ---------------------------------------------------------------------- JZS · 2015-05-10 12:55 Quote from: DonAlfredo on May 10, 2015, 02:28:34 pmYes ... 100Mb/s ... up/down ...  But I am willing to look at your situation ! Please tell me exactly. From : ftp://ftp.freepascal.org/pub/ What do you download. Because it has sources, binaries, .... To make it more easy: give me the links to the files you use ! Just to be clear, I did not use them successfully. I ended up using the svn to complete the download and finalized with fpcup. I was thinking about the possibility of making fpcup using the following sources (as an extra separate option to compile only without svn): ftp://ftp.freepascal.org/pub/fpc/snapshot/trunk/i386-win32/ ftp://ftp.freepascal.org/pub/lazarus/snapshots/ ---------------------------------------------------------------------- DonAlfredo · 2015-05-10 13:17 Thanks for the info. Could you please explain (again) to me how you plan to use fpcup when using only fpc binaries. This ftp://ftp.freepascal.org/pub/fpc/snapshot/trunk/i386-win32/ is a link to a FPC binary distribution for windows (32bit). Fpcup cannot do anything with these binaries. Fpcup can only do something usefull (compile) with sources ! You can use these binaries (FPC as well as Lazarus) out-of-the-box. I think I am misunderstanding you somehow ! ---------------------------------------------------------------------- JZS · 2015-05-10 14:00 Let me explain how I arrived to the idea, probably it will make things clearer. Before I managed to make fpcup run successfully I was looking for a way to avoid downloading via fpcup so I just downloaded those sources and via "Make.exe" I was able to get 1.5, but due to the fact that fpc2.6.4's make.exe is used the generated Lazarus is always hooked up to FPC2.6.4. So I spent long time trying to link Laz1.5 and fpc3.1.1 from the sources but in vain, nothing worked out, I was always getting Laz1.5 with fpc2.6.4, upon changing the links (under Lazarus option) to point to fpc3.1.1, it errors out for missing file(s). So after using fpcup with the sources that I already downloaded via tortoise SVN and just copy 'n' paste to fpcup development folder and it compiled and installed without issues, I thought there would be a simple way to make those sources available, and just point fpcup to finish up and installing. It seems like deviation of fpcup function, but under the same objective (installing Lazarus & fpc). I know that sounds trivial task for fpcup in comparison to it's main functions. But after spending some time trying to download from svn and it fails and the need to cleanup and resume, it looks a great task to just add such simple (?) functionality to fpcup (i.e find the sources ready and just compile and link). Hope that makes sense. ---------------------------------------------------------------------- DonAlfredo · 2015-05-10 14:15 Well, call me stupid ... ... but I still do not understand ! The FPC link you presented does not contain sources ! It contains a binary for win32 !! Again, fpcup cannot do anything without sources !!! Fpcup uses sources to create a working FPC / Lazarus environment on- the-fly ! Adding a functionality to let fpcup use plain sources from a zipfile or a directory with sources would not be very difficult to implement. But then again, I need to understand what the benefit is for such a feature and how to implement it to fit your needs. And the problem is: I still do not understand what your needs are (see comment about link). ---------------------------------------------------------------------- JZS · 2015-05-10 14:25 Quote from: DonAlfredo on May 10, 2015, 04:15:04 pmWell, call me stupid ... ... but I still do not understand ! The FPC link you presented does not contain sources ! It contains a binary for win32 !! Again, fpcup cannot do anything without sources !!! Fpcup uses sources to create a working FPC / Lazarus environment on- the-fly ! Adding a functionality to let fpcup use plain sources from a zipfile or a directory with sources would not be very difficult to implement. But then again, I need to understand what the benefit is for such a feature and how to implement it to fit your needs. And the problem is: I still do not understand what your needs are (see comment about link). Sources are available: ftp://ftp.freepascal.org/pub/fpc/snapshot/trunk/source/ The benefit is in the difference when downloading 44.2 MB zip file, and 482 MB svn. I could be aiming too high but sure this saves a lot of time. ---------------------------------------------------------------------- DonAlfredo · 2015-05-10 17:21 Ok. Now I understand. This is already possible with fpcup. For FPC, just do Code: [Select]fpcup --only="FPCCleanOnly,FPCBuildOnly" For Lazarus, just do Code: [Select]fpcup --only="LazarusCleanOnly,LazarusBuildOnly,LazarusConfigOnly" So, tell fpcup to skip the getthesource part ! And only run fpcup with all source files already in (the correct) place ! ---------------------------------------------------------------------- JZS · 2015-05-12 08:46 Thank you DonAlfredo.