SDF Dataset Multiline Usage and Other Questions lazarus / 16-Databases ---------------------------------------------------------------------- ericlaw02 · 2012-09-14 17:59 Hi there, I'm a newbie to Lazarus and GUI/RAD programming, coming from command-line Free Pascal/Turbo Pascal, so I'd like to say sorry first if I look ignorant. I'm writing a GUI-based application on Lazarus which has a function to store records (eg. name, address, memos, etc.), after some researching on the wiki, I opted to write that part using an SDF dataset with data aware controls for it's simplicity, since I had no prior SQL experience, and I wanted the database to be self-sufficient, and not need any external applications to edit/create the database. However, when I was tried to use a multiline TDBMemo control, the multiline function didn't work. (SdfDataSet1.AllowMultiLine set to true in object inspector) I inputted Code: [Select]kjkljk kljkljk into the memo field for John Doe (as below), and I checked the file output afterwards: The output ended up being something like this Code: [Select]Surname,GivenName,memo Doe,Jane, Doe,John,kjkljk kljkljk I did some further inspection, and noticed that the setting AllowMultiLine (in sdfdataset1) in object inspector resets everytime I reopen Lazarus, so I suspect there's a bug, so I tried to assign "SdfDataSet1.AllowMultiLine:=True;" manually in the form code, and the file output ended up to be: Code: [Select]Surname,GivenName,memo Doe,Jane, Doe,John,"kjkljk kljkljk" However, in GUI viewing the DBMemo that holds the field "memo", I can only see one line ("kjkljk"), and "kljkljk" ended up being the first field for the next record. The question is, how should I implement this properly? Else, if it couldn't be implemented, are there any other methods of achieving the goal? Another question is, is it possible to implement binary search in TSDFDataSet? As far as I know, TDataSet.Locate is broken, and it is necessary to build a locate/searching code manually, but so far I have only been able to write a simple linear search using TDataset First/Next/EOF commands. Sorry if the post ended up being too long winded, I'm not an native English speaker, and I could elaborate more if you don't understand what I'm asking. I'm currently using Lazarus 1.0 (2012-08-28), FPC ver. 2.6.0, SVN rev. 38415, x86_64-win64-win32/win64 build and on 64-bit Windows 7 zh-TW locale if that helps. Thanks! ---------------------------------------------------------------------- BigChimp · 2012-09-14 18:28 Unfortunately, the multiline property for sfdataset has not yet been implemented for reading (see bug (http://bugs.freepascal.org/view.php?id=22213 - tucked away in the comments unfortunately). If you want to look into patching sdfdataset, that would be great! This, IIRC, also applies to ZMSQL, which is based on sdfdataset. csvdocument does implement support for multiline data but it's not a dataset; you'll have to manipulate the data yourself (load/save) and e.g. display in a stringgrid. I don't think a stringgrid will show line endings (though it will gladly save them); see e.g. http://lazaru s.freepascal.org/index.php/topic,17113.msg101008.html#msg101008 You might have better luck with the native dbf support. Even though there's deprecation warnings, it may work well for you (and apparently well presented patches are accepted - if not I'll certainly help by writing test cases etc). Don't know by heart if/how it supports multiline data but it may very well. ---------------------------------------------------------------------- ezlage · 2012-09-14 18:29 Edit the LFM file, and set TRUE to AllowMultiLine manually. See the attachment. But friend, why not use TDBF? I'm used with TDBF. Locate and many others things works well. Regards. Sorry by my poor English. ---------------------------------------------------------------------- BigChimp · 2012-09-14 18:41 Quote from: ezlage on September 14, 2012, 08:29:35 pmEdit the LFM file, and set TRUE to AllowMultiLine manually. See the attachment. Ezequiel: are you sure you can read multiline data with sdfdataset? My tests fail (see the bug I posted). Thanks, BigChimp ---------------------------------------------------------------------- BigChimp · 2012-09-14 18:44 Quote from: BigChimp on September 14, 2012, 08:28:14 pmUnfortunately, the multiline property for sfdataset has not yet been implemented for reading (see bug (http://bugs.freepascal.org/view.php?id=22213 - tucked away in the comments unfortunately). Added a new bug report to make sure we don't lose track of this: http://bugs.freepascal.org/view.php?id=22882 It's still on my to do list, though unfortunately not at the top. Any help appreciated. Thanks, BigChimp ---------------------------------------------------------------------- ezlage · 2012-09-14 18:46 BigChimp, I have not tested. I thought that it works.  Thanks. ---------------------------------------------------------------------- taazz · 2012-09-14 18:53 Quote from: BigChimp on September 14, 2012, 08:44:54 pmQuote from: BigChimp on September 14, 2012, 08:28:14 pmUnfortunately, the multiline property for sfdataset has not yet been implemented for reading (see bug (http://bugs.freepascal.org/view.php?id=22213 - tucked away in the comments unfortunately). Added a new bug report to make sure we don't lose track of this: http://bugs.freepascal.org/view.php?id=22882 It's still on my to do list, though unfortunately not at the top. Any help appreciated. Thanks, BigChimp hmm, I think I can tackle this one. One question before I start. Has the SDFData.pp changed from the 1.0 release or is it the same? ---------------------------------------------------------------------- BigChimp · 2012-09-14 19:07 Quote from: taazz on September 14, 2012, 08:53:20 pmQuote from: BigChimp on September 14, 2012, 08:44:54 pmQuote from: BigChimp on September 14, 2012, 08:28:14 pmUnfortunately, the multiline property for sfdataset has not yet been implemented for reading (see bug (http://bugs.freepascal.org/view.php?id=22213 - tucked away in the comments unfortunately). Added a new bug report to make sure we don't lose track of this: http://bugs.freepascal.org/view.php?id=22882 It's still on my to do list, though unfortunately not at the top. Any help appreciated. Thanks, BigChimp hmm, I think I can tackle this one. One question before I start. Has the SDFData.pp changed from the 1.0 release or is it the same? Great. It's an fpc unit though, not a Lazarus unit. It's been changed a bit since 2.6.0 (the FPC version Lazarus ships with) - especially with regards to multiline support. I think you can safely download the trunk version from svn.freepascal.org/svn/fpc/trunk/packages/fcl-db/src/sdf/sdfdata.pp create a backup of your 2.6.x version and copy it over, and work from there, though. Alternatively, you could use e.g. fpcup or svn to check out an FPC trunk+Lazarus trunk release... ---------------------------------------------------------------------- taazz · 2012-09-14 19:41 Quote from: BigChimp on September 14, 2012, 09:07:34 pmGreat. It's an fpc unit though, not a Lazarus unit. It's been changed a bit since 2.6.0 (the FPC version Lazarus ships with) - especially with regards to multiline support. I think you can safely download the trunk version from svn.freepascal.org/svn/fpc/trunk/packages/fcl-db/src/sdf/sdfdata.pp create a backup of your 2.6.x version and copy it over, and work from there, though. Alternatively, you could use e.g. fpcup or svn to check out an FPC trunk+Lazarus trunk release... Ok, got the file, from a fast look around it does not have any code for reading/writing data from/to the disk. Everything is handled in the TStringlist class so it makes no sense to try to patch the SDF datasets. So far I'm thinking to subclass Tstringlist by 1) Introduce a new property QuoteChar :Char; (or something along those lines) 2) override its SettextStr virtual method and add my own processing as needed. This will create the list impact on the final code, of course I'm open to other ideas or recommendations. Sorry I'm not comfortable enough to work with the svn repository directly yet. so I'm going to work on that file for now. I Will look at it this weekend. Regards Jo ---------------------------------------------------------------------- taazz · 2012-09-15 03:16 Ok, I managed to finish it. Please find attached the complete file sdfData.pp and my data file that I used for tests. Everything seems to work correctly in my end but since I did it after an allnighter I would appreciate if some one else could test it in case I missed something. In short the changes I've done ... 1) Corrected the bug with the AllowMultiline property ignoring the true state. 2) Created a new class TSDFStringList which can understand quoted fields 3) TSDFStringList has a property QuoteChar when it is set to #0 then it works as its ancestor every crlf is a new record this way the Fixed Length dataset still works with out changes. 4) Removed the hard coded Quote <"> and made it a property of TSDFDataset named FieldQuote. Now you can select which Quote to use. Every single line of code that I changed has been marked with //jkoz XXXX short description. Where XXXX is a code that you can find on top of the unit with some kind of generic todo list. Now I need to reinstall lazarus 1.0 I deleted my sdfdata.pp trying to clean my directories up :/. ---------------------------------------------------------------------- ericlaw02 · 2012-09-15 07:14 First, thank you everyone for your replies! I tried taazz's fix as well, and here's my results: When I try to set my SdfDataSet to Active in Lazarus object inspector, an error window popups and says: (database.csv is my filename for the sdfdataset) Code: [Select]Unable to open file "database.csv" In addition, AllowMultiLine still reverts to False in Lazarus object inspector when reopening Lazarus. However, when editing the *.lfm file manually as suggested by ezlage, AllowMultiLine could be seen shown as "True" in object inspector, but setting it to "False" again reverts it to default behaviour. I also tried building a new application with tdfdataset to test if it was an issue due to my bad coding, but the issues still persist. I suspect the two issues above might be either an issue with my compiling of taazz's fixes, or an implementation issue in Lazarus, might try to have a clean install of Lazarus later. (or as an afterthought, it could be user error due to my newbie-ness) However, taazz's fixes works for me if I assign the SdfDataSet's "AllowMultiLine" and "Active" properties manually, with the TDBMemo showing multiline properly. It also saves multiline properly as well if I edit the TDBMemo. Thanks! ---------------------------------------------------------------------- BigChimp · 2012-09-15 07:49 Ericlaw, I think the problem is here: http://svn.freepascal.org/svn/fpc/trunk/packages/fcl- db/src/sdf/sdfdata.pp Code: [Select]published     property AllowMultiLine: Boolean read FFMultiLine write SetMultiLine default True; //Whether or not to allow fields containing CR and/or LFCode: [Select]constructor TSdfDataSet.Create(AOwner: TComponent); begin   inherited Create(AOwner);   FDelimiter := ',';   FFirstLineAsSchema := FALSE;   FFMultiLine :=False; The object inspector may be changing it to true but the constructor immediately changes it to false again when it gets created at runtime. Could you test with this: Code: [Select]published     property AllowMultiLine: Boolean read FFMultiLine write SetMultiLine default False; //Whether or not to allow fields containing CR and/or LFCode: [Select]constructor TSdfDataSet.Create(AOwner: TComponent); begin   inherited Create(AOwner);   FDelimiter := ',';   FFirstLineAsSchema := FALSE; If that works, I'll submit a patch (or you can do it if you want) so it gets fixed in FPC trunk. @taazz: thanks for your hard work, I'll have a look! ---------------------------------------------------------------------- taazz · 2012-09-15 08:29 the problem was there alright, but the problem he is having with the allowmultiselect is that lazarus does not have the changes compiled in not until he rebuilds the IDE. ---------------------------------------------------------------------- BigChimp · 2012-09-15 08:36 Sorry taazz, hadn't looked yet at your code. Doing so now. ---------------------------------------------------------------------- taazz · 2012-09-15 09:02 Quote from: BigChimp on September 15, 2012, 09:49:33 amEriclaw, I think the problem is here: http://svn.freepascal.org/svn/fpc/trunk/packages/fcl- db/src/sdf/sdfdata.pp Code: [Select]published     property AllowMultiLine: Boolean read FFMultiLine write SetMultiLine default True; //Whether or not to allow fields containing CR and/or LFCode: [Select]constructor TSdfDataSet.Create(AOwner: TComponent); begin   inherited Create(AOwner);   FDelimiter := ',';   FFirstLineAsSchema := FALSE;   FFMultiLine :=False; The object inspector may be changing it to true but the constructor immediately changes it to false again when it gets created at runtime. Now that I'm fully awake let me try to explain it a bit more. The problem is not the code inspector been overriden by the constructor, the problem is the component streaming mechanism. The default declaration is there so the streaming mechanism knows what is the default value and avoid saving it in the form file by setting default to true when the AlloMultiline is true it is not saved in the lfm. As far as I know the streaming mechanism takes plase after all the constructors have finished executing so there is no conflict between them at all. Be aware that this unit has been tampered so it can compile in older versions of the DB.pas file, I have marked the changes that need to be deleted before creating a patch make sure to delete them including the commented lines surrounding it. Quote from: BigChimp on September 15, 2012, 10:36:29 amSorry taazz, hadn't looked yet at your code. Doing so now. Don't worry. take your time I prefer to be tested thoroughly before creating a patch that is why I upload it with all the "debug" changes still active. ---------------------------------------------------------------------- BigChimp · 2012-09-15 09:06 Thanks, taazz, that explanation makes sense. ---------------------------------------------------------------------- ericlaw02 · 2012-09-15 10:28 Alright, so it seems the error ended up to be the thing between the keyboard and the chair  I skimped out the procedure (or rather, not reading instructions and just did it through trial and error) and ended up just recompiling the component didn't recompile the whole IDE. Reverted to taazz's original fixed code (I tried BigChimp's fix beforehand, but I forgot to report back), and AllowMultiLine seems to work properly in the IDE. However, setting SdfDataSet.Active to true in IDE still popups "Unable to open file" for my existing application, however, it works perfectly when I try it in an new application. So I tried adding an "Active = True" manually in the "object SdfDataSet1" portion of the project *.lfm, and it works. However, setting the "Active" property back to false in the IDE, and reopening Lazarus seems to have the problem again. It seems Lazarus automatically deletes the "Active" property in the *.lfm file if it's false (because it's the default setting?), so I tried to add "Active = False" instead in the *.lfm file, and it seems to work normally. Lazarus also puts the "Active" property below the FieldDefs in the *.lfm file as well, if I set it to "True" in the IDE. So I think Lazarus somehow freaks out now when the Active property isn't present in the *.lfm file? (or it could be an user issue again, but i'd hope not) ---------------------------------------------------------------------- BigChimp · 2012-09-15 11:55 Quote from: ericlaw02 on September 14, 2012, 07:59:48 pmAnother question is, is it possible to implement binary search in TSDFDataSet? As far as I know, TDataSet.Locate is broken, and it is necessary to build a locate/searching code manually, but so far I have only been able to write a simple linear search using TDataset First/Next/EOF commands. While going through taazz's fixes I stumbled on this note in the original code: Quote12/Mar/04  Lazarus version (Sergey Smirnov AKA SSY)       Locate and CheckString functions are removed because of Variant data type.       Many things are changed for FPC/Lazarus compatibility. As support for variants presumably has vastly improved since 2004, it may be possible to bring back the locate function... after looking for the original code ---------------------------------------------------------------------- taazz · 2012-09-15 12:01 Quote from: ericlaw02 on September 15, 2012, 12:28:42 pmHowever, setting SdfDataSet.Active to true in IDE still popups "Unable to open file" for my existing application, however, it works perfectly when I try it in an new application. As far as I remember sdfDataset.FileMustExist controls this message. if the file does not exist in the specified path then when ever you activate the database and this property is true you will see that message. If the file do exist then there is something wrong with your path that trips up the IDE eg. spaces in the directory names, unicode characters, non english directory names are a coupld of things to look for. ---------------------------------------------------------------------- taazz · 2012-09-15 12:12 Quote from: BigChimp on September 15, 2012, 01:55:24 pmQuote from: ericlaw02 on September 14, 2012, 07:59:48 pmAnother question is, is it possible to implement binary search in TSDFDataSet? As far as I know, TDataSet.Locate is broken, and it is necessary to build a locate/searching code manually, but so far I have only been able to write a simple linear search using TDataset First/Next/EOF commands. While going through taazz's fixes I stumbled on this note in the original code: Quote12/Mar/04  Lazarus version (Sergey Smirnov AKA SSY)       Locate and CheckString functions are removed because of Variant data type.       Many things are changed for FPC/Lazarus compatibility. As support for variants presumably has vastly improved since 2004, it may be possible to bring back the locate function... after looking for the original code depends what are the problems that variants introduced, are they support type of problems eg variants behave differently on various operating systems and CPUs(endianes of numbers etc)? or where they simple speed, data type problems? Of course that comment says nothing on why it had to be done so we have to get some kind of history before actually start looking for the code or ways to implement this. On a side note does SDF supprot indexing or sorting at all? and Which is the branch that holds the v1.0 release for lazarus and FPC? any one has managed to get a stable behavior from the debugger yet? On one time I can step in to lcl code with no problem the next minute there are no debug information no bullets to see on the lcl code no break points working in it etc but everything works on my application. Rand mode off. sorry about that had to vent. ---------------------------------------------------------------------- BigChimp · 2012-09-15 12:33 OT: taazz, I'm going to open a new topic on your code. My test code is probably to rough for it... Quote from: taazz on September 15, 2012, 02:12:15 pmWhich is the branch that holds the v1.0 release for lazarus and FPC? any one has managed to get a stable behavior from the debugger yet? Lazarus 1.0 http://svn.freepascal.org/svn/lazarus/tags/lazarus_1_0/ is released with FPC 2.6.0 http://svn.freepascal.org/svn/fpc/tags/release_2_6_0/ However, you may want to look into FPC 2.6 fixes (fixes from trunk are regularly backported there) and Lazarus SVN - that seems stable enough and gets a lot of bugfixes. You can use e.g. fpcup to get that by default. Otherwise, you could presumably use Lazarus 1.0 fixes http://svn.freepascal.org/svn/lazarus/branches/fixes_1_0/ though I haven't used it. ---------------------------------------------------------------------- taazz · 2012-09-15 12:56 Quote from: BigChimp on September 15, 2012, 02:33:26 pmOT: taazz, I'm going to open a new topic on your code. My test code is probably to rough for it... sure the more test cases I get the better the final result will be. Also this is a good opportunity to learn the ins and outs of the SQLDB framework. Quote from: BigChimp on September 15, 2012, 02:33:26 pmQuote from: taazz on September 15, 2012, 02:12:15 pmWhich is the branch that holds the v1.0 release for lazarus and FPC? any one has managed to get a stable behavior from the debugger yet? Lazarus 1.0 http://svn.freepascal.org/svn/lazarus/tags/lazarus_1_0/ is released with FPC 2.6.0 http://svn.freepascal.org/svn/fpc/tags/release_2_6_0/ However, you may want to look into FPC 2.6 fixes (fixes from trunk are regularly backported there) and Lazarus SVN - that seems stable enough and gets a lot of bugfixes. You can use e.g. fpcup to get that by default. Otherwise, you could presumably use Lazarus 1.0 fixes http://svn.freepascal.org/svn/lazarus/branches/fixes_1_0/ though I haven't used it. Thank you, I have in my todo list to set up an svn based environment at some point, Fcup is on my radar as well and I will probably poke you when that time comes but for now I need to find some time to clean up my hard drive, amongst other things.