How do I concentate multiple small SdfDataSets into a single large
SdfDataSet?
lazarus / 16-Databases

----------------------------------------------------------------------
funk247 · 2014-09-17 12:22

I'm importing data from many csv files which is to be exported to SQL
insert statements as part of a data transfer project.

The data to be imported is in the form of many small tables, so what I
would like to do is import each individually and combine them into 1
large table so that I can pick and choose which columns to export as
some of them may not be required. I also need to clean and reformat
some of the data before exporting it but have already created the
routines to cater for this.

Importing the individual tables is no issue but I've reached a mental
block when it comes to combining them, what would the best way to
proceed currently I'm thinking along these lines?

>> Loop through files in dir
>> Open file
>> Loop through headers
>> Create Schema for receiving dataset
>> Loop through rows
>> Clean data
>> Insert into receiving dataset
>> Close file
>> Next file

There's about 10 files I need to combine atm but the solution will be
expanded so ideally I'm looking for something flexible.

Hopefully someone has had experience with this kind of thing and can
help?