Subj : Nodediffs & db:s
To   : Jesper Sörensen
From : Jan Vermeulen
Date : Tue Jan 07 2003 01:37 am

    Quoting Jesper Sörensen on Mon 6 Jan 2003 22:34 to Jan Vermeulen:

 js>>> The problem is how do you know what data to add, update or delete?

 JV>>     Diff-files tell you which records to add, which to replace, which
 JV>> to delete and which to leave alone.

 js> Well, yes, but the "records" are lines (nodes, hosts, regions,
 js> comments, errors etc.) in the text file nodelist, and the diff is
 js> useless if you don't have the original file. You don't know if the
 js> record to delete is a node (and its full node number if so).

    Have you ever done serious work on nodelists?

    If your diff says D5 you just delete those 5 lines. It is not necessary to 
go in savvy mode and salvage a record.

    And yes, of course, you will need to place each node between brackets as in 
<record> this is a text line <record/>.

 js>>> The diffs don't say "add 2:204/255..."; they say "copy (ignore) 17
 js>>> lines", "delete 3 lines", "add the following 4 lines" and so on, so
 js>>> you still need to have the original nodelist to be able to resolve
 js>>> the diff into something useful. :-(

 JV>>     You are expected to work on complete records that have been
 JV>> arranged in a given order -- what are you planning to do to the data
 JV>> that you can't locate your records anymore?

 js> I'm importing the data into a db table and use the Fidonet address
 js> as the primary key (but it's also indexed on e.g. sysop name &
 js> location for faster searches). I'm perfectly able to locate the
 js> data, just not via the line number the node had in the original
 js> nodelist (which would be a pita to work with since it's likely to
 js> change every week).

    The nodelist entries have no line numbers either. Their line number is 
their offset in lines from the start of the list, prolog included.

    You get a new diff each week to update last week's nodelist. As long as you 
have kept your records in order (and why shouldn't you?), you move them to an 
auxiliary base record by record, inserting the new nodes as per the diff 
telling you Axx, copying where it says Cxx and skipping where it says Dxx. Then 
kill the old file and rename the new one to the old one's name.

    Easy, ain't it? And that has been invented only eighteen years ago. It's 
just come of age...

 js> I'm sure it would be possible to make the table an exact copy of
 js> the nodelist file (with line numbers and everything) but it would
 js> be a very complex thing to work with and most db designers would
 js> probably start crying if they saw it. It's simpler to truncate the
 js> table and reload the entire nodelist from the updated file.

    You could, but you don't to have to.

 JV>>     Do you want to scatter your data all over the place without
 JV>> creating an index file or what?

 js> Of course not.

    I thought so ;-)

 js> If you know of a magic way to convert the nodediffs into SQL
 js> insert/update/delete commands please tell me how, but if you would
 js> try this yourself I'm sure you'd see the problem.

    I'm not going to look backwards, Jesper. All I can add here is to advise 
you to do some serious low-level programming. It's refreshing.

    Do you know there is even elegance in writing assembler?


    -=<[ JV ]>=-

 
* Origin: The Poor Man's Workstation -- Wormerveer NL (2:280/100)

.