Subj : A TCP/IP variation of -N UNZIP?
To   : Jonathan de Boyne Pollard
From : Mike Luther
Date : Sun Aug 26 2001 03:11 am

I think, Johnathan, from what I've learned so far, that a part of what you've 
offered is correct.


 JdBP> 1.    Have some utility run on the remote end that creates a file 
 JdBP> containing the set of (binary) differences between 
 JdBP> two successive versions of the database file.  Have 
 JdBP> a utility on the local end that takes such a 
 JdBP> "differences" file and a database and applies the 
 JdBP> former to the latter.  Compress just the differences 
 JdBP> file for downloading.  This is an analogue of the 
 JdBP> approach used by many large "open source" software 
 JdBP> projects, where instead of compressed snapshots of 
 JdBP> the entire source tree one can download compressed 
 JdBP> differences between one version and the next.

There appear to be two basic granularities of the project.  One relates to 
perhaps hundreds of thousands of 'things' that are common across the 
entreprise.  Even at that, what is suggested above, with modern equipment,is, I 
think feasible.

 JdBP> 2.    Increase the granularity of the compressed bundle.  Rather than 
 JdBP> having BUNDLE.ZIP containing A.TXT, B.TXT, and 
 JdBP> C.TXT, have A.ZIP, B.ZIP, and C.ZIP each containing 
 JdBP> a single file.  Download each individual archive 
 JdBP> file if its datestamp indicates that it has been 
 JdBP> updated.  This approach presumes that, as your 
 JdBP> original message implied, your database is a set of 
 JdBP> multiple individual files rather than a single 
 JdBP> humungous BUNDLE.DAT file with a complex internal 
 JdBP> structure.

And the above is exactly what is needed, simply for storage needs.

I'm thinking carefully on how it may be possible to implement the above.


The other issue I face is, I think, more complicated.

A given box generates perhaps a hundred transactions a day.  The field of 
action in the market spans, even at present, over a million and a half boxes.  
Thus the transaction processing load which has to be handled and stored on a 
daily basis is .. realistically, over a million transaction processing events a 
day.  Each of them has to be recoverable and massaged as to what the 
disposition of them was, at Empire Central, at least ONCE at the originating 
embedded boxlette end.  From that point on any one of them actually has to be 
recoverable, on demand, for 50 years or longer, in some cases we already know 
about.


Mike @ 1:117/3001

--- Maximus/2 3.01
 * Origin: Ziplog Public Port (1:117/3001)

.