Subj : Re: Storing 20 million randomly accessible documents in compressed
To   : comp.programming
From : Thad Smith
Date : Mon Sep 05 2005 08:10 pm

[Jongware] wrote:
>>I'm currently writing a program which deals with a massive 20 million
>>or more very small xml documents. The program will only read (and not
>>write or modify) the documents. I wanted to know what method I can use
>>to be able to compress these documents (if uncompressed, they occupy
>>more than 1.7 GB) and at the same time being able to access them
>>randomly by a unique name/code. I need them compressed since I want to
>>put them on a CD (and not a DVD).
> 
> 1.7 Gb/20 million files amounts to about 85 bytes per file...
> think about a general zip library. 

For 85 bytes per file, it may make sense to generate a static dictionary 
for all files.  I don't know what existing libraries would help with that.

Thad

.