Subj : Storing 20 million randomly accessible documents in compressed form
To   : comp.programming
From : M.Barren
Date : Mon Sep 05 2005 02:28 am

Hi,

I'm currently writing a program which deals with a massive 20 million
or more very small xml documents. The program will only read (and not
write or modify) the documents. I wanted to know what method I can use
to be able to compress these documents (if uncompressed, they occupy
more than 1.7 GB) and at the same time being able to access them
randomly by a unique name/code. I need them compressed since I want to
put them on a CD (and not a DVD).

My current idea is to zip each 100 or so of them together so that I can
efficiently decompress only 100 to access one and not all 20 million.

If you have any other ideas, please let me know

cheers,
Michael

.