Subj : Re: programming with large memory requirements
To   : comp.programming
From : Randy Howard
Date : Wed Sep 28 2005 02:43 am

KRK wrote
(in article <dhboou$b6d$2@grapevine.wam.umd.edu>):

> I want to run a program that continually processes > 9 GBytes of data 
> located in binary files.  Now, the program reads in the data as needed.

If you only read the data in as needed, then the total amount 
isn't an issue, provided you free up that you're no longer 
using.  

> Is there any sort of "External RAM" that can be loaded, and treated from 
> within a program as if it is a hard drive (i.e using C statements like 
> fopen() and fread() ) using a regular 32 bit PC?

The file system has quite sophisticated buffer cache already, 
trying to reinvent that wheel (on top of the existing one) is 
unlikely to be of much benefit.

> I don't care that much about increasing speed.  I don't like having to 
> continuously access the hard drive non-stop for days at a time. 
 
Unless you can reorder your disk operations to benefit the 
cache?  For example, if you are reading and writing data from 
random locations in the binary files, can you do it sequentially 
instead? 

Try describing the file operations you are performing (nevermind 
the calculations on the data).  If the drive is churning 
constantly, then your application is I/O bound, and you need 
faster drives (or perhaps a RAID 0 working set across multiple 
drives). 



-- 
Randy Howard (2reply remove FOOBAR)

.