Subj : Re: programming with large memory requirements
To   : comp.programming
From : David
Date : Wed Sep 28 2005 02:37 am

On Tue, 27 Sep 2005 15:33:52 UTC, "KRK" <karlk@eng.umd.edu> wrote:

> I want to run a program that continually processes > 9 GBytes of data 
> located in binary files.  Now, the program reads in the data as needed.
> 
> Is there any sort of "External RAM" that can be loaded, and treated from 
> within a program as if it is a hard drive (i.e using C statements like 
> fopen() and fread() ) using a regular 32 bit PC?
> 
> I don't care that much about increasing speed.  I don't like having to 
> continuously access the hard drive non-stop for days at a time. 

  RAM Drives and Virtual Memory can make this task easier, but they both
hold a portion of the data in memory and the rest on your normal external
storage -- the disks.  You may be able to speed your processing up and
reduce the activity on your hard drives by partitioning the problem such
that the normal RAM is used more than the drives.

  Another method is prefetching data.  If you have to repeatedly scan
9GB of data and you are only reading this in small chuncks, that is a
poor use of your resourses.  Read in much larger quantities of data
(perhaps a 1GB into RAM) and perform as much work on that data at once.

  Adding a little multithreading might also optimize the process a bit.
One or more of the threads processes data in memory while another
thread gets the next batch of data from the hard drive.

  In general, adding more RAM to your system, and making proper use of
it, will greatly increase your performance with the least cost.  A
little extra thought and a better processing method might take that
multi-day process and reduce it to an hour or less.  I guess it all
depends on what you are trying to do.

  David

.