Subj : Committing file changes
To   : David Noon
From : Coridon Henshaw
Date : Mon Aug 14 2000 04:33 pm

On Monday August 14 2028 at 20:23, David Noon wrote to Coridon Henshaw:

 DN> How low do you want to go?

I'm building an open-source databasing offline Usenet news system, basically 
along the lines of standard Fidonet message tossers and readers, except 
designed from the ground up for Usenet news.  As I intend the system to be 
portable, I'd like to keep the number of platform-specific API calls to an 
absolute minimum.

Incidentally, if anyone would like to take a look at an alpha copy of NewsDB, 
or would like to contribute to the project, drop me a message.

 DN> Firstly, you should not be doing buffered I/O if your updates must be
 DN> committed immediately, so you should not use fopen() and fwrite() without 
a
 DN> setbuf() call to suppress buffer allocation.  Better yet, you should
 DN> consider using open() and write() instead, and use the UNIX-like
 DN> unbuffered I/O routines.

I'm concerned that disabling buffering entirely is going to hurt performance 
very badly as my application does lots of short (4 to 256 byte) IO calls.  
Relying on the disk cache to handle this kind of load seems a bit wasteful.

 DN> Moreover, if your data resources are critically important then you should
 DN> be handling any traps that occur in your program and cleaning up the
 DN> critical data resources in an orderly manner. This is far and away the 
most
 DN> professional approach to the situation. About the only things you can't
 DN> handle are kernel level traps and power outages.

The problem I ran into was that the kernel trapped (for reasons unrelated to 
this project) a few hours after I wrote an article into the article database.  
Since database was still in open (I leave the article reader running 24x7), the 
file system structures were inconsistant enough that CHKDSK truncated the 
database well before its proper end point.  As you say, catching exceptions 
wouldn't help much here.

My database format and engine implementations are robust enough to cope with 
applications dying unexpectedly without finishing write operations; they're not 
robust enough to handle boot-up CHKDSK removing 80Kb of data from the end of a 
100Kb file.

 DN> In your situation, I would have used the second facility before
 DN> considering any intermediate commits.

It's not intermediate commits I need: what I need is some way to flush out 
write operations made to files which might be open for days or weeks at a time.

--- GoldED/2 3.0.1
 * Origin: Life sucks and then you croak. (1:250/820)

.