Subj : Committing file changes To : Coridon Henshaw From : David Noon Date : Sun Aug 20 2000 01:00 am Hi Coridon, Replying to a message of Coridon Henshaw to David Noon: CH> I'm building an open-source databasing offline Usenet news system, CH> basically along the lines of standard Fidonet message tossers and CH> readers, except designed from the ground up for Usenet news. As I CH> intend the system to be portable, I'd like to keep the number of CH> platform-specific API calls to an absolute minimum. Thats poses some difficulties. A combination of safety, performance and platform-independence is a big ask. I would tend to compromise that last one before I compromised the first two. DN>> Firstly, you should not be doing buffered I/O if your updates must be DN>> committed immediately, so you should not use fopen() and fwrite() DN>> without a setbuf() call to suppress buffer allocation. Better yet, DN>> you should consider using open() and write() instead, and use the DN>> UNIX-like unbuffered I/O routines. CH> I'm concerned that disabling buffering entirely is going to hurt CH> performance very badly as my application does lots of short (4 to 256 CH> byte) IO calls. Relying on the disk cache to handle this kind of CH> load seems a bit wasteful. Since 4-to-256 bytes does not constitute a typical Usenet article, those would not be your logical syncpoints. You should be physically writing the data to disk at your syncpoints and only at your syncpoints. DN>> Moreover, if your data resources are critically important then you DN>> should be handling any traps that occur in your program and cleaning DN>> up the critical data resources in an orderly manner. This is far and DN>> away the most professional approach to the situation. About the only DN>> things you can't handle are kernel level traps and power outages. CH> The problem I ran into was that the kernel trapped (for reasons CH> unrelated to this project) a few hours after I wrote an article into CH> the article database. Since database was still in open (I leave the CH> article reader running 24x7), the file system structures were CH> inconsistant enough that CHKDSK truncated the database well before CH> its proper end point. As you say, catching exceptions wouldn't help CH> much here. The flip side is that kernel traps are far less frequent than application traps, especially during development of the application. If your data integrity is critical you should not only be handling any exceptions that arise, but you should be rolling back to your most recent syncpoint when an error does arise. CH> My database format and engine implementations are robust enough to CH> cope with applications dying unexpectedly without finishing write CH> operations; they're not robust enough to handle boot-up CHKDSK CH> removing 80Kb of data from the end of a 100Kb file. So you do have a syncpoint architecture, then? DN>> In your situation, I would have used the second facility before DN>> considering any intermediate commits. CH> It's not intermediate commits I need: what I need is some way to flush CH> out write operations made to files which might be open for days or CH> weeks at a time. That's what an intermediate commit is. The way industrial strength database management systems work [since at least the days of IMS/360, over 30 years ago] is that an application would have defined within it points in its execution where a logical unit of work was complete and the state of the data on disk should by synchronized with the state of the data in memory; this is how the term "syncpoint" arose, and the processing between syncpoints became known as a transaction. The process of writing the changes in data to disk became known as commiting the changes. The SQL statement that performs this operation under DB2, Oracle, Sybase and other RDBMS's is COMMIT. These RDBMS's also have another statement, coded as ROLLBACK. This backs out a partially complete unit of work when an error condition has arisen. The upshot is that the content of the database on disk can be assured to conform to the data model the application is suposed to support. It does not mean that every byte of input has been captured; it means, instead, that the data structures on disk are consistent with some design. This seems to me to be the type of activity you really want to perform. One of your problems is that your input stream is not persistent, as it would be a socket connected to a NNTP server [if I read your design correctly, and assume you are coding from the ground up]. This means that you need to be able to restart a failed instance of the application, resuming from its most recent succesful syncpoint. The usual method to deal with this is to use a log or journal file that keeps track of "in flight" transactions; the journal is where your I/O remains unbuffered. If your NNTP server allows you to re-fetch articles -- and most do -- you can keep your journal in RAM or on a RAMDISK; this prevents performance hits for doing short I/O's. This design and implementation seem like a lot of work, and I suppose they are. But some old timers were doing this on machines with only 128KiB of RAM when I was in high school, so a modern PC should handle it easily. To save yourself a lot of coding, you might care to use a commercial DBMS; a copy of DB2 UDB Personal Developer Edition can be had free for the download, or on CD for the price of the medium and shipping. Start at: http://www.software.ibm.com/data/db2/ and follow the links to the download areas, or ask Indelible Blue about CD prices. Using a multi-platform commercial product will provide you with platform independence, as well as safety. It is the simplest and most robust approach unless you are prepared either to do a lot of coding or compromise on the safety of your application. Regards Dave --- FleetStreet 1.25.1 * Origin: My other computer is an IBM S/390 (2:257/609.5) .