From mike@boobaz.net Tue Feb 27 23:50:06 2001 Received: from mxu1.u.washington.edu (mxu1.u.washington.edu [140.142.32.8]) by lists.u.washington.edu (8.9.3+UW00.05/8.9.3+UW00.12) with ESMTP id XAA80658 for ; Tue, 27 Feb 2001 23:50:02 -0800 Received: from boobaz.net (c1056043-a.sttln1.wa.home.com [24.19.193.36]) by mxu1.u.washington.edu (8.9.3+UW00.02/8.9.3+UW99.09) with ESMTP id XAA14190 for ; Tue, 27 Feb 2001 23:50:02 -0800 Received: from c1056043-a (c1056043-a [24.19.193.36]) by boobaz.net (8.11.1/8.11.1) with ESMTP id f1S7o1800635 for ; Tue, 27 Feb 2001 23:50:01 -0800 Date: Tue, 27 Feb 2001 23:50:01 -0800 (PST) From: Mike X-Sender: mike@c1056043-a.sttln1.wa.home.com To: LUG Subject: Re: Unix inode question --SUMMARY (4 answers)-- THANKS ALL (fwd) Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Excellent responses below to a question about depleting i-nodes on a filesystem. --------------------------- -=<(| mike@boobaz.net |)>=- ---------- Forwarded message ---------- Date: Tue, 27 Feb 2001 12:19:22 -0800 Subject: Re: Unix inode question --SUMMARY (4 answers)-- THANKS ALL ---------Thanks Dave D: Most Unix file systems have fixed length i-node tables. You can tell how many are left with "df -i". The risk of a corrupt block in the middle of an 8000 entry directory is as high as corruption of any other block, so backups of the files is the best way to recover if you do have a problem. More likely is high-latency of processing file changes in a directory that large, which is likely using double (and perhaps triple) indirect blocks for storing the directory itself. See the article by Wietse Venema on i-nodes and file system allocation on my home page. ---------Thanks (and Hi!) Cere: In answer to you inode question the inode limits are filesystem type - plus how you set up your filesystem - specific. Do a man mkfs or mke2fs if on a recently made linux box. In any case, the inode limit is more likely to come about on a filesystem bases rather than on a directory level. I sure there is some stucture size limit for the directory but I suspect that it is at least as large if not larger than the filesystem inode limit. Sorry I can't be specificly more helpful but you didn't give any specific OS/filesystem info... ---------Thanks Ken: I don't know about a crash, but you're risking poor performance. The unix filesystem is notoriously bad as a database package. All searches of the directory are done sequentially. If the directory you're talking about gets updated frequently you'll also suffer from locking that needs to be done to create a new entry -- all searches of the directory are suspended while the updating process scans through the directory looking for an empty slot and rewrites that disk block. Every getwd() (get working directory) call will have to do a stat() call on 50% (on average) of the directories in each parent directory as it tries to figure out where it is. Doing 4000 stat()s must take awhile (actually, you're probably spending most of your time in the newer directories so you'll be doing 8000 stat()s more often than not). Everytime you start a new [sub]shell it does a getwd() as do many other commands. Of course if you're not cd'd into a subdirectory of the monster directory then you're searching it on every file open so you lose either way. You'll probably get an ENOSPC (no space on device) type error when you try to create a new entry. Your df command should tell you how close you are to running out of inodes. On linux you can use "df -i". ---------And finally, Thanks Steven: Most typically, an overly large directory will cause performance degradation on accesses and updates to the filesytem. Modifying or searching a large directory takes more I/O and searching a large directory takes more CPU. In my experience, a more heavily utilized disk is more likely to fail than a less utilized disk, but it's not so significant when added to failures caused by excessive heat (dead fan) or power-supplies. If you are actively modifying a large directory, you increase your exposure to filesystem damage when there is a system interruption (crash). Also, if you have many processes actively attempting to update a single directory, you can also run into significant performance degredation due to filesystem locking. Typically, a Unix filesystem will not allocate additional inode space as needed. Your filesystem capabilities will vary depending on the specific Unix filesystem (E.G. bsd,jfs,ext2fs,advfs) you have. The "df -i" command will show the available free blocks and free inodes. Typically a program failure for failing to create a file (out of inodes) is handled about as well as a failure due to lack of filesystem space (out of blocks, out of disk quota). The typical Unix disk quota implimentation sets limits both on blocks and inodes. .