Subj : hpt squish corruption
To   : mark lewis
From : Gerrit Kuehn
Date : Sat Jun 09 2018 21:05:20

Hello mark!

09 Jun 18 13:57, mark lewis wrote to Gerrit Kuehn:

 GK>> "a few minutes" is exactly what I would call "for ages".

 ml> wow, really??

For mails, definitely.
On other occasions we had data distributed into many files that needed to be 
processed... at one point it became completely unmanagable to keep all files in 
one directory because everything was slowing down. We solved this in a rather 
simple way and put files into subdirectories depending on the crc8 checksum of 
their name. This divided the number of files per directory by 256 (on average), 
and everything was smooth again.

 ml> the above was after a fresh boot... the file cache was relatively 
 ml> empty... a second search returned within seconds... possibly up to 30 
 ml> seconds depending on the search...

As I said: if you have 50k files, 100k files or even more, and need to process 
them all, these delays sum up every time you access one file. A simple "ls -l" 
will give you the creeps. Shell scripts break at various points due to 
"argument list too long" or similar errors. Depending on the filesystem you 
use, there may be other dragons hiding.
The situation is easy enough to avoid, so why asking for trouble?


Regards,
Gerrit

....  9:05PM  up 153 days, 22:58, 9 users, load averages: 0.17, 0.12, 0.09

--- Msged/BSD 6.1.2
 * Origin: We are a nation (2:240/12)

.