From nobody@FreeBSD.org  Wed Mar 31 17:30:57 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id DBB001065676
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 31 Mar 2010 17:30:57 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id C9F268FC22
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 31 Mar 2010 17:30:57 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o2VHUvm3016402
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 31 Mar 2010 17:30:57 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o2VHUv9U016401;
	Wed, 31 Mar 2010 17:30:57 GMT
	(envelope-from nobody)
Message-Id: <201003311730.o2VHUv9U016401@www.freebsd.org>
Date: Wed, 31 Mar 2010 17:30:57 GMT
From: Martin Birgmeier <martin.birgmeier@aon.at>
To: freebsd-gnats-submit@FreeBSD.org
Subject: dirhash in 7.3 gratuitously frees hashes when it shouldn't
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         145246
>Category:       kern
>Synopsis:       [ufs] dirhash in 7.3 gratuitously frees hashes when it shouldn't [regression]
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 31 17:40:00 UTC 2010
>Closed-Date:    
>Last-Modified:  Thu Apr  1 15:00:18 UTC 2010
>Originator:     Martin Birgmeier
>Release:        RELENG_7_3_0_RELEASE
>Organization:
MBi at home
>Environment:
FreeBSD gandalf.xyzzy 7.3-RELEASE FreeBSD 7.3-RELEASE #0: Sun Mar 21 20:08:05 CET 2010     root@atpcdvvc.xyzzy:/usr/VOL/OBJ/FreeBSD/RELENG_7_3_0_RELEASE/src/sys/XYZZY  i386
>Description:
The new dirhash function ufsdirhash_lowmem() is called in low-memory
situations. I have a KDE repository mirror on a machine with 1.25 GiB of
memory, with 2 directories of each more than 10e6 entries.

Due to memory pressure, the hashes are now freed multiple times even
during a single subversion command, such that subversion exhibits extremely
long waiting times and is nearly unusable. This is also due to bug
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/117954 , which is still
unsolved and leads to several tens of seconds of complete freezes of the
machine whenever the dirhash is (re-) computed.
>How-To-Repeat:
Mirror the SVN repo (rsync://rsync.kde.org/svnmirror/) on a FreeBSD 7.3
machine with 1.25 GiB of memory. Note: This does not succeed any more
because the rsync server (on rsync.kde.org) times out before the local
dirhash can be created for the two directories in question.
>Fix:
Probably go back to the old dirhash behavior (7.2), or free dirhashes
only as last resort in low-mem situations. Note: I never had memory
problems in 7.2 with the old dirhash behavior, so clearly there was still
enough memory (to be gotten from somewhere else, probably).

Can I revert the dirhash changes by just reverting ufs_dirhash.c?


>Release-Note:
>Audit-Trail:

From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/145246: dirhash in 7.3 gratuitously frees hashes when it
 shouldn't
Date: Wed, 31 Mar 2010 20:18:02 +0200

 I reverted to SVN revision 195783, and it's working o.k. again (except 
 for the old issue 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/117954, of course).
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu Apr 1 01:19:41 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=145246 

From: John Baldwin <jhb@freebsd.org>
To: bug-followup@freebsd.org,
 martin.birgmeier@aon.at
Cc:  
Subject: Re: kern/145246: [ufs] dirhash in 7.3 gratuitously frees hashes when it shouldn't [regression]
Date: Thu, 1 Apr 2010 08:56:03 -0400

 I suspect the fix here is to provide a tunable/sysctl that can be used to 
 disable the dirhash lowmem handler for workloads where it does more harm than 
 good.
 
 -- 
 John Baldwin

From: Martin Birgmeier <Martin.Birgmeier@aon.at>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/145246: [ufs] dirhash in 7.3 gratuitously frees hashes when
 it shouldn't [regression]
Date: Thu, 01 Apr 2010 16:54:13 +0200

 I believe that there should be some notion of "value" given to resources 
 which may potentially be destroyed and recreated - computing the dirhash 
 takes a lot of time (see 
 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/117954 ), and this work 
 should not be thrown away lightly. (This is why dirhash exists in the 
 first place.)
 
 Maybe the lowmem handler could be modified to only use the dirhash 
 lowmem handler as a last resort. In my case, the kernel did not really 
 seem to be that short on memory, as it has now been running with 
 moderate to high load for 24 hours with the reversion to SVN rev 195783 
 for ufs_dirhash.c and dirhash.h.
 
 I have configured vfs.ufs.dirhash_maxmem=33554432 (32M) by the way. 
 Compared to 1.25G, this is only 2.5%, so I believe not really something 
 that's eating lots of memory.
 
 Regards,
 
 Martin
>Unformatted:
