From lefcha@hellug.gr  Wed Sep 10 01:27:57 2003
Return-Path: <lefcha@hellug.gr>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 2F71116A4BF
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 10 Sep 2003 01:27:57 -0700 (PDT)
Received: from forthnet.gr (athmta03.forthnet.gr [193.92.150.22])
	by mx1.FreeBSD.org (Postfix) with ESMTP id B239F43FBF
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 10 Sep 2003 01:27:54 -0700 (PDT)
	(envelope-from lefcha@hellug.gr)
Received: from localhost.my.domain (ppp2-isdn-50.the.forthnet.gr [213.16.247.50])
	by forthnet.gr (8.12.2/8.12.2) with ESMTP id h8A8RqBY007697
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 10 Sep 2003 11:27:52 +0300
Received: from localhost.my.domain (localhost [127.0.0.1])
	by localhost.my.domain (8.12.9/8.12.9) with ESMTP id h8A8Rq7H000702
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 10 Sep 2003 11:27:52 +0300 (EEST)
	(envelope-from lefcha@localhost.my.domain)
Received: (from lefcha@localhost)
	by localhost.my.domain (8.12.9/8.12.9/Submit) id h8A8RqlA000701;
	Wed, 10 Sep 2003 11:27:52 +0300 (EEST)
	(envelope-from lefcha)
Message-Id: <200309100827.h8A8RqlA000701@localhost.my.domain>
Date: Wed, 10 Sep 2003 11:27:52 +0300 (EEST)
From: Lefteris Chatzibarbas <lefcha@hellug.gr>
Reply-To: Lefteris Chatzibarbas <lefcha@hellug.gr>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Syncer "giving up" on buffers and ext2 filesystems
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         56675
>Category:       kern
>Synopsis:       Syncer "giving up" on buffers and ext2 filesystems [still a problem in 5.3RC1]
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    truckman
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Sep 10 01:30:09 PDT 2003
>Closed-Date:    Fri Sep 16 03:04:09 GMT 2005
>Last-Modified:  Fri Sep 16 03:04:09 GMT 2005
>Originator:     Lefteris Chatzibarbas
>Release:        FreeBSD 5.1-CURRENT i386
>Organization:
>Environment:
System: FreeBSD bsd 5.1-CURRENT FreeBSD 5.1-CURRENT #0: Mon Sep 8 21:39:20 EEST 2003 root@bsd:/usr/obj/usr/src/sys/CUSTOM i386

>Description:
Syncer is "giving up" on buffers during shutdown if ext2fs partitions are
mounted.  During the next boot all filesystems are checked because of improper
dismount.  Here follows the exact message during shutdown:

  Waiting (max 60 seconds) for system process `vnlru' to stop...stopped
  Waiting (max 60 seconds) for system process `bufdaemon' to stop...stopped
  Waiting (max 60 seconds) for system process `syncer' to stop...stopped
 
  syncing disks, buffers remaining... 8 8 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6
  giving up on 6 buffers
  Uptime: 41m20s
  pfs_vncache_unload(): 1 entries remaining
  Shutting down ACPI
  Rebooting...

The related thread on freebsd-current@ mailing list:

  http://lists.freebsd.org/pipermail/freebsd-current/2003-September/009731.html

>How-To-Repeat:
One can repeat the problem by having some ext2fs partitions mounted when
shutting down.

>Fix:

>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->closed 
State-Changed-By: schweikh 
State-Changed-When: Tue Feb 17 12:38:32 PST 2004 
State-Changed-Why:  
This was fixed a while ago. Please try with a recent -current. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=56675 
State-Changed-From-To: closed->open 
State-Changed-By: schweikh 
State-Changed-When: Tue Feb 17 23:45:08 PST 2004 
State-Changed-Why:  
Apparently I confused this with some other fix. bde says 
it is still an issue. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=56675 

Adding to audit trail from email in freebsd-bugs:

Date: Fri, 29 Oct 2004 13:21:19 +0100
From: Stefek Zaba <stefek.zaba@hp.com>

Head above parapet: I was bitten by outstanding bug kern/56675 during a 
trial install of 5.3RC1 - I'd mounted an ext3 partition (using ext2fs, 
since ext3 is "backwards compatible", loosely speaking), called shutdown 
to exit "cleanly" without a preceding umount of the ext3 partition, and 
had the "syncing disks, buffers remaning... n n n n n n n
giving up on n buffers" message as described in 56675.

56675 has been open since 10sep03 (filed against 5.1CURRENT), with a dup 
recorded and closed (as a dup) as 63398 against 5.2CURRENT on 27feb04, 
and another dup as kern/70834 as recently as 23aug04. The open one rates 
the Severity as Serious, and Priority as Medium. For a FreeBSD 
production install, that may even overstate the case, as there you'd be 
running all your mounted FSes as UFS, I presume. But for the 
multibooting dabblers with Penguin-flavoured data, it's kinda offputting 
to have their early encounters with FreeBSD marred by a dirty shutdown; 
and the text of these bugs allege that not only the e2fs data, but 
"central" UFS partitions, can also be corrupted by this. So in 
situations where ext2 partitions are in use, it's at least arguable that 
  the Severity is higher than Serious... and a fix to allow smoother 
coexistence with the ext2/ext3 world would be a welcome encouragement to 
a "low-risk" Just Try The BSD Way ;-)

I've no idea, though, whether this is a relatively quick fix that "just" 
needs an existing fs guru to spend a couple of hours of their (already 
overcommitted) time on, or whether there are dragons in the ext2fs code 
or its interactions with the rest of vnode handling which make this a 
well-known maintenance nightmare. Apologies in advance if I'm scratching 
at an open wound... And if it is a pain to fix, might there be any 
partial relief available by making shutdown umount non-UFS partitions 
"earlier"?

Stefek

From: Martin Kammerhofer <Martin.Kammerhofer@kfunigraz.ac.at>
To: freebsd-gnats-submit@FreeBSD.org, lefcha@hellug.gr
Cc:  
Subject: Re: kern/56675: Syncer "giving up" on buffers and ext2 filesystems 
 [still a problem in 5.3RC1]
Date: Tue, 16 Nov 2004 17:34:10 +0100

 This is still an issue with 5-current. I have no patch for the kernel
 bug,  but there is
 a quick userspace workaround. Since unmounting ext2 before shutdown
 works
 I inserted the following script into /etc/rc.d for auto-unmount during
 shutdown.
 This bug could bite quite a few,  it should be documented in the 5.3
 ERRATA!
 
 #! /bin/sh
 #
 # $Id: ext2fs,v 1.1 2004/11/15 11:02:56 martin Exp $
 #
 # Unmount ext2fs upon shutdown to work around a kernel bug.
 
 # PROVIDE: ext2fs
 # REQUIRE: mountcritlocal
 # BEFORE: mountcritremote
 # KEYWORD: nojail shutdown
 
 ext2fs_enable="YES"
 ext2fs_flags="-v"    # optional (u)mount verbosity
 
 . /etc/rc.subr
 
 name="ext2fs"
 rcvar=`set_rcvar`
 start_cmd="/sbin/mount -a -t ext2fs $ext2fs_flags"
 stop_cmd="/sbin/umount -A -t ext2fs -f $ext2fs_flags"
 
 load_rc_config $name
 run_rc_command "$1"
 
 #EOF#
 
 

From: Michael Nottebrock <michaelnottebrock@gmx.net>
To: freebsd-gnats-submit@freebsd.org, lefcha@hellug.gr
Cc:  
Subject: Re: kern/56675: Syncer "giving up" on buffers and ext2 filesystems [still a problem in 5.3RC1]
Date: Sat, 12 Mar 2005 09:22:23 +0100

 Here's a better workaround which I've been using for a long time now (it does 
 not use umount -t and thus does not need mounted ext2 filesystems to be 
 listed in /etc/fstab in order to work).
 
 -- 
    ,_,   | Michael Nottebrock               | lofi@freebsd.org
  (/^ ^\) | FreeBSD - The Power to Serve     | http://www.freebsd.org
    \u/   | K Desktop Environment on FreeBSD | http://freebsd.kde.org
 
 
 --- rc.shutdown	Sun Jun  6 13:46:27 2004
 +++ rc.shutdown	Thu Jun 17 18:54:00 2004
 @@ -97,6 +97,13 @@
  
  # Insert other shutdown procedures here
  
 +extfs=`eval mount | grep ext2fs | awk '{print $1 }'`
 +
 +for _elem in $extfs; do
 +	echo -n "Unmounting ext2/ext3 filesystems: "
 +	umount -f $_elem
 +	echo -n "$_elem "
 +done
  
  echo '.'
  exit 0
 
State-Changed-From-To: open->feedback 
State-Changed-By: truckman 
State-Changed-When: Wed Sep 7 00:56:49 GMT 2005 
State-Changed-Why:  
The following patch was sent to the PR submitter for testing: 

Index: sys/kern/kern_shutdown.c 
=================================================================== 
RCS file: /home/ncvs/src/sys/kern/kern_shutdown.c,v 
retrieving revision 1.163.2.4 
diff -u -r1.163.2.4 kern_shutdown.c 
--- sys/kern/kern_shutdown.c	7 Jan 2005 20:09:08 -0000	1.163.2.4 
+++ sys/kern/kern_shutdown.c	6 Sep 2005 03:28:00 -0000 
@@ -249,6 +249,16 @@ 
dumpsys(&dumper); 
} 

+static int 
+isbufbusy(struct buf *bp) 
+{ 
+	if (((bp->b_flags & (B_INVAL | B_PERSISTENT)) == 0 && 
+	    BUF_REFCNT(bp) > 0) || 
+	    ((bp->b_flags & (B_DELWRI | B_INVAL)) == B_DELWRI)) 
+		return (1); 
+	return (0); 
+} 
+ 
/* 
* Shutdown the system cleanly to prepare for reboot, halt, or power off. 
*/ 
@@ -304,16 +314,9 @@ 
*/ 
for (iter = pbusy = 0; iter < 20; iter++) { 
nbusy = 0; 
-			for (bp = &buf[nbuf]; --bp >= buf; ) { 
-				if ((bp->b_flags & B_INVAL) == 0 && 
-				    BUF_REFCNT(bp) > 0) { 
+			for (bp = &buf[nbuf]; --bp >= buf; ) 
+				if (isbufbusy(bp)) 
nbusy++; 
-				} else if ((bp->b_flags & (B_DELWRI | B_INVAL)) 
-						== B_DELWRI) { 
-					/* bawrite(bp);*/ 
-					nbusy++; 
-				} 
-			} 
if (nbusy == 0) { 
if (first_buf_printf) 
printf("No buffers busy after final sync"); 
@@ -359,8 +362,7 @@ 
*/ 
nbusy = 0; 
for (bp = &buf[nbuf]; --bp >= buf; ) { 
-			if (((bp->b_flags&B_INVAL) == 0 && BUF_REFCNT(bp)) || 
-			    ((bp->b_flags & (B_DELWRI|B_INVAL)) == B_DELWRI)) { 
+			if (isbufbusy(bp)) { 
if (bp->b_dev == NULL) { 
TAILQ_REMOVE(&mountlist, 
bp->b_vp->v_mount, mnt_list); 
Index: sys/kern/vfs_bio.c 
=================================================================== 
RCS file: /home/ncvs/src/sys/kern/vfs_bio.c,v 
retrieving revision 1.444.2.3 
diff -u -r1.444.2.3 vfs_bio.c 
--- sys/kern/vfs_bio.c	26 Jun 2005 23:55:50 -0000	1.444.2.3 
+++ sys/kern/vfs_bio.c	6 Sep 2005 03:24:06 -0000 
@@ -1466,7 +1466,8 @@ 
if (bp->b_bufsize || bp->b_kvasize) 
bufspacewakeup(); 

-	bp->b_flags &= ~(B_ASYNC | B_NOCACHE | B_AGE | B_RELBUF | B_DIRECT); 
+	bp->b_flags &= ~(B_ASYNC | B_NOCACHE | B_AGE | B_RELBUF | B_DIRECT | 
+	    B_PERSISTENT); 
if ((bp->b_flags & B_DELWRI) == 0 && (bp->b_xflags & BX_VNDIRTY)) 
panic("brelse: not dirty"); 
/* unlock */ 
Index: sys/sys/buf.h 
=================================================================== 
RCS file: /home/ncvs/src/sys/sys/buf.h,v 
retrieving revision 1.167.2.3 
diff -u -r1.167.2.3 buf.h 
--- sys/sys/buf.h	16 Aug 2005 15:09:44 -0000	1.167.2.3 
+++ sys/sys/buf.h	6 Sep 2005 03:34:06 -0000 
@@ -205,7 +205,7 @@ 
#define	B_CACHE		0x00000020	/* Bread found us in the cache. */ 
#define	B_VALIDSUSPWRT	0x00000040	/* Valid write during suspension. */ 
#define	B_DELWRI	0x00000080	/* Delay I/O until buffer reused. */ 
-#define	B_00000100	0x00000100	/* Available flag. */ 
+#define	B_PERSISTENT	0x00000100	/* Perm. ref'ed while fs mounted. */ 
#define	B_DONE		0x00000200	/* I/O completed. */ 
#define	B_EINTR		0x00000400	/* I/O was interrupted */ 
#define	B_00000800	0x00000800	/* Available flag. */ 
@@ -232,8 +232,8 @@ 

#define PRINT_BUF_FLAGS "2040b3137cluster36vmio35ram34b27"  
"33paging32b2531writeinprog30b2327relbuf26dirty25b20"  
-	"24b1923phys22clusterok21malloc20nocache17locked16inval"  
-	"15scanned14nowdrain13eintr12done11b810delwri7validsuspwrt"  
+	"24b1923b1822clusterok21malloc20nocache17b1416inval"  
+	"15b1214b1113eintr12done11persistent10delwri7validsuspwrt"  
"6cache5deferred4direct3async2needcommit1age" 

/* 
Index: sys/gnu/ext2fs/fs.h 
=================================================================== 
RCS file: /home/ncvs/src/sys/gnu/ext2fs/Attic/fs.h,v 
retrieving revision 1.16.2.1 
diff -u -r1.16.2.1 fs.h 
--- sys/gnu/ext2fs/fs.h	31 Jan 2005 23:26:02 -0000	1.16.2.1 
+++ sys/gnu/ext2fs/fs.h	6 Sep 2005 03:25:01 -0000 
@@ -150,11 +150,16 @@ 

/* 
* Historically, ext2fs kept it's metadata buffers on the LOCKED queue.  Now, 
- * we simply change the lock owner to kern so that it may be released from 
- * another context.  Later, we release the buffer, and conditionally write it 
- * when we're done. 
+ * we simply change the lock owner to kern so that we may use it from contexts 
+ * other than the one that originally locked it.  When we are finished with 
+ * the buffer, we release it, writing it first if it was dirty.  The 
+ * B_PERSISTENT flag is cleared by brelse(), which bwrite() calls after the 
+ * buffer is written in the B_DIRTY case. 
*/ 
-#define LCK_BUF(bp)	BUF_KERNPROC(bp); 
+#define LCK_BUF(bp) {  
+	(bp)->b_flags |= B_PERSISTENT;  
+	BUF_KERNPROC(bp);  
+} 

#define ULCK_BUF(bp) {  
long flags;  


Responsible-Changed-From-To: freebsd-bugs->truckman 
Responsible-Changed-By: truckman 
Responsible-Changed-When: Wed Sep 7 00:56:49 GMT 2005 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=56675 
State-Changed-From-To: feedback->patched 
State-Changed-By: truckman 
State-Changed-When: Thu Sep 8 06:31:29 GMT 2005 
State-Changed-Why:  
Patched in HEAD: 
1.18      +8 -8      src/sys/gnu/fs/ext2fs/fs.h 
1.175     +13 -11    src/sys/kern/kern_shutdown.c 
1.189     +4 -4      src/sys/sys/buf.h 

Will MFC to RELENG_5 in a few days. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=56675 
State-Changed-From-To: patched->closed 
State-Changed-By: truckman 
State-Changed-When: Fri Sep 16 03:02:08 GMT 2005 
State-Changed-Why:  
Closed in RELENG_5 by this commit: 

From: Don Lewis <truckman@FreeBSD.org> 
Subject: cvs commit: src/sys/gnu/ext2fs fs.h src/sys/kern kern_shutdown.c         src/sys/sys buf.h 
Date: Mon, 12 Sep 2005 04:25:53 +0000 (UTC) 
To: src-committers@FreeBSD.org, 
cvs-src@FreeBSD.org, 
cvs-all@FreeBSD.org 

truckman    2005-09-12 04:25:53 UTC 

FreeBSD src repository 

Modified files:        (Branch: RELENG_5) 
sys/gnu/ext2fs       fs.h  
sys/kern             kern_shutdown.c  
sys/sys              buf.h  
Log: 
MFC     src/sys/gnu/fs/ext2fs/fs.h      1.18 
src/sys/kern/kern_shutdown.c    1.175 
src/sys/sys/buf.h               1.189 

FreeBSD src repository 

Modified files: 
sys/gnu/fs/ext2fs    fs.h 
sys/kern             kern_shutdown.c 
sys/sys              buf.h 
Log: 
Add a new struct buf flag bit, B_PERSISTENT, and use it to tag 
struct bufs that are persistently held by ext2fs.  Ignore any buffers 
with this flag in the code in boot() that counts "busy" and dirty 
buffers and attempts to sync the dirty buffers, which is done before 
attempting to unmount all the file systems during shutdown. 

This fixes the problem caused by any ext2fs file systems that are 
mounted at system shutdown time, which caused boot() to give up on 
a non-zero number of buffers and skip the call to vfs_unmountall(). 
This left all the mounted file systems in a dirty state and caused 
them to all require cleanup by fsck on reboot. 

Move the two separate copies of the "busy" buffer test in boot() 
to a separate function. 

Nuke the useless spl() stuff in the ext2fs ULCK_BUF() macro. 

Bring the PRINT_BUF_FLAGS definition in sys/buf.h up to date with 
this and previous flag changes. 

PR:             kern/56675, kern/85163 
Tested by:      "Matthias Andree" matthias.andree at gmx.de 
Reviewed by:    bde 
MFC after:      3 days 

Revision  Changes    Path 
1.18      +8 -8      src/sys/gnu/fs/ext2fs/fs.h 
1.175     +13 -11    src/sys/kern/kern_shutdown.c 
1.189     +4 -4      src/sys/sys/buf.h 

PR:             kern/56675, kern/85163 
Tested by:      "Matthias Andree" matthias.andree at gmx.de 
Reviewed by:    bde 

Revision   Changes    Path 
1.16.2.2   +8 -8      src/sys/gnu/ext2fs/fs.h 
1.163.2.5  +13 -11    src/sys/kern/kern_shutdown.c 
1.167.2.4  +3 -3      src/sys/sys/buf.h 


http://www.freebsd.org/cgi/query-pr.cgi?pr=56675 
>Unformatted:
