From mjacob@feral.com  Wed Nov 29 20:10:55 2006
Return-Path: <mjacob@feral.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 8DF4116A407
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 29 Nov 2006 20:10:55 +0000 (UTC)
	(envelope-from mjacob@feral.com)
Received: from ns1.feral.com (ns1.feral.com [192.67.166.1])
	by mx1.FreeBSD.org (Postfix) with ESMTP id B7D1C43C9D
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 29 Nov 2006 20:10:51 +0000 (GMT)
	(envelope-from mjacob@feral.com)
Received: from colfax.in1.lcl (colfax.in1.lcl [172.16.1.26])
	by ns1.feral.com (8.13.8/8.13.8) with ESMTP id kATKAixJ026216
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 29 Nov 2006 12:10:54 -0800 (PST)
	(envelope-from mjacob@feral.com)
Received: from colfax.in1.lcl (localhost [127.0.0.1])
	by colfax.in1.lcl (8.13.8/8.13.8) with ESMTP id kATKAi0L001121
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 29 Nov 2006 12:10:44 -0800 (PST)
	(envelope-from mjacob@colfax.in1.lcl)
Received: (from root@localhost)
	by colfax.in1.lcl (8.13.8/8.13.8/Submit) id kATKAipM001120;
	Wed, 29 Nov 2006 12:10:44 -0800 (PST)
	(envelope-from mjacob)
Message-Id: <200611292010.kATKAipM001120@colfax.in1.lcl>
Date: Wed, 29 Nov 2006 12:10:44 -0800 (PST)
From: mjacob@freebsd.org
Reply-To: mjacob@freebsd.org
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: panic while rebooting with a dead disk
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         106030
>Category:       kern
>Synopsis:       [ufs] [panic] panic in ufs from geom when a dead disk is invalidated
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-fs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Nov 29 20:20:12 GMT 2006
>Closed-Date:    Sun Jun 19 08:38:40 UTC 2011
>Last-Modified:  Sun Jun 19 08:38:40 UTC 2011
>Originator:     Matthew Jacob
>Release:        FreeBSD 7.0-CURRENT i386
>Organization:
Feral Software
>Environment:
System: FreeBSD colfax.in1.lcl 7.0-CURRENT FreeBSD 7.0-CURRENT #33: Tue Nov 28 22:28:44 PST 2006 mjacob@colfax.in1.lcl:/home/FreeBSD/p4/newisp/i386/compile/GENERIC i386


>Description:

I had a mounted ufs disk that went away. I rebooted so as to avoid a
panic. Too bad. Geom paniced on me anyway:

Syncing disks, vnodes remaining...2 (da8:isp1:0:6:2): Invalidating pack
g_vfs_done():da8a[WRITE(offset=81920, length=4096)]error = 6
panic: bundirty: buffer 0xc6d76f70 still on queue 1
cpuid = 0
KDB: enter: panic
[thread pid 3 tid 100000 ]
Stopped at      kdb_enter+0x2b: nop
db> bt
Tracing pid 3 tid 100000 td 0xc1e98000
kdb_enter(c0936604) at kdb_enter+0x2b
panic(c093f33c,c6d76f70,1,c6d76f70,cba0ec48,...) at panic+0x127
bundirty(c6d76f70) at bundirty+0x35
brelse(c6d76f70) at brelse+0x82f
bufdone_finish(c6d76f70) at bufdone_finish+0x34c
bufdone(c6d76f70) at bufdone+0xaa
ffs_backgroundwritedone(c6d76f70) at ffs_backgroundwritedone+0xca
bufdone(c6d76f70) at bufdone+0x8f
g_vfs_done(c21475ac) at g_vfs_done+0x8a
biodone(c21475ac) at biodone+0x58
g_io_schedule_up(c1e98000) at g_io_schedule_up+0xe6
g_up_procbody(0,cba0ed38) at g_up_procbody+0x5a
fork_exit(c067d58c,0,cba0ed38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcba0ed6c, ebp = 0 ---


It's unclear to me where this should be fixed. Since device invalidation
is an inherently asynchronous process that could happen at any time, it
seems to me that GEOM should be a bit more tolerant here.

>How-To-Repeat:

Turn a disk off that has a mounted filesystem and just do a reboot.

>Fix:



>Release-Note:
>Audit-Trail:

From: Remko Lodder <remko@FreeBSD.org>
To: mjacob@freebsd.org

 > Syncing disks, vnodes remaining...2 (da8:isp1:0:6:2): Invalidating pack
 > g_vfs_done():da8a[WRITE(offset=81920, length=4096)]error = 6
 
 Well, it wants to synchronise the data in the caches to the
 disk and cannot find it.. I think a panic is the best thing
 to do to prevent any weird things happening.  What else
 should be done when the disk it once had mounted goes away?
 you have different problems already when that happends..
 
 Perhaps someone has a nice light which can be shined on this?
 
 p.s. Do not mail as root! create a normal user id for that
 with normal privileges, the super user account should only
 be used for system maintenance and updates and not for
 regular things like mailing etc, it posses an unnecessary
 risk for you and your system(s).
 
 -- 
 Kind regards,
 
       Remko Lodder               ** remko@elvandar.org
       FreeBSD                    ** remko@FreeBSD.org
 
       /* Quis custodiet ipsos custodes */

From: Robert Watson <rwatson@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/106030: panic while rebooting with a dead disk
Date: Wed, 29 Nov 2006 21:51:32 +0000 (GMT)

 > It's unclear to me where this should be fixed. Since device invalidation is 
 > an inherently asynchronous process that could happen at any time, it seems 
 > to me that GEOM should be a bit more tolerant here.
 
 That looks a lot like a UFS/buffer cache panic, not a GEOM panic?
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge

From: mjacob@freebsd.org
To: Robert Watson <rwatson@freebsd.org>
Cc: freebsd-bugs@freebsd.org
Subject: Re: kern/106030: panic while rebooting with a dead disk
Date: Wed, 29 Nov 2006 14:17:15 -0800 (PST)

 > That looks a lot like a UFS/buffer cache panic, not a GEOM panic?
 
 Good point.
 

From: Robert Watson <rwatson@FreeBSD.org>
To: mjacob@freebsd.org
Cc: Remko Lodder <remko@freebsd.org>, bug-follouwp@FreeBSD.org
Subject: Re: kern/106030: panic while rebooting with a dead disk
Date: Wed, 29 Nov 2006 22:45:53 +0000 (GMT)

 On Wed, 29 Nov 2006, mjacob@freebsd.org wrote:
 
 > A panic should be the last resort. If I/O is returned indicating the device 
 > has gone, a binval on all cached data and a forced close of the file table 
 > entry and notification of all user processes is the reasonable thing to do. 
 > Most real Unix'es that were hardened from the orginal v7 product learned to 
 > do this. FreeBSD hasn't.
 
 This is a panic on shutdown in the file system.  All user processes have 
 exited, and UFS is unable to sync cached data to disk, so there is no way to 
 report the error to a user process.
 
 > As I've repeatedly said, mostly to deaf ears in FreeBSD, a device error 
 > should never be the cause for panic *unless* there is absolutely no way to 
 > notify user processes of the error *and* data corruption may have silently 
 > occurred. Inconvenience to an existing design is not really a good argument.
 
 The context of your panic note appear to be during system shutdown during the 
 final syncing of vnode data before unmount -- is this not the case?
 
 > A read error to a device that has disappeared shouldn't cause a panic, even 
 > with a filesystem mounted. A write error to same shouldn't cause a panic - 
 > the error propagates back up the stack to the actual I/O invocation. If it 
 > was writebehind or dirty paging activity that can no longer be associated 
 > with any thread, then a panic is a policy decision that only the invoker of 
 > the I/O can make. Not the device driver. Not the volume manager (which is 
 > what GEOM is).
 
 There are certainly situations where FreeBSD panics rather than tolerating 
 invalid file system data, but I believe those problems are entirely at the 
 file system layer.  There is a kernel printf from GEOM, but the panic occurs 
 in the buffer cache code, presumably when UFS discovers life sucks more than 
 it thought.  I'd like to see UFS grow more tolerant of this sort of thing, and 
 simply lose the data rather than panicking.
 
 That said, I think the more pressing issue is actually with FAT, since 
 reliable server configurations frequently run UFS over RAID, but most FAT 
 devices are not only not reliable, but also removeable, which we currently 
 fail to tolerate at all when the FAT file system is mounted.  A practice run 
 on tolerating device removal for FAT would probably prepare us to address the 
 UFS issues more competently, as well as shake out issues in VM, etc, that 
 might arise.  For example, I believe we currently fail rather poorly when 
 paging in data from a failing swap device.  Certainly there's no good way to 
 get out of the situation, but I think we perform one of the less good bad 
 ways.
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge

From: mjacob@freebsd.org
To: Robert Watson <rwatson@freebsd.org>
Cc: bug-follouwp@freebsd.org
Subject: Re: kern/106030: panic while rebooting with a dead disk
Date: Wed, 29 Nov 2006 15:08:54 -0800 (PST)

 > This is a panic on shutdown in the file system.  All user processes have 
 > exited, and UFS is unable to sync cached data to disk, so there is no way to 
 > report the error to a user process.
 
 Yes- but it is also true that this would happen at a time other than 
 reboot. In fact, I rebooted rather than try and run with a dead disk 
 mounted and much to my annoyance I *still* couldn't avoid a panic. My 
 only other choice would have been to do a 'reboot -n'. Bad in either 
 case.
 
 >
 > There are certainly situations where FreeBSD panics rather than tolerating 
 > invalid file system data, but I believe those problems are entirely at the 
 > file system layer.  There is a kernel printf from GEOM, but the panic occurs 
 > in the buffer cache code, presumably when UFS discovers life sucks more than 
 > it thought.  I'd like to see UFS grow more tolerant of this sort of thing, 
 > and simply lose the data rather than panicking.
 
 Yes.
 
 > That said, I think the more pressing issue is actually with FAT, since 
 > reliable server configurations frequently run UFS over RAID, but most FAT 
 > devices are not only not reliable, but also removeable, which we currently 
 > fail to tolerate at all when the FAT file system is mounted.  A practice run 
 > on tolerating device removal for FAT would probably prepare us to address the 
 > UFS issues more competently, as well as shake out issues in VM, etc, that 
 > might arise.  For example, I believe we currently fail rather poorly when 
 > paging in data from a failing swap device.  Certainly there's no good way to 
 > get out of the situation, but I think we perform one of the less good bad 
 > ways.
 
 Uhh- this conversation just took a rather bizaare twist. It's not just a 
 question of making UFS more fault tolerant- UFS is sort of a dead horse 
 by now and RAID may not help when it's a channel failure (e.g., fibre 
 channel or iSCSI). I'd rather see efforts put into ZFS (and fixing the 
 XFS port to actually work)- but that is besides the point. It's more of 
 a case to make sure that we don't panic when we don't have to. Now we do 
 too much.
 
 But these are very good points- thanks for the review of my somewhat 
 botched bug report.

------------
(I'll change the summary)... Note that errno=6 is a pretty
good hint to the upper layers that the device is now gone.

I really don't see how I can make multipathing work for Ironport unless
I or somebody else fixes this.

(da7:isp1:0:6:1): isp1: watchdog timeout for handle 0x363
(da7:isp1:0:6:1): isp1: watchdog timeout for handle 0x364
(da6:isp1:0:6:0): isp1: watchdog timeout for handle 0x362
(da7:isp1:0:6:1): isp1: watchdog timeout for handle 0x365
(da6:isp1:0:6:0): isp1: watchdog timeout for handle 0x366
(da7:isp1:0:6:1): isp1: watchdog timeout for handle 0x367
g_vfs_done():(da6:isp1:0:6:0): lost device
(da7:isp1:0:6:1): lost device
(da8:isp1:0:6:2): lost device
(da8:isp1:0:6:2): removing device entry
(da7:isp1:0:6:1): Invalidating pack
stripe/billa[WRITE(offset=65536, length=2048)]error = 6
g_vfs_done():stripe/billa[WRITE(offset=6144000, length=2048)]error = 6
g_vfs_done():stripe/billa[WRITE(offset=192790528, length=16384)]error = 6
panic: bundirty: buffer 0xc6d8a4d0 still on queue 1
cpuid = 1
KDB: enter: panic
[thread pid 3 tid 100000 ]
Stopped at      kdb_enter+0x2b: nop
db> bt
Tracing pid 3 tid 100000 td 0xc1e98000
kdb_enter(c09365e4) at kdb_enter+0x2b
panic(c093f31c,c6d8a4d0,1,c6d8a4d0,cba0ec48,...) at panic+0x127
bundirty(c6d8a4d0) at bundirty+0x35
brelse(c6d8a4d0) at brelse+0x82f
bufdone_finish(c6d8a4d0) at bufdone_finish+0x34c
bufdone(c6d8a4d0) at bufdone+0xaa
ffs_backgroundwritedone(c6d8a4d0) at ffs_backgroundwritedone+0xca
bufdone(c6d8a4d0) at bufdone+0x8f
g_vfs_done(c22b9000) at g_vfs_done+0x8a
biodone(c22b9000) at biodone+0x58
g_io_schedule_up(c1e98000) at g_io_schedule_up+0xe6
g_up_procbody(0,cba0ed38) at g_up_procbody+0x5a
fork_exit(c067d570,0,cba0ed38) at fork_exit+0xac
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xcba0ed6c, ebp = 0 ---


Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 18 02:59:31 UTC 2009 
Responsible-Changed-Why:  
Analysis showed that this is either a UFS or buffer cache problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=106030 
State-Changed-From-To: open->feedback 
State-Changed-By: jh 
State-Changed-When: Sat May 14 17:10:55 UTC 2011 
State-Changed-Why:  
Can you still reproduce this on a supported release? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=106030 
State-Changed-From-To: feedback->closed 
State-Changed-By: jh 
State-Changed-When: Sun Jun 19 08:38:39 UTC 2011 
State-Changed-Why:  
Feedback timeout. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=106030 
>Unformatted:
