From Jim.Pirzyk@disney.com Tue Sep 28 09:25:04 1999
Return-Path: <Jim.Pirzyk@disney.com>
Received: from mail.disney.com (mail.disney.com [204.128.192.15])
	by hub.freebsd.org (Postfix) with ESMTP id C13B215753
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 28 Sep 1999 09:24:59 -0700 (PDT)
	(envelope-from Jim.Pirzyk@disney.com)
Received: from pain10.corp.disney.com (root@pain10.corp.disney.com [153.7.110.100])
	by mail.disney.com (8.9.1/8.9.1) with SMTP id JAA07893
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 28 Sep 1999 09:24:49 -0700 (PDT)
Received: from louie.fa.disney.com by pain.corp.disney.com with ESMTP for FreeBSD-gnats-submit@freebsd.org; Tue, 28 Sep 1999 09:24:49 -0700
Received: from snowhite.faf.fa.disney.com (snowhite.faf.fa.disney.com [153.7.115.1])
	by louie.fa.disney.com (8.9.2/8.9.2) with ESMTP id JAA18564
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 28 Sep 1999 09:24:35 -0700 (PDT)
	(envelope-from pirzyk@fa.disney.com)
Received: from demons.faf.fa.disney.com (demons.faf.fa.disney.com [153.7.115.13])
	by snowhite.faf.fa.disney.com (8.9.2/8.9.2) with ESMTP id MAA10799
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 28 Sep 1999 12:24:34 -0400 (EDT)
	(envelope-from pirzyk@fa.disney.com)
Received: (from pirzyk@localhost)
	by demons.faf.fa.disney.com (8.9.3/8.9.2) id MAA33527;
	Tue, 28 Sep 1999 12:24:33 -0400 (EDT)
	(envelope-from pirzyk@fa.disney.com)
Message-Id: <199909281624.MAA33527@demons.faf.fa.disney.com>
Date: Tue, 28 Sep 1999 12:24:33 -0400 (EDT)
From: Jim.Pirzyk@disney.com
Reply-To: Jim.Pirzyk@disney.com
To: FreeBSD-gnats-submit@freebsd.org
Subject: vinum & Raid 5 errors
X-Send-Pr-Version: 3.2

>Number:         14018
>Category:       kern
>Synopsis:       vinum + Raid5 Causes system panics and hangs
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    grog
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Sep 28 09:30:01 PDT 1999
>Closed-Date:    Wed Oct 13 16:06:36 PDT 1999
>Last-Modified:  Wed Oct 13 16:10:38 PDT 1999
>Originator:     Jim Pirzyk
>Release:        FreeBSD 3.3-RELEASE i386
>Organization:
>Environment:

	Setup a Raid 5 partition in vinum with 4 9GB drives.

	The raid5 configuration looked like this:

# Vinum configuration of demons.faf.fa.disney.com, saved at Tue Sep 28 08:11:34 1999
drive a device /dev/da0h
drive b device /dev/da1h
drive c device /dev/da2h
drive d device /dev/da3h
volume garage
	plex org raid5 256k
		sd drive a len 0
		sd drive b len 0
		sd drive c len 0
		sd drive d len 0

	A core dump is available.  System has been changed to use striped
	filesystem and is stable.  The filesystem also has quotas and softupdates.
	Removed softupdates from the partition and still failed.  Removed it
	from the kernel and the system still failed.  Only removing the raid5
	partition seems to have solved the problem.

>Description:

	The system would hang or panic after about 6 hours.  The system
	panic messages would say 'page fault while in kernel mode'

Here is what gdb says about one of the dumps:
	Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x0
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0x0
stack pointer           = 0x10:0xc024ca78
frame pointer           = 0x10:0xc024cab4
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = bio 
trap number             = 12
panic: page fault

syncing disks... 

Fatal trap 12: page fault while in kernel mode
fault virtual address   = 0x30
fault code              = supervisor read, page not present
instruction pointer     = 0x8:0xc01de294
stack pointer           = 0x10:0xc024c868
frame pointer           = 0x10:0xc024c86c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = Idle
interrupt mask          = bio 
trap number             = 12
panic: page fault

dumping to dev 20001, offset 786432
dump 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 

>How-To-Repeat:

	Do high volume filesystem accesses on the Raid 5 parition.

>Fix:

	Do not use raid5 plex organization in vinum.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->grog  
Responsible-Changed-By: cpiazza 
Responsible-Changed-When: Tue Sep 28 09:56:13 PDT 1999 
Responsible-Changed-Why:  
Over to maintainer 
State-Changed-From-To: open->feedback 
State-Changed-By: grog 
State-Changed-When: Tue Sep 28 15:12:42 PDT 1999 
State-Changed-Why:  
Waiting for feedback from submitter. 
State-Changed-From-To: feedback->closed 
State-Changed-By: grog 
State-Changed-When: Wed Oct 13 16:06:36 PDT 1999 
State-Changed-Why:  
Submitter unable to reproduce the problem.  Problem 1 is almost certainly fixed, but problem 
2 is still at large. 
>Unformatted:

Greg Lehey, 29 September 1999

    This looks like the problem described at
    http://www.lemis.com/vinum/bugs.html:

	28 September 1999: We've seen some strange panics when using
	Vinum RAID-5 with soft updates. The symptoms are:

	Fatal trap 12: page fault while in kernel mode
	fault virtual address   = 0x0
	fault code              = supervisor read, page not present
	instruction pointer     = 0x8:0x0

	Note particularly the instruction pointer value. This happens
	during normal file access.

	This problem doesn't happen everywhere, but where it does,
	it's quite reliable. If you're planning to use this
	constellation, be sure to test carefully.

	Technical explanation: A buffer header gets corrupted between
	the time the top half of the driver issues the request to the
	disk driver, and when the I/O completes. Currently, the
	evidence is pointing towards the disk driver, but the
	corruption is of such an unusual nature that it's difficult to
	guess what's going on.

    Yours is the first report that confirms my suspicions that this is
    not a soft updates problem.  I'd like to see more information from
    the dump, as described in vinum(4) and
    http://www.lemis.com/vinum/how-to-debug.html.

Greg Lehey, 1 October 1999

On Thursday, 30 September 1999 at 12:16:27 -0400, Jim Pirzyk wrote:
> On Tue, 28 Sep 1999, you wrote:
>
> Just an update, I have not been able to get a clean kernel dump from the system.
> I have also seen these two other known problems
>

> 1) the system panics when 'vinum start' is executed on boot up.  What I had
> 	been doing is to rebuild a new kernel (after rm'ing /sys/compile/<NAME>)
> 	This seemed to make it work.  I did create the daXs1Y devices after
> 	reading your note.  I did have the system panic twice on startup even
> 	after the devices were created.  I do not have the vinumio.c patch
> 	installed (I was going to wait until the deadlock problem was fixed
>       too).

It looks as if you're going to have to install it anyway.  I was
puzzled about the fact that having the slice entries stopped the
problem from happening, but that was an empirical observation.

>  2) As mentioned above, the resource deadlock.  I have a kernel with the DDB
> 	and after the processes deadlock, I go into the kernel debugger and
> 	issue the 'panic' to dump the kernel.  I then get the 'page fault while in
> 	kernel mode' and the system locks up.

Yes, I've found this one too, and I think I have a fix.  I'd be
interested in trying this out before committing, because I'm not 100%
sure it will fix the problems.

> I will be on vacation until Tuesday and I will be picking this back
> up then.

Fine.

Could you please send me a stack trace of the dump you do have, using
the methods described inI'd like to see more information from the
dump, as described in vinum(4) and
http://www.lemis.com/vinum/how-to-debug.html.

From: Jim Pirzyk <Jim.Pirzyk@disney.com>
Date: Wed, 13 Oct 1999 13:22:14 -0400

An update.  I was not able to get a clean dump of the system to run
through the debugger.  When every I went down into the debugger, and
typed 'panic', it would hang on dumping the data.

I have since had to put the server into production (ie I cannot test on it), so
I just recreated the striped filesystem instead.                                  

Date: Thu, 14 Oct 1999 08:37:43 +0930 
From: Greg Lehey <grog@lemis.com>

OK.  I'll close the PR.  I think that (1) is fixed now; we've found
two different causes, and I'm relatively confident there wasn't a
third.  It's a pity you don't have time to look at the second problem,
but I'll follow that up elsewhere.
