From nobody@FreeBSD.org  Mon Jun  2 05:48:16 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D193D106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  2 Jun 2008 05:48:16 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id AEE9B8FC2E
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  2 Jun 2008 05:48:16 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m525kSr0043503
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 2 Jun 2008 05:46:28 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m525kS5C043502;
	Mon, 2 Jun 2008 05:46:28 GMT
	(envelope-from nobody)
Message-Id: <200806020546.m525kS5C043502@www.freebsd.org>
Date: Mon, 2 Jun 2008 05:46:28 GMT
From: kevin <kevinxlinuz@163.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: kernel panic on mutex sleepq chain
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         124200
>Category:       amd64
>Synopsis:       [panic] kernel panic on mutex sleepq chain
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    jhb
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jun 02 05:50:05 UTC 2008
>Closed-Date:    Mon Sep 29 15:45:54 UTC 2008
>Last-Modified:  Mon Sep 29 15:45:54 UTC 2008
>Originator:     kevin
>Release:        8.0 current
>Organization:
163.com
>Environment:
FreeBSD xxx-laptop.people.163.org 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sun Jun  1 13:23:39 CST 2008     root@xxx-laptop.people.163.org:/usr/obj/usr/src/sys/G8laptop  amd64

>Description:
On my thinkpad t61p,when I running pkg_delete -a
panic: mutex sleepq chain not owned at /usr/src/sys/kern/subr_sleepqueue.c:645
cpuid =1
KDB: enter: panic
db>bt
[thread pid] 125 tid 100113 td 0xffffff0001d61000
kdb_enter() at kdb_enter+0x3d
panic() at panic+0x16c
assert_mtx() at assert_mtx
sleepq_resume_thread() at sleepq_resume_thread+0x96
sleepq_broadcast() at sleepq_broadcast+0x79
cv_broadcastpri() at cv_broadcastpri+0x3f
zio_done() at zio_done+0x1ac
zio_done() ar zio_done+0x151
zio_vdev_io_assess() at zio_vdev_io_assess+0xdd
taskq_thread() at taskq_thread+0x17d
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0,rip =0, rsp = 0xffffffffd76b4d30, rdp =0 ---

dmesg:
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #0: Sun Jun  1 13:23:39 CST 2008
    root@datastream-laptop.people.163.org:/usr/obj/usr/src/sys/G8laptop
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz (2394.02-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 2
usable memory = 2117898240 (2019 MB)
avail memory  = 2047000576 (1952 MB)
ACPI APIC Table: <LENOVO TP-7L   >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
This module (opensolaris) contains code covered by the
Common Development and Distribution License (CDDL)
see http://opensolaris.org/os/licensing/opensolaris_license/
ACPI Warning (tbfadt-0505): Optional field "Gpe1Block" has zero address or length:        0    102C/0 [20070320]
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <LENOVO TP-7L> on motherboard
acpi0: [ITHREAD]
acpi_ec0: <Embedded Controller: GPE 0x12, ECDT> port 0x62,0x66 on acpi0
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 7ef00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0xd2000000-0xd2ffffff,0xe0000000-0xefffffff,0xd0000000-0xd1ffffff irq 16 at device 0.0 on pci1
em0: <Intel(R) PRO/1000 Network Connection 6.9.5> port 0x1840-0x185f mem 0xfe200000-0xfe21ffff,0xfe225000-0xfe225fff irq 20 at device 25.0 on pci0
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:1c:25:1c:fb:d0
uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0x1860-0x187f irq 20 at device 26.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0x1880-0x189f irq 21 at device 26.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem 0xfe226c00-0xfe226fff irq 22 at device 26.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb2: EHCI version 1.0
usb2: companion controllers, 2 ports each: usb0 usb1
usb2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0
usb2: USB revision 2.0
uhub2: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2
uhub2: 4 ports with 4 removable, self powered
pcm0: <Intel 82801H High Definition Audio Controller> mem 0xfe220000-0xfe223fff irq 17 at device 27.0 on pci0
pcm0: [ITHREAD]
pcib2: <ACPI PCI-PCI bridge> irq 20 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pci2: <memory> at device 0.0 (no driver attached)
pcib3: <ACPI PCI-PCI bridge> irq 21 at device 28.1 on pci0
pci3: <ACPI PCI bus> on pcib3
iwn0: <Intel(R) PRO/Wireless 4965BGN> mem 0xd7dfe000-0xd7dfffff irq 17 at device 0.0 on pci3
iwn0: Reg Domain: MoW1, address 00:1d:e0:48:13:2f
iwn0: [ITHREAD]
iwn0: 11a rates: 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
iwn0: 11b rates: 1Mbps 2Mbps 5.5Mbps 11Mbps
iwn0: 11g rates: 1Mbps 2Mbps 5.5Mbps 11Mbps 6Mbps 9Mbps 12Mbps 18Mbps 24Mbps 36Mbps 48Mbps 54Mbps
iwn0: 11na MCS: 15Mbps 30Mbps 45Mbps 60Mbps 90Mbps 120Mbps 135Mbps 150Mbps 30Mbps 60Mbps 90Mbps 120Mbps 180Mbps 240Mbps 270Mbps 300Mbps
iwn0: 11ng MCS: 15Mbps 30Mbps 45Mbps 60Mbps 90Mbps 120Mbps 135Mbps 150Mbps 30Mbps 60Mbps 90Mbps 120Mbps 180Mbps 240Mbps 270Mbps 300Mbps
pcib4: <ACPI PCI-PCI bridge> irq 22 at device 28.2 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 23 at device 28.3 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 20 at device 28.4 on pci0
pci13: <ACPI PCI bus> on pcib6
uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0x18a0-0x18bf irq 16 at device 29.0 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0x18c0-0x18df irq 17 at device 29.1 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3
usb4: USB revision 1.0
uhub4: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb4
uhub4: 2 ports with 2 removable, self powered
uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0x18e0-0x18ff irq 18 at device 29.2 on pci0
uhci4: [GIANT-LOCKED]
uhci4: [ITHREAD]
usb5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4
usb5: USB revision 1.0
uhub5: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb5
uhub5: 2 ports with 2 removable, self powered
ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfe227000-0xfe2273ff irq 19 at device 29.7 on pci0
ehci1: [GIANT-LOCKED]
ehci1: [ITHREAD]
usb6: EHCI version 1.0
usb6: companion controllers, 2 ports each: usb3 usb4 usb5
usb6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1
usb6: USB revision 2.0
uhub6: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb6
uhub6: 6 ports with 6 removable, self powered
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci21: <ACPI PCI bus> on pcib7
cbb0: <RF5C476 PCI-CardBus Bridge> mem 0xf8100000-0xf8100fff irq 16 at device 0.0 on pci21
cardbus0: <CardBus bus> on cbb0
pccard0: <16-bit PCCard bus> on cbb0
cbb0: [ITHREAD]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ATA controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1830-0x183f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel AHCI controller> port 0x1c48-0x1c4f,0x1c1c-0x1c1f,0x1c40-0x1c47,0x1c18-0x1c1b,0x1c20-0x1c3f mem 0xfe226000-0xfe2267ff irq 16 at device 31.2 on pci0
atapci1: [ITHREAD]
atapci1: AHCI Version 01.10 controller with 3 ports PM not supported
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: port not implemented
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci1
ata4: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_button0: <Sleep Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD]
psm0: model Synaptics Touchpad, device ID 0
battery0: <ACPI Control Method Battery> on acpi0
acpi_acad0: <AC Adapter> on acpi0
acpi_ibm0: <IBM ThinkPad ACPI Extras> on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcefff,0xcf000-0xcffff,0xd0000-0xd0fff,0xd1000-0xd3fff,0xe0000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ubt0: <Broadcom Corp BCM2045B, class 224/1, rev 2.00/1.00, addr 2> on uhub0
ubt0: Interface 0 endpoints: interrupt=0x81, bulk-in=0x82, bulk-out=0x2
ubt0: Interface 1 (alt.config 4) endpoints: isoc-in=0x83, isoc-out=0x3; wMaxPacketSize=64; nframes=5, buffer size=320
ugen0: <STMicroelectronics Biometric Coprocessor, class 0/0, rev 1.00/0.01, addr 3> on uhub0
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounters tick every 1.000 msec
ZFS filesystem version 6
ZFS storage pool version 6
acd0: DVDR <Optiarc DVD RW AD-7910A/1.D1> at ata0-master UDMA33
ad4: 152627MB <FUJITSU MHW2160BH PL 0084001E> at ata2-master SATA150
pcm0: <HDA Codec: Analog Devices AD1984>
pcm0: <HDA Driver Revision: 20080420_0052>
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
GEOM_LABEL: Label for provider ad4s1 is ntfs/System.
GEOM_LABEL: Label for provider ad4s5 is ntfs/Program.
GEOM_LABEL: Label for provider ad4s6 is ntfs/ Data.
lock order reversal: (sleepable after non-sleepable)
 1st 0xffffff0001b66028 struct mount mtx (struct mount mtx) @ /usr/src/sys/kern/vfs_subr.c:343
 2nd 0xffffff0001b66000 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:370
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x5f3
__lockmgr_args() at __lockmgr_args+0xc5e
vfs_busy() at vfs_busy+0xd0
vfs_mount_alloc() at vfs_mount_alloc+0x80
vfs_mountroot() at vfs_mountroot+0x23b
start_init() at start_init+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffff80c35d30, rbp = 0 ---
lock order reversal: (sleepable after non-sleepable)
 1st 0xffffff0001a9ada8 vnode interlock (vnode interlock) @ /usr/src/sys/fs/devfs/devfs_vnops.c:288
 2nd 0xffffff0001a9ad80 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2044
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x5f3
__lockmgr_args() at __lockmgr_args+0x527
vop_stdlock() at vop_stdlock+0x39
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
_vn_lock() at _vn_lock+0x50
vget() at vget+0x70
devfs_allocv() at devfs_allocv+0xf9
devfs_root() at devfs_root+0x49
set_rootvnode() at set_rootvnode+0x2d
vfs_mountroot() at vfs_mountroot+0x2f8
start_init() at start_init+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffff80c35d30, rbp = 0 ---
lock order reversal:
 1st 0xffffff0001a9ad80 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2044
 2nd 0xffffff0001b619a0 devfsmount (devfsmount) @ /usr/src/sys/fs/devfs/devfs_vnops.c:292
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x5f3
_sx_xlock() at _sx_xlock+0x41
devfs_allocv() at devfs_allocv+0x111
devfs_root() at devfs_root+0x49
set_rootvnode() at set_rootvnode+0x2d
vfs_mountroot() at vfs_mountroot+0x2f8
start_init() at start_init+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffff80c35d30, rbp = 0 ---
Trying to mount root from ufs:/dev/ad4s3a
lock order reversal: (sleepable after non-sleepable)
 1st 0xffffff0001a9a878 bufobj interlock (bufobj interlock) @ /usr/src/sys/kern/vfs_bio.c:2442
 2nd 0xffffffffc0fa2de8 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2456
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x5f3
__lockmgr_args() at __lockmgr_args+0x527
getblk() at getblk+0xda
breadn() at breadn+0x34
bread() at bread+0x1e
ffs_blkatoff() at ffs_blkatoff+0x56
ufs_lookup() at ufs_lookup+0x5ab
vfs_cache_lookup() at vfs_cache_lookup+0xe5
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x97
lookup() at lookup+0x4ec
namei() at namei+0x453
kern_unlinkat() at kern_unlinkat+0x9d
vfs_mountroot_try() at vfs_mountroot_try+0x3fa
vfs_mountroot() at vfs_mountroot+0x3e5
start_init() at start_init+0x62
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffff80c35d30, rbp = 0 ---
WARNING: attempt to net_add_domain(bluetooth) after domainfinalize()
WARNING: attempt to net_add_domain(netgraph) after domainfinalize()
em0: link state changed to UP
Accounting enabled

>How-To-Repeat:
update to current kernel.I don't sure when it will panic.it looks like random.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-amd64->benjsc 
Responsible-Changed-By: benjsc 
Responsible-Changed-When: Tue Jun 10 02:15:53 UTC 2008 
Responsible-Changed-Why:  
Problem currently under investigation by benjsc,attilo,kris, stay tuned! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=124200 
State-Changed-From-To: open->patched 
State-Changed-By: jhb 
State-Changed-When: Mon Sep 8 19:45:08 UTC 2008 
State-Changed-Why:  
Fix committed to HEAD, will MFC in a week or so. 


Responsible-Changed-From-To: benjsc->jhb 
Responsible-Changed-By: jhb 
Responsible-Changed-When: Mon Sep 8 19:45:08 UTC 2008 
Responsible-Changed-Why:  
Fix committed to HEAD, will MFC in a week or so. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=124200 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/124200: commit references a PR
Date: Mon,  8 Sep 2008 19:45:20 +0000 (UTC)

 jhb         2008-09-08 19:44:57 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/kern             subr_sleepqueue.c 
   Log:
   SVN rev 182875 on 2008-09-08 19:44:57Z by jhb
   
   Close a race in sleepq_broadcast() where the sleepq could be reused after
   it had been assigned to the last sleeping thread.  That thread might have
   started running on another CPU and have reused that sleep queue.  Fix it
   by just walking the thread queue using TAILQ_FOREACH_SAFE() rather than
   a while loop.
   
   PR:             amd64/124200
   Discovered by:  tegge
   Tested by:      benjsc
   MFC after:      1 week
   
   Revision  Changes    Path
   1.57      +2 -3      src/sys/kern/subr_sleepqueue.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: jhb 
State-Changed-When: Mon Sep 29 15:45:39 UTC 2008 
State-Changed-Why:  
Fix merged to 7.x. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=124200 
>Unformatted:
