From nobody@FreeBSD.org  Sun May  4 11:31:45 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 1BE7D1065672
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  4 May 2008 11:31:45 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 1AD298FC17
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  4 May 2008 11:31:45 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m44BUuAY055689
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 4 May 2008 11:30:56 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m44BUuG3055688;
	Sun, 4 May 2008 11:30:56 GMT
	(envelope-from nobody)
Message-Id: <200805041130.m44BUuG3055688@www.freebsd.org>
Date: Sun, 4 May 2008 11:30:56 GMT
From: kevin <kevinxlinuz@163.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: lock order reversal
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         123392
>Category:       kern
>Synopsis:       [vfs] [lor] lock order reversal in vfs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun May 04 11:40:02 UTC 2008
>Closed-Date:    Sun Feb 07 20:04:35 UTC 2010
>Last-Modified:  Sun Feb 07 20:04:35 UTC 2010
>Originator:     kevin
>Release:        FreeBSD 8.0-CURRENT i386
>Organization:
163.com
>Environment:
FreeBSD xxx.people.163.org 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sun May  4 17:01:38 CST 2008     root@xxx.people.163.org:/usr/obj/usr/src/sys/G8C  i386

>Description:
when I update system from 7-release to 8-current.every thing works well,
but I find some thing on vfs.
-----------------
dmesg 
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #0: Sun May  4 17:01:38 CST 2008
    root@datastream.people.163.org:/usr/obj/usr/src/sys/G8C
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (2793.01-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf41  Stepping = 1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x441d<SSE3,RSVD2,MON,DS_CPL,CNXT-ID,xTPR>
real memory  = 1072103424 (1022 MB)
avail memory = 1040637952 (992 MB)
ACPI APIC Table: <DELL   170L   >
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <DELL 170L   > on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, f00000 (3) failed
acpi0: reservation of 1000000, 3ee70000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xed98-0xed9f mem 0xe8000000-0xefffffff,0xfeb80000-0xfebfffff irq 16 at device 2.0 on pci0
agp0: <Intel 82865G (865G GMCH) SVGA controller> on vgapci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
uhci0: <Intel 82801EB (ICH5) USB controller USB-A> port 0xff80-0xff9f irq 16 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 82801EB (ICH5) USB controller USB-A> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801EB (ICH5) USB controller USB-B> port 0xff60-0xff7f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 82801EB (ICH5) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801EB (ICH5) USB controller USB-D> port 0xff20-0xff3f irq 16 at device 29.3 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 82801EB (ICH5) USB controller USB-D> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 82801EB/R (ICH5) USB 2.0 controller> mem 0xffa80800-0xffa80bff irq 23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: wrong number of companions (4 != 3)
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 82801EB/R (ICH5) USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 8 ports with 8 removable, self powered
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
fxp0: <Intel 82801BA (D865) Pro/100 VE Ethernet> port 0xdf40-0xdf7f mem 0xfeaff000-0xfeafffff irq 20 at device 8.0 on pci1
miibus0: <MII bus> on fxp0
inphy0: <i82562ET 10/100 media interface> PHY 1 on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:13:20:d9:77:d1
fxp0: [ITHREAD]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH5 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf mem 0xfeb7fc00-0xfeb7ffff irq 18 at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel ICH5 SATA150 controller> port 0xfe00-0xfe07,0xfe10-0xfe13,0xfe20-0xfe27,0xfe30-0xfe33,0xfea0-0xfeaf irq 18 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pcm0: <Intel ICH5 (82801EB)> port 0xee00-0xeeff,0xedc0-0xedff mem 0xfeb7fa00-0xfeb7fbff,0xfeb7f900-0xfeb7f9ff irq 17 at device 31.5 on pci0
pcm0: [ITHREAD]
pcm0: primary codec not ready!
pcm0: <Analog Devices AD1980 AC97 Codec>
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xca7ff,0xca800-0xcbfff pnpid ORM0000 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ums0: <vendor 0x0461 USB Optical Mouse, class 0/0, rev 2.00/2.00, addr 2> on uhub1
ums0: 3 buttons and Z dir.
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounter "TSC" frequency 2793012705 Hz quality 800
Timecounters tick every 1.000 msec
ad0: 38146MB <WDC WD400BB-75JHC0 06.01C06> at ata0-master UDMA100
ZFS filesystem version 6
ZFS storage pool version 6
acd0: CDROM <GCR-8485B/1.05> at ata1-master UDMA33
ad4: 238475MB <Maxtor 7Y250M0 YAR511W0> at ata2-master SATA150
WARNING: WITNESS option enabled, expect reduced performance.
lock order reversal: (sleepable after non-sleepable)
 1st 0xc3e57020 struct mount mtx (struct mount mtx) @ /usr/src/sys/kern/vfs_subr.c:338
 2nd 0xc3e57000 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:365
KDB: stack backtrace:
db_trace_self_wrapper(c0765c5e,eee4eb6c,c057b70e,c076833d,c3e57000,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c076833d,c3e57000,c076e245,c076e245,c076e7e2,...) at kdb_backtrace+0x29
witness_checkorder(c3e57000,1,c076e7e2,16d,4,...) at witness_checkorder+0x6de
__lockmgr_args(c3e57000,200100,c3e57020,0,0,...) at __lockmgr_args+0x220
vfs_busy(c3e57000,200,0,c3a17cc0,1,...) at vfs_busy+0x1bc
vfs_mount_alloc(0,c07a0960,c076e588,c3a17cc0,c05b9060,...) at vfs_mount_alloc+0x78
vfs_mountroot(c07eb690,4,c075d7e8,264,0,...) at vfs_mountroot+0x26c
start_init(0,eee4ed38,c075f17e,30d,c3a15cf8,...) at start_init+0x65
fork_exit(c05077f0,0,eee4ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeee4ed70, ebp = 0 ---
lock order reversal: (sleepable after non-sleepable)
 1st 0xc3be4eb0 vnode interlock (vnode interlock) @ /usr/src/sys/fs/devfs/devfs_vnops.c:197
 2nd 0xc3be4e90 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2046
KDB: stack backtrace:
db_trace_self_wrapper(c0765c5e,eee4ea98,c057b70e,c076833d,c3be4e90,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c076833d,c3be4e90,c0759138,c0759138,c076e7e2,...) at kdb_backtrace+0x29
witness_checkorder(c3be4e90,9,c076e7e2,7fe,c3be4e90,...) at witness_checkorder+0x6de
__lockmgr_args(c3be4e90,80100,c3be4eb0,0,0,...) at __lockmgr_args+0x777
vop_stdlock(eee4eb90,c0759315,c075c4d5,80100,c3be4e38,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c07a09e0,eee4eb90,c07d2a60,c3be4e38,80100,...) at VOP_LOCK1_APV+0xa5
_vn_lock(c3be4e38,80100,c076e7e2,7fe,c0759315,...) at _vn_lock+0x5e
vget(c3be4e38,80100,c3a17cc0,c6,c3e52854,...) at vget+0x9c
devfs_allocv(c3e55000,c3e57000,eee4ec20,c3a17cc0,c3a17d64,...) at devfs_allocv+0x11d
devfs_root(c3e57000,80000,c083dfb8,c3a17cc0,4,...) at devfs_root+0x51
set_rootvnode(c083dfa0,0,c076e147,5f6,c05b9060,...) at set_rootvnode+0x2d
vfs_mountroot(c07eb690,4,c075d7e8,264,0,...) at vfs_mountroot+0x34c
start_init(0,eee4ed38,c075f17e,30d,c3a15cf8,...) at start_init+0x65
fork_exit(c05077f0,0,eee4ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeee4ed70, ebp = 0 ---
lock order reversal:
 1st 0xc3be4e90 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2046
 2nd 0xc3e52854 devfsmount (devfsmount) @ /usr/src/sys/fs/devfs/devfs_vnops.c:201
KDB: stack backtrace:
db_trace_self_wrapper(c0765c5e,eee4ebb4,c057b70e,c076833d,c3e52854,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c076833d,c3e52854,c07592d4,c07592d4,c0759315,...) at kdb_backtrace+0x29
witness_checkorder(c3e52854,9,c0759315,c9,c7,...) at witness_checkorder+0x6de
_sx_xlock(c3e52854,0,c0759315,c9,c3e52854,...) at _sx_xlock+0x7d
devfs_allocv(c3e55000,c3e57000,eee4ec20,c3a17cc0,c3a17d64,...) at devfs_allocv+0x142
devfs_root(c3e57000,80000,c083dfb8,c3a17cc0,4,...) at devfs_root+0x51
set_rootvnode(c083dfa0,0,c076e147,5f6,c05b9060,...) at set_rootvnode+0x2d
vfs_mountroot(c07eb690,4,c075d7e8,264,0,...) at vfs_mountroot+0x34c
start_init(0,eee4ed38,c075f17e,30d,c3a15cf8,...) at start_init+0x65
fork_exit(c05077f0,0,eee4ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeee4ed70, ebp = 0 ---
Trying to mount root from ufs:/dev/ad4s1a
lock order reversal: (sleepable after non-sleepable)
 1st 0xc3be4ba0 bufobj interlock (bufobj interlock) @ /usr/src/sys/kern/vfs_bio.c:2442
 2nd 0xe3a77088 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:2456
KDB: stack backtrace:
db_trace_self_wrapper(c0765c5e,eee4e790,c057b70e,c076833d,e3a77088,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c076833d,e3a77088,c076d139,c076d139,c076c6e8,...) at kdb_backtrace+0x29
witness_checkorder(e3a77088,9,c076c6e8,998,4,...) at witness_checkorder+0x6de
__lockmgr_args(e3a77088,81900,c3be4ba0,c076d095,50,...) at __lockmgr_args+0x777
getblk(c3be4af0,0,0,800,0,...) at getblk+0x153
breadn(c3be4af0,0,0,800,0,...) at breadn+0x44
bread(c3be4af0,0,0,800,0,...) at bread+0x4c
ffs_blkatoff(c3be4af0,0,0,0,eee4e9a8,...) at ffs_blkatoff+0xd1
ufs_lookup(eee4e9ec,c3be4af0,eee4eb50,c3be4af0,eee4ea0c,...) at ufs_lookup+0x2b5
VOP_CACHEDLOOKUP_APV(c07c22e0,eee4e9ec,eee4eb50,eee4eb3c,c39e0900,...) at VOP_CACHEDLOOKUP_APV+0xa5
vfs_cache_lookup(eee4ea6c,eee4ea6c,500000c,80000,c3be4af0,...) at vfs_cache_lookup+0xd0
VOP_LOOKUP_APV(c07c22e0,eee4ea6c,c076def5,1b0,eee4eb3c,...) at VOP_LOOKUP_APV+0xa5
lookup(eee4eb24,c076def5,d8,c0,c39e0a2c,...) at lookup+0x56e
namei(eee4eb24,eee4eb28,c057aeec,c05326bc,c083d810,...) at namei+0x44b
kern_unlinkat(c3a17cc0,ffffff9c,c076e584,1,eee4ec5c,...) at kern_unlinkat+0x46
kern_unlink(c3a17cc0,c076e584,1,631,0,...) at kern_unlink+0x27
vfs_mountroot_try(c076e73e,c075c4d7,c0757354,1,c05b9060,...) at vfs_mountroot_try+0x472
vfs_mountroot(c07eb690,4,c075d7e8,264,0,...) at vfs_mountroot+0x40e
start_init(0,eee4ed38,c075f17e,30d,c3a15cf8,...) at start_init+0x65
fork_exit(c05077f0,0,eee4ed38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeee4ed70, ebp = 0 ---
IP Filter: v4.1.28 initialized.  Default = pass all, Logging = enabled
ukbd0: <Dell Dell USB Keyboard, class 0/0, rev 1.10/3.01, addr 3> on uhub1
kbd2 at ukbd0
lock order reversal:
 1st 0xc61ab5d0 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2046
 2nd 0xc07ea0f8 kernel linker (kernel linker) @ /usr/src/sys/kern/kern_linker.c:687
KDB: stack backtrace:
db_trace_self_wrapper(c0765c5e,f2e586c8,c057b70e,c076833d,c07ea0f8,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c076833d,c07ea0f8,c075fb74,c075fb74,c075fbb8,...) at kdb_backtrace+0x29
witness_checkorder(c07ea0f8,9,c075fbb8,2af,f2e58718,...) at witness_checkorder+0x6de
_sx_xlock(c07ea0f8,0,c075fbb8,2af,c075fbb8,...) at _sx_xlock+0x7d
linker_file_lookup_set(c483b800,c075fc0a,f2e58734,f2e58730,0,...) at linker_file_lookup_set+0x4a
linker_file_register_sysctls(c07ea0f8,c075fbb8,19d,17c,0,...) at linker_file_register_sysctls+0x2d
linker_load_module(c6256d50,0,f2e58940,f2e5893c,f2e58938,...) at linker_load_module+0x92f
linker_load_dependencies(c4787d00,0,c6256ba0,2a4,4ba0,...) at linker_load_dependencies+0x194
link_elf_load_file(c07acd20,c42c48a0,f2e58c24,17c,0,...) at link_elf_load_file+0x4d7
linker_load_module(0,f2e58c4c,c075fbb8,3c7,81d5dd0,...) at linker_load_module+0x8cb
kern_kldload(c486bcc0,c3b92400,f2e58c70,0,0,...) at kern_kldload+0xc8
kldload(c486bcc0,f2e58cfc,4,c0768aae,c07a5ba0,...) at kldload+0x74
syscall(f2e58d38) at syscall+0x2a3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (304, FreeBSD ELF32, kldload), eip = 0x284d68fb, esp = 0xbfbfea6c, ebp = 0xbfbfea78 ---
drm0: <Intel i865G GMCH> on vgapci0
info: [drm] AGP at 0xe8000000 128MB
info: [drm] Initialized i915 1.5.0 20060119
drm0: [ITHREAD]

>How-To-Repeat:
I just update src and build everything and kernel.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-i386->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 5 00:34:55 UTC 2008 
Responsible-Changed-Why:  
Reclassify. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=123392 
State-Changed-From-To: open->closed 
State-Changed-By: gavin 
State-Changed-When: Sun Feb 7 20:03:13 UTC 2010 
State-Changed-Why:  
Close this PR.  At least one of the reported LORs has since been 
fixed, and the others all have their own PRs (which makes tracking 
progress with fixing them individually easier) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=123392 
>Unformatted:
