From nobody@FreeBSD.org  Thu Nov  6 08:19:42 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B81D81065675
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  6 Nov 2008 08:19:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id A6F088FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Thu,  6 Nov 2008 08:19:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id mA68JgRN050419
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 6 Nov 2008 08:19:42 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id mA68Jgt7050418;
	Thu, 6 Nov 2008 08:19:42 GMT
	(envelope-from nobody)
Message-Id: <200811060819.mA68Jgt7050418@www.freebsd.org>
Date: Thu, 6 Nov 2008 08:19:42 GMT
From: Sun Chang-Ming <snnn119@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: lock order reversal in zfs
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         128633
>Category:       kern
>Synopsis:       [zfs] [lor] lock order reversal in zfs
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    pjd
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Nov 06 08:20:01 UTC 2008
>Closed-Date:    
>Last-Modified:  Mon Sep 28 17:59:57 UTC 2009
>Originator:     Sun Chang-Ming
>Release:        8.0-CURRENT-200810
>Organization:
Perfect World Co.,Ltd. 
>Environment:
FreeBSD localhost.localdomain 8.0-CURRENT-200810 FreeBSD 8.0-CURRENT-200810 #0: Wed Oct 15 05:16:07 UTC 2008     root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
When I was booting my system,I saw some warnings of lock order reversal.
Please see the attachment for details.
>How-To-Repeat:
I don't know
>Fix:
I don't know

Patch attached with submission follows:

Trying to mount root from zfs:upool
lock order reversal:
 1st 0xc3877000 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:372
 2nd 0xc373adf4 devfs (devfs) @ /usr/src/sys/kern/vfs_lookup.c:428
 3rd 0xc3876d34 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:372
KDB: stack backtrace:
db_trace_self_wrapper(c0bc5e25,c3276828,c0838f75,4,c0bc17b5,...) at db_trace_self_wrapper+0x26
kdb_backtrace(4,c0bc17b5,c3876d34,c3523458,c3276880,...) at kdb_backtrace+0x29
_witness_debugger(c0bc86f3,c3876d34,c0bcef54,c3523458,c0bcf4f9,...) at _witness_debugger+0x25
witness_checkorder(c3876d34,1,c0bcf4f9,174,0,...) at witness_checkorder+0x800
__lockmgr_args(c3876d34,200100,c3876d50,0,0,...) at __lockmgr_args+0x237
vfs_busy(c3876d34,200,0,c373ae10,c109bc80,...) at vfs_busy+0x1bc
vfs_mount_alloc(c373ad9c,c109bc80,c384e1a0,c3539700,c37bd960,...) at vfs_mount_alloc+0x74
vfs_donmount(c3574d20,4001,c3276b50,c385e540,6,...) at vfs_donmount+0x1262
kernel_mount(c37bdc50,4001,c387a100,ffffffff,c3276bc0) at kernel_mount+0x84
kernel_vmount(4001,c0bcf143,c3571320,c0bcf152,c0bbc3d6,...) at kernel_vmount+0x63
vfs_mountroot_try(c0bcf455,c0bbc3d6,c0bb4490,1,c0879450,...) at vfs_mountroot_try+0x132
vfs_mountroot(c0d10ad0,4,c0bbd72b,265,c3574d20,...) at vfs_mountroot+0x418
start_init(0,c3276d38,c0bbf107,322,c3572d0c,...) at start_init+0x65
fork_exit(c07c0b00,0,c3276d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3276d70, ebp = 0 ---
lock order reversal:
 1st 0xc373a8b8 zfs (zfs) @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1123
 2nd 0xc3877000 vfslock (vfslock) @ /usr/src/sys/kern/vfs_subr.c:372
KDB: stack backtrace:
db_trace_self_wrapper(c0bc5e25,c32768f4,c0838f75,4,c0bc17b5,...) at db_trace_self_wrapper+0x26
kdb_backtrace(4,c0bc17b5,c0bcf4f9,c3523458,c327694c,...) at kdb_backtrace+0x29
_witness_debugger(c0bc86da,c3877000,c0bcef54,c3523458,c0bcf4f9,...) at _witness_debugger+0x25
witness_checkorder(c3877000,1,c0bcf4f9,174,0,...) at witness_checkorder+0x800
__lockmgr_args(c3877000,200100,c387701c,0,0,...) at __lockmgr_args+0x237
vfs_busy(c3877000,0,0,1ac,c3276b24,...) at vfs_busy+0x1bc
lookup(c3276b0c,c0bcec00,d4,bc,c353982c,...) at lookup+0x7bd
namei(c3276b0c,c3276b24,c0838dbc,c0e51548,c0bcf29b,...) at namei+0x44b
kern_unlinkat(c3574d20,ffffff9c,c0bcf29b,1,c3276c5c,...) at kern_unlinkat+0x46
kern_unlink(c3574d20,c0bcf29b,1,631,c0bbc3d6,...) at kern_unlink+0x27
vfs_mountroot_try(c0bcf455,c0bbc3d6,c0bb4490,1,c0879450,...) at vfs_mountroot_try+0x46e
vfs_mountroot(c0d10ad0,4,c0bbd72b,265,c3574d20,...) at vfs_mountroot+0x418
start_init(0,c3276d38,c0bbf107,322,c3572d0c,...) at start_init+0x65
fork_exit(c07c0b00,0,c3276d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc3276d70, ebp = 0 ---
lock order reversal:
 1st 0xc3578044 user map (user map) @ /usr/src/sys/vm/vm_map.c:3115
 2nd 0xc373a6a0 zfs (zfs) @ /usr/src/sys/kern/vfs_subr.c:2049
KDB: stack backtrace:
db_trace_self_wrapper(c0bc5e25,c327691c,c0838f75,4,c0bc17b5,...) at db_trace_self_wrapper+0x26
kdb_backtrace(4,c0bc17b5,c0bcf4f9,c3523660,c3276974,...) at kdb_backtrace+0x29
_witness_debugger(c0bc86da,c373a6a0,c109790d,c3523660,c0bcf4f9,...) at _witness_debugger+0x25
witness_checkorder(c373a6a0,1,c0bcf4f9,801,0,...) at witness_checkorder+0x800
__lockmgr_args(c373a6a0,200501,c373a6bc,0,0,...) at __lockmgr_args+0x237
vop_stdlock(c3276a78,c0bea9cd,c0bbc3d4,200501,c373a648,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c109bd20,c3276a78,c0cdd0c0,c373a648,200501,...) at VOP_LOCK1_APV+0xa5
_vn_lock(c373a648,200501,c0bcf4f9,801,4,...) at _vn_lock+0x5e
vget(c373a648,200501,c3574d20,4b4,0,...) at vget+0xc9
vnode_pager_lock(c1866c98,0,c0be7f93,127,c3276c18,...) at vnode_pager_lock+0x1e0
vm_fault(c3578000,80db000,2,8,80db020,...) at vm_fault+0x1df
trap_pfault(5,0,c0bf7580,2f5,c3572d0c,...) at trap_pfault+0x118
trap(c3276d38) at trap+0x289
calltrap() at calltrap+0x6
--- trap 0xc, eip = 0x80480e5, esp = 0xbfbfeef0, ebp = 0xbfbfef10 ---

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Thu Nov 6 11:24:51 UTC 2008 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=128633 
State-Changed-From-To: open->feedback 
State-Changed-By: pjd 
State-Changed-When: pon 28 wrz 2009 17:58:06 UTC 
State-Changed-Why:  
Can you show how how lines around line 1123 in 
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c looks like 
on the source your kernel was compiled from? Same for sys/kern/vfs_subr.c:372. 


Responsible-Changed-From-To: freebsd-fs->pjd 
Responsible-Changed-By: pjd 
Responsible-Changed-When: pon 28 wrz 2009 17:58:06 UTC 
Responsible-Changed-Why:  
I'll take this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=128633 
>Unformatted:
