From nobody@FreeBSD.org  Wed May 16 13:26:42 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 2C36916A405
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 16 May 2007 13:26:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 1AACF13C459
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 16 May 2007 13:26:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l4GDQfr3084586
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 16 May 2007 13:26:41 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l4GDLe1Z076917;
	Wed, 16 May 2007 13:21:40 GMT
	(envelope-from nobody)
Message-Id: <200705161321.l4GDLe1Z076917@www.freebsd.org>
Date: Wed, 16 May 2007 13:21:40 GMT
From: Nikolay Pavlov<quetzal@zone3000.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: ZFS lock order reversal
X-Send-Pr-Version: www-3.0

>Number:         112711
>Category:       kern
>Synopsis:       [zfs] ZFS lock order reversal
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pjd
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed May 16 13:30:03 GMT 2007
>Closed-Date:    Wed Jun 06 20:33:51 GMT 2007
>Last-Modified:  Wed Jun  6 20:40:11 GMT 2007
>Originator:     Nikolay Pavlov
>Release:        7.0-CURRENT
>Organization:
zone3000.net
>Environment:
FreeBSD orion.zone3000.net 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Fri May 11 01:37:28 EEST 2007
root@orion.zone3000.net:/usr/obj/usr/src/sys/GENERIC  i386

>Description:
I am getting this lock order reversal while compiling ports.
I am using a simple disk pool.

quetzal@orion:~> zpool list                                                                                                                          <932>
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
pool                   11,6G   1,37G   10,3G    11%  ONLINE     -
quetzal@orion:~> zpool status                                                                                                                        <933>
  pool: pool
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        pool        ONLINE       0     0     0
          ad0s3     ONLINE       0     0     0

errors: No known data errors

quetzal@orion:~> zfs list                                                                                                                            <935>
NAME             USED  AVAIL  REFER  MOUNTPOINT
pool            1,37G  10,1G    18K  /pool
pool/distfiles   904M  10,1G   904M  /usr/ports/distfiles
pool/local       158M  10,1G   158M  /usr/local
pool/packages   55,0M  10,1G  55,0M  /usr/ports/packages
pool/ports       285M  10,1G   285M  /usr/ports


May 14 02:46:02 orion kernel: lock order reversal:
May 14 02:46:02 orion kernel: 1st 0xc737da20 zfs:&dr->dt.di.dr_mtx (zfs:&dr->dt.di.dr_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/
fs/zfs/dbuf.c:1866
May 14 02:46:02 orion kernel: 2nd 0xd111bcd0 zfs:&db->db_mtx (zfs:&db->db_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.
c:1888
May 14 02:46:02 orion kernel: KDB: stack backtrace:
May 14 02:46:02 orion kernel: db_trace_self_wrapper(c0962c19) at db_trace_self_wrapper+0x25
May 14 02:46:02 orion kernel: kdb_backtrace(0,ffffffff,c0a72c70,c0a72400,c0a0de2c,...) at kdb_backtrace+0x29
May 14 02:46:02 orion kernel: witness_checkorder(d111bcd0,9,c85b1f8c,760) at witness_checkorder+0x586
May 14 02:46:02 orion kernel: _sx_xlock(d111bcd0,c85b1f8c,760,c837bd20,637,...) at _sx_xlock+0x52
May 14 02:46:02 orion kernel: dbuf_sync_list(c737da38,d0020e80,0,7,c8515000,...) at dbuf_sync_list+0x15b
May 14 02:46:02 orion kernel: dbuf_sync_list(c4c559e0,d0020e80,257,3c,0,...) at dbuf_sync_list+0xde
May 14 02:46:02 orion kernel: dnode_sync(c4c55910,d0020e80,c4b32898,c4c55910,30,...) at dnode_sync+0x3a8
May 14 02:46:02 orion kernel: dmu_objset_sync_dnodes(c8515000,d0020e80,637,0,ca1a5ac8,...) at dmu_objset_sync_dnodes+0x29
May 14 02:46:02 orion kernel: dmu_objset_sync(c4b32800,c486a000,d0020e80,c476a800,0,...) at dmu_objset_sync+0x11d
May 14 02:46:02 orion kernel: dsl_pool_sync(c43b8200,637,0,c476a800,637,...) at dsl_pool_sync+0x13e
May 14 02:46:02 orion kernel: spa_sync(c476a800,637,0,c43b82ac,c85b592d,...) at spa_sync+0x33f
May 14 02:46:02 orion kernel: txg_sync_thread(c43b8200,e6a1ed38) at txg_sync_thread+0x183
May 14 02:46:02 orion kernel: fork_exit(c8584924,c43b8200,e6a1ed38) at fork_exit+0xac
May 14 02:46:02 orion kernel: fork_trampoline() at fork_trampoline+0x8
May 14 02:46:02 orion kernel: --- trap 0, eip = 0, esp = 0xe6a1ed70, ebp = 0 ---
May 14 02:46:02 orion kernel: lock order reversal:
May 14 02:46:02 orion kernel: 1st 0xc4908520 zfs:&dr->dt.di.dr_mtx (zfs:&dr->dt.di.dr_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/
fs/zfs/dbuf.c:1866
May 14 02:46:02 orion kernel: 2nd 0xd18faaa0 zfs:&db->db_mtx (zfs:&db->db_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.
c:1888
May 14 02:46:02 orion kernel: KDB: stack backtrace:
May 14 02:46:02 orion kernel: db_trace_self_wrapper(c0962c19) at db_trace_self_wrapper+0x25
May 14 02:46:02 orion kernel: kdb_backtrace(0,ffffffff,c0a72108,c0a72400,c0a0de2c,...) at kdb_backtrace+0x29
May 14 02:46:02 orion kernel: witness_checkorder(d18faaa0,9,c85b1f8c,760) at witness_checkorder+0x586
May 14 02:46:02 orion kernel: _sx_xlock(d18faaa0,c85b1f8c,760,c81c2620,637,...) at _sx_xlock+0x52
May 14 02:46:02 orion kernel: dbuf_sync_list(c4908538,c9513380,c58bb520,637,0,...) at dbuf_sync_list+0x15b
May 14 02:46:02 orion kernel: dbuf_sync_list(c793a810,c9513380,257,3c,0,...) at dbuf_sync_list+0xde
May 14 02:46:02 orion kernel: dnode_sync(c793a740,c9513380,c4b32898,c793a740,30,...) at dnode_sync+0x3a8
May 14 02:46:02 orion kernel: dmu_objset_sync_dnodes(c8515000,c9513380,637,0,c7e2f678,...) at dmu_objset_sync_dnodes+0x29
May 14 02:46:02 orion kernel: dmu_objset_sync(c4b32800,c4982000,c9513380,c476a800,0,...) at dmu_objset_sync+0x11d
May 14 02:46:02 orion kernel: dsl_pool_sync(c43b8200,637,0,c476a800,637,...) at dsl_pool_sync+0x13e
May 14 02:46:02 orion kernel: spa_sync(c476a800,637,0,c43b82ac,c85b592d,...) at spa_sync+0x33f
May 14 02:46:02 orion kernel: txg_sync_thread(c43b8200,e6a1ed38) at txg_sync_thread+0x183
May 14 02:46:02 orion kernel: fork_exit(c8584924,c43b8200,e6a1ed38) at fork_exit+0xac
May 14 02:46:02 orion kernel: fork_trampoline() at fork_trampoline+0x8
May 14 02:46:02 orion kernel: --- trap 0, eip = 0, esp = 0xe6a1ed70, ebp = 0 ---


vfs.zfs.dnlc.enable: 0
vfs.zfs.dnlc.max_nentries: 68506
vfs.zfs.dnlc.nentries: 0
vfs.zfs.dnlc.ncsize: 34253
vfs.zfs.arc_min: 16777216
vfs.zfs.arc_max: 167772160
vfs.zfs.mdcomp_disable: 0
vfs.zfs.prefetch_disable: 0
vfs.zfs.zio.taskq_threads: 0
vfs.zfs.recover: 0
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.debug: 0
kstat.zfs.misc.dnlcstats.hits: 0
kstat.zfs.misc.dnlcstats.misses: 0
kstat.zfs.misc.dnlcstats.negative_cache_hits: 0
kstat.zfs.misc.dnlcstats.enters: 0
kstat.zfs.misc.dnlcstats.double_enters: 0
kstat.zfs.misc.dnlcstats.purge_total_entries: 0
kstat.zfs.misc.dnlcstats.purge_all: 0
kstat.zfs.misc.dnlcstats.purge_vp: 0
kstat.zfs.misc.dnlcstats.purge_vfs: 0
kstat.zfs.misc.dnlcstats.purge_fs1: 0
kstat.zfs.misc.dnlcstats.pick_free: 0
kstat.zfs.misc.dnlcstats.pick_heuristic: 0
kstat.zfs.misc.dnlcstats.pick_last: 0
kstat.zfs.misc.arcstats.hits: 376113
kstat.zfs.misc.arcstats.misses: 84432
kstat.zfs.misc.arcstats.demand_data_hits: 232938
kstat.zfs.misc.arcstats.demand_data_misses: 3687
kstat.zfs.misc.arcstats.demand_metadata_hits: 123332
kstat.zfs.misc.arcstats.demand_metadata_misses: 75398
kstat.zfs.misc.arcstats.prefetch_data_hits: 296
kstat.zfs.misc.arcstats.prefetch_data_misses: 184
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 19547
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 5163
kstat.zfs.misc.arcstats.mru_hits: 44441
kstat.zfs.misc.arcstats.mru_ghost_hits: 1065
kstat.zfs.misc.arcstats.mfu_hits: 311926
kstat.zfs.misc.arcstats.mfu_ghost_hits: 5535
kstat.zfs.misc.arcstats.deleted: 245988
kstat.zfs.misc.arcstats.recycle_miss: 272594
kstat.zfs.misc.arcstats.mutex_miss: 129
kstat.zfs.misc.arcstats.evict_skip: 4839885
kstat.zfs.misc.arcstats.hash_elements: 6228
kstat.zfs.misc.arcstats.hash_elements_max: 13979
kstat.zfs.misc.arcstats.hash_collisions: 65501
kstat.zfs.misc.arcstats.hash_chains: 979
kstat.zfs.misc.arcstats.hash_chain_max: 6
kstat.zfs.misc.arcstats.p: 36156672
kstat.zfs.misc.arcstats.c: 46258688
kstat.zfs.misc.arcstats.c_min: 16777216
kstat.zfs.misc.arcstats.c_max: 167772160
kstat.zfs.misc.arcstats.size: 43229184

>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->pjd 
Responsible-Changed-By: remko 
Responsible-Changed-When: Wed May 16 14:17:30 UTC 2007 
Responsible-Changed-Why:  
reassign to maintainer 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112711 

From: Nikolay Pavlov <quetzal@zone3000.net>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc:  
Subject: Re: kern/112711: ZFS lock order reversal
Date: Wed, 6 Jun 2007 12:19:11 +0300

 On Wednesday, 16 May 2007 at 13:30:03 +0000, FreeBSD-gnats-submit@FreeBSD.org wrote:
 > Thank you very much for your problem report.
 > It has the internal identification `kern/112711'.
 > The individual assigned to look at your
 > report is: freebsd-bugs. 
 > 
 > You can access the state of your problem report at any time
 > via this link:
 > 
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=112711
 > 
 > >Category:       kern
 > >Responsible:    freebsd-bugs
 > >Synopsis:       ZFS lock order reversal
 > >Arrival-Date:   Wed May 16 13:30:03 GMT 2007
 
 I do not see this behavior after kernel update to Jun 4 2007, so i think
 you can close this ticket.
 
 -- 
 ======================================================================  
 - Best regards, Nikolay Pavlov. <<<-----------------------------------    
 ======================================================================  
 
State-Changed-From-To: open->closed 
State-Changed-By: remko 
State-Changed-When: Wed Jun 6 20:33:49 UTC 2007 
State-Changed-Why:  
Reporter mentions this can be closed (also concluded by kris@) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=112711 

From: Kris Kennaway <kris@obsecurity.org>
To: Nikolay Pavlov <quetzal@zone3000.net>
Cc: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Subject: Re: kern/112711: ZFS lock order reversal
Date: Wed, 6 Jun 2007 16:32:00 -0400

 On Wed, Jun 06, 2007 at 12:19:11PM +0300, Nikolay Pavlov wrote:
 > On Wednesday, 16 May 2007 at 13:30:03 +0000, FreeBSD-gnats-submit@FreeBSD.org wrote:
 > > Thank you very much for your problem report.
 > > It has the internal identification `kern/112711'.
 > > The individual assigned to look at your
 > > report is: freebsd-bugs. 
 > > 
 > > You can access the state of your problem report at any time
 > > via this link:
 > > 
 > > http://www.freebsd.org/cgi/query-pr.cgi?pr=112711
 > > 
 > > >Category:       kern
 > > >Responsible:    freebsd-bugs
 > > >Synopsis:       ZFS lock order reversal
 > > >Arrival-Date:   Wed May 16 13:30:03 GMT 2007
 > 
 > I do not see this behavior after kernel update to Jun 4 2007, so i think
 > you can close this ticket.
 
 Yeah, it was a false positive as discussed about 10^8 times on the
 mailing list (hint: check there first next time ;).
 
 Kris
>Unformatted:
