From nik@crf-consulting.co.uk  Sat Mar 13 02:39:55 2004
Return-Path: <nik@crf-consulting.co.uk>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id C684216A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 13 Mar 2004 02:39:55 -0800 (PST)
Received: from crf-consulting.co.uk (82-44-220-218.cable.ubr10.haye.blueyonder.co.uk [82.44.220.218])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 1230F43D2F
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 13 Mar 2004 02:39:55 -0800 (PST)
	(envelope-from nik@crf-consulting.co.uk)
Received: from clan.nothing-going-on.org (clan.nothing-going-on.org [192.168.1.20])
	by crf-consulting.co.uk (8.12.9p2/8.12.9) with ESMTP id i2DAdsIP059709
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 13 Mar 2004 10:39:54 GMT
	(envelope-from nik@catkin)
Received: from clan.nothing-going-on.org (localhost [127.0.0.1])
	by clan.nothing-going-on.org (8.12.11/8.12.10) with ESMTP id i2DAdrmE093080
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 13 Mar 2004 10:39:53 GMT
	(envelope-from nik@clan.nothing-going-on.org)
Received: (from nik@localhost)
	by clan.nothing-going-on.org (8.12.11/8.12.10/Submit) id i2DAdrOE093079;
	Sat, 13 Mar 2004 10:39:53 GMT
	(envelope-from nik)
Message-Id: <200403131039.i2DAdrOE093079@clan.nothing-going-on.org>
Date: Sat, 13 Mar 2004 10:39:53 GMT
From: Nik Clayton <nik@crf-consulting.co.uk>
Reply-To: Nik Clayton <nik@crf-consulting.co.uk>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: panic: lockmgr: locking against myself (kern_lock.c:370)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         64206
>Category:       kern
>Synopsis:       panic: lockmgr: locking against myself (kern_lock.c:370)
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kan
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Mar 13 02:40:11 PST 2004
>Closed-Date:    Tue Mar 16 14:09:07 PST 2004
>Last-Modified:  Tue Mar 16 14:09:07 PST 2004
>Originator:     Nik Clayton
>Release:        FreeBSD 5.2-CURRENT i386
>Organization:
>Environment:

System: FreeBSD clan.nothing-going-on.org 5.2-CURRENT FreeBSD 5.2-CURRENT #8: Thu Mar 11 23:26:36 GMT 2004 nik@clan.nothing-going-on.org:/local/1/usr/src/sys/i386/compile/CLAN i386

-current built from a source tree pulled down on 2004-03-09

>Description:

After a clean boot, with background_fsck="NO" in /etc/rc.conf (so the disks 
had been fsck'd at startup), on a filesystem with softupdates enabled, I 
could reliably reproduce a panic by doing something disk intensive, like

    cd /usr/ports/multimedia/kdemultimedia3
    make clean extract

The panic would happen within a few seconds of the extraction starting.

This was completely reproducible:

    reboot
    disks get fscked
    login
    'make clean extract'
    panic

and I did so several times for testing.

A Google groups search for "kern_lock.c panic 370" shows numerous reports
of this over the past year or so.  Most of them have linked this to 
background fsck.

Since I had had background fsck enabled previously, I'll theorise wildly,
and suggest that the problem is an interaction between the softupdates code
and snapshot code (since the previous, incomplete, background fscks will
have left snapshots behind when the system panic'd before they completed).

>How-To-Repeat:

As above.

>Fix:

For the time being, I've turned off softupdates.  Obviously, that's not a 
realistic long term fix.

I have crash dumps, and the kernel.debug that was running when the panics
happened.  Contact me for copies.

Here's a typescript of the backtrace from one of the panics.

Script started on Thu Mar 11 09:32:26 2004
clan# gdb -k kernel.debug /var/crash/vmcore.0
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: lockmgr: locking against myself
panic messages:
---
panic: lockmgr: locking against myself
at line 370 in file ../../../kern/kern_lock.c
cpuid = 0; 
Debugger("panic")
panic: from debugger
at line 453 in file ../../../ddb/db_command.ccpuid = 0; 
boot() called on cpu#0
Uptime: 16h22m45s
Dumping 511 MB
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496
---
Reading symbols from /boot/kernel/snd_mss.ko...done.
Loaded symbols for /boot/kernel/snd_mss.ko
Reading symbols from /boot/kernel/snd_pcm.ko...done.
Loaded symbols for /boot/kernel/snd_pcm.ko
Reading symbols from /local/1/usr/src/sys/i386/compile/CLAN/modules/local/1/usr/src/sys/modules/acpi/acpi.ko...done.
Loaded symbols for /local/1/usr/src/sys/i386/compile/CLAN/modules/local/1/usr/src/sys/modules/acpi/acpi.ko
Reading symbols from /local/1/usr/src/sys/i386/compile/CLAN/modules/local/1/usr/src/sys/modules/nfsserver/nfsserver.ko.debug...done.
Loaded symbols for /local/1/usr/src/sys/i386/compile/CLAN/modules/local/1/usr/src/sys/modules/nfsserver/nfsserver.ko.debug
Reading symbols from /local/1/usr/src/sys/i386/compile/CLAN/modules/local/1/usr/src/sys/modules/linux/linux.ko.debug...done.
Loaded symbols for /local/1/usr/src/sys/i386/compile/CLAN/modules/local/1/usr/src/sys/modules/linux/linux.ko.debug
#0  doadump () at ../../../kern/kern_shutdown.c:240
240		dumping++;
(kgdb) backtrace
#0  doadump () at ../../../kern/kern_shutdown.c:240
#1  0xc051ab0f in boot (howto=260) at ../../../kern/kern_shutdown.c:374
#2  0xc051af7b in __panic () at ../../../kern/kern_shutdown.c:552
#3  0xc0445b42 in db_panic () at ../../../ddb/db_command.c:453
#4  0xc0445a92 in db_command (last_cmdp=0xc06f96e0, cmd_table=0x0, 
    aux_cmd_tablep=0xc06c6cc4, aux_cmd_tablep_end=0xc06c6cc8)
    at ../../../ddb/db_command.c:348
#5  0xc0445be5 in db_command_loop () at ../../../ddb/db_command.c:475
#6  0xc0448da5 in db_trap (type=3, code=0) at ../../../ddb/db_trap.c:73
#7  0xc06582ec in kdb_trap (type=3, code=0, regs=0xdda752d0)
    at ../../../i386/i386/db_interface.c:171
#8  0xc066db5c in trap (frame=
      {tf_fs = 24, tf_es = -1047003120, tf_ds = 16, tf_edi = -1066738309, tf_esi = 1, tf_ebp = -576236772, tf_isp = -576236804, tf_ebx = 0, tf_edx = 0, tf_ecx = -1056882688, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1067088379, tf_cs = 8, tf_eflags = 658, tf_esp = -1066666039, tf_ss = -1066733316})
    at ../../../i386/i386/trap.c:579
#9  0xc0658605 in Debugger (msg=0x0) at machine/cpufunc.h:60
#10 0xc051aed2 in __panic (file=0xc06addd9 "../../../kern/kern_lock.c", 
    line=370, fmt=0xc06add7b "lockmgr: locking against myself")
    at ../../../kern/kern_shutdown.c:536
#11 0xc050cc95 in lockmgr (lkp=0xce8f1f14, flags=34144290, interlkp=0x2000020, 
    td=0xc4a13540) at ../../../kern/kern_lock.c:439
#12 0xc056d9d9 in getblk (vp=0xc467db2c, blkno=7150880, size=16384, slpflag=0, 
    slptimeo=0, flags=0) at machine/pcpu.h:156
#13 0xc0569102 in breadn (vp=0xc467db2c, blkno=0, size=0, rablkno=0x0, 
    rabsize=0x0, cnt=0, cred=0x0, bpp=0x0) at ../../../kern/vfs_bio.c:700
#14 0xc05690ac in bread (vp=0x0, blkno=0, size=0, cred=0x0, bpp=0x0)
    at ../../../kern/vfs_bio.c:682
#15 0xc05e99ff in ffs_alloccg (ip=0xc4699000, cg=19, bpref=1787680, size=16384)
    at ../../../ufs/ffs/ffs_alloc.c:1287
#16 0xc05e9447 in ffs_hashalloc (ip=0xc4699000, cg=19, pref=0, size=16384, 
    allocator=0xc05e9910 <ffs_alloccg>) at ../../../ufs/ffs/ffs_alloc.c:1155
#17 0xc05e72c2 in ffs_alloc (ip=0xc4699000, lbn=223468, bpref=1787680, 
    size=16384, cred=0xc1976200, bnp=0xdda75610)
    at ../../../ufs/ffs/ffs_alloc.c:157
#18 0xc05eeaf3 in ffs_balloc_ufs2 (vp=0xc4698b2c, startoffset=0, size=16384, 
    cred=0xc1976200, flags=0, bpp=0xdda75720)
    at ../../../ufs/ffs/ffs_balloc.c:774
#19 0xc05f7aa0 in ffs_copyonwrite (devvp=0xc467db2c, bp=0xce848618)
    at ../../../ufs/ffs/ffs_snapshot.c:2035
#20 0xc04deab2 in spec_xstrategy (vp=0xc467db2c, bp=0xce848618)
    at ../../../fs/specfs/spec_vnops.c:493
#21 0xc04debab in spec_specstrategy (ap=0x0)
    at ../../../fs/specfs/spec_vnops.c:553
#22 0xc04ddc18 in spec_vnoperate (ap=0x0)
    at ../../../fs/specfs/spec_vnops.c:122
#23 0xc0569934 in bwrite (bp=0xce848618) at vnode_if.h:1141
#24 0xc056a42c in bawrite (bp=0x0) at ../../../kern/vfs_bio.c:1150
#25 0xc05ea8fa in ffs_nodealloccg (ip=0xc5aadc94, cg=19, ipref=65, mode=33188)
    at ../../../ufs/ffs/ffs_alloc.c:1637
#26 0xc05e9447 in ffs_hashalloc (ip=0xc5aadc94, cg=19, pref=0, size=33188, 
    allocator=0xc05ea380 <ffs_nodealloccg>)
    at ../../../ufs/ffs/ffs_alloc.c:1155
#27 0xc05e8b69 in ffs_valloc (pvp=0xc66a0410, mode=33188, cred=0xc48d2600, 
    vpp=0xdda75908) at ../../../ufs/ffs/ffs_alloc.c:857
#28 0xc0613bbf in ufs_makeinode (mode=33188, dvp=0xc66a0410, vpp=0xdda75bf0, 
    cnp=0xdda75c04) at ../../../ufs/ufs/ufs_vnops.c:2386
#29 0xc0610639 in ufs_create (ap=0xdda75a70)
    at ../../../ufs/ufs/ufs_vnops.c:200
#30 0xc0614168 in ufs_vnoperate (ap=0x0) at ../../../ufs/ufs/ufs_vnops.c:2822
#31 0xc058b0ee in vn_open_cred (ndp=0xdda75bdc, flagp=0xdda75cdc, cmode=420, 
    cred=0xc48d2600, fdidx=0) at vnode_if.h:118
#32 0xc058af43 in vn_open (ndp=0x0, flagp=0x0, cmode=0, fdidx=0)
    at ../../../kern/vfs_vnops.c:93
#33 0xc0584098 in kern_open (td=0xc4a13540, path=0x0, pathseg=UIO_USERSPACE, 
    flags=2562, mode=420) at ../../../kern/vfs_syscalls.c:971
#34 0xc0583fc0 in open (td=0x0, uap=0x0) at ../../../kern/vfs_syscalls.c:941
#35 0xc066e600 in syscall (frame=
      {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = 134672579, tf_esi = 2561, tf_ebp = -1077942136, tf_isp = -576234124, tf_ebx = 420, tf_edx = -19, tf_ecx = 67, tf_eax = 5, tf_trapno = 0, tf_err = 2, tf_eip = 672040575, tf_cs = 31, tf_eflags = 514, tf_esp = -1077942628, tf_ss = 47}) at ../../../i386/i386/trap.c:1008
#36 0x280e867f in ?? ()
---Can't read userspace from dump, or kernel process---

(kgdb) q

Script done on Thu Mar 11 09:32:46 2004
>Release-Note:
>Audit-Trail:

From: Pawel Jakub Dawidek <pjd@FreeBSD.org>
To: Nik Clayton <nik@crf-consulting.co.uk>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: kern/64206: panic: lockmgr: locking against myself (kern_lock.c:370)
Date: Tue, 16 Mar 2004 19:21:45 +0100

 --3MMMIZFJzhAsRj/+
 Content-Type: text/plain; charset=iso-8859-2
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 On Sat, Mar 13, 2004 at 10:39:53AM +0000, Nik Clayton wrote:
 +> >Description:
 +>=20
 +> After a clean boot, with background_fsck=3D"NO" in /etc/rc.conf (so the =
 disks=20
 +> had been fsck'd at startup), on a filesystem with softupdates enabled, I=
 =20
 +> could reliably reproduce a panic by doing something disk intensive, like
 +>=20
 +>     cd /usr/ports/multimedia/kdemultimedia3
 +>     make clean extract
 +>=20
 +> The panic would happen within a few seconds of the extraction starting.
 +>=20
 +> This was completely reproducible:
 +>=20
 +>     reboot
 +>     disks get fscked
 +>     login
 +>     'make clean extract'
 +>     panic
 +>=20
 +> and I did so several times for testing.
 +>=20
 +> A Google groups search for "kern_lock.c panic 370" shows numerous reports
 +> of this over the past year or so.  Most of them have linked this to=20
 +> background fsck.
 +>=20
 +> Since I had had background fsck enabled previously, I'll theorise wildly,
 +> and suggest that the problem is an interaction between the softupdates c=
 ode
 +> and snapshot code (since the previous, incomplete, background fscks will
 +> have left snapshots behind when the system panic'd before they completed=
 ).
 
 Thanks for reporting this! I've exactly same problem, but I was sure this
 is HW issue:)
 
 In my configuration I've only one partition, softupdates are off,
 so I don't know why snapshot is created (as it looks from your debug).
 
 --=20
 Pawel Jakub Dawidek                       http://www.FreeBSD.org
 pjd@FreeBSD.org                           http://garage.freebsd.pl
 FreeBSD committer                         Am I Evil? Yes, I Am!
 
 --3MMMIZFJzhAsRj/+
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.2.4 (FreeBSD)
 
 iD8DBQFAV0W5ForvXbEpPzQRAvKCAJ9oajjagXLC/R/7CzFHg8xN6ehMhgCfWseG
 UuR5M2tHLQojBOPUYEGx++c=
 =kEnz
 -----END PGP SIGNATURE-----
 
 --3MMMIZFJzhAsRj/+--
Responsible-Changed-From-To: freebsd-bugs->kan 
Responsible-Changed-By: kan 
Responsible-Changed-When: Tue Mar 16 14:07:27 PST 2004 
Responsible-Changed-Why:  
I'll handle this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64206 
State-Changed-From-To: open->closed 
State-Changed-By: kan 
State-Changed-When: Tue Mar 16 14:08:04 PST 2004 
State-Changed-Why:  
Fixed in CVS. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=64206 
>Unformatted:
