From root@cakebox.homeunix.net  Sat May 17 16:48:23 2008
Return-Path: <root@cakebox.homeunix.net>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 66FEF1065670
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 17 May 2008 16:48:23 +0000 (UTC)
	(envelope-from root@cakebox.homeunix.net)
Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.174])
	by mx1.freebsd.org (Postfix) with ESMTP id BBDF78FC13
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 17 May 2008 16:48:22 +0000 (UTC)
	(envelope-from root@cakebox.homeunix.net)
Received: from cakebox.homeunix.net (stgt-4d023f6e.pool.mediaWays.net [77.2.63.110])
	by mrelayeu.kundenserver.de (node=mrelayeu2) with ESMTP (Nemesis)
	id 0MKwtQ-1JxPNq0YiI-0004UD; Sat, 17 May 2008 18:35:46 +0200
Received: from cakebox.homeunix.net (localhost [127.0.0.1])
	by cakebox.homeunix.net (Postfix) with ESMTP id 82CF032085A;
	Sat, 17 May 2008 18:35:00 +0200 (CEST)
Received: by cakebox.homeunix.net (Postfix, from userid 0)
	id A66CE320835; Sat, 17 May 2008 18:34:56 +0200 (CEST)
Message-Id: <20080517163456.A66CE320835@cakebox.homeunix.net>
Date: Sat, 17 May 2008 18:34:56 +0200 (CEST)
From: freebsd@nagilum.org
To: FreeBSD-gnats-submit@freebsd.org
Cc: freebsd@nagilum.org
Subject: [crash] 7.0-STABLE on Soekris net4801 (sys/vm/vm_pageout.c)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         123768
>Category:       i386
>Synopsis:       [panic] [vm] 7.0-STABLE locking issue on Soekris net4801 (sys/vm/vm_pageout.c)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kib
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat May 17 16:50:01 UTC 2008
>Closed-Date:    Sun Nov 23 16:10:22 UTC 2008
>Last-Modified:  Sun Nov 23 16:10:22 UTC 2008
>Originator:     Alexander
>Release:        FreeBSD 7.0-STABLE i386
>Organization:
private
>Environment:
System: FreeBSD cakebox.tis 7.0-STABLE FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008 root@cpqak.tis:/usr/obj/usr/src/sys/net4801 i386

	Soekris Net4801, FreeBSD STABLE i386 April/May 2008
>Description:
	Crash under high network and cpu load.
	Here the results of two dumps, I'll collect some more
	as long as the problem exists.
	I haven't enabled WITNESS since I don't think it will help in this case
	but if you think otherwise let me know.
1:
[root@cakebox ~]# kgdb -v /boot/kernel/kernel /var/crash/vmcore.0
kgdb: core file: /var/crash/vmcore.0
kgdb: kernel image: /boot/kernel/kernel
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
There is no member named pathname.
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_journal.ko

Unread portion of the kernel message buffer:
lockmgr: thread 0xc0fdf440 unlocking unheld lock
panic: vn_finished_write: neg cnt
Uptime: 16h29m13s
Physical memory: 123 MB
Dumping 50 MB: 35 19 3

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc0504669 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc0504820 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:572
#3  0xc0574115 in vn_finished_write (mp=0xc0feb538)
    at /usr/src/sys/kern/vfs_vnops.c:1049
#4  0xc0637816 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1028
#5  0xc04e8d7b in fork_exit (callout=0xc0636a30 <vm_pageout>, arg=0x0, 
    frame=0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
#6  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205
(kgdb) list *0xc0574115
0xc0574115 is in vn_finished_write (/usr/src/sys/kern/vfs_vnops.c:1050).
1045                    return;
1046            MNT_ILOCK(mp);
1047            mp->mnt_writeopcount--;
1048            if (mp->mnt_writeopcount < 0)
1049                    panic("vn_finished_write: neg cnt");
1050            if ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0 &&
1051                mp->mnt_writeopcount <= 0)
1052                    wakeup(&mp->mnt_writeopcount);
1053            MNT_IUNLOCK(mp);
1054    }
(kgdb) q

2:
root@cakebox ~ > kgdb -v /boot/kernel/kernel /var/crash/vmcore.1 
kgdb: core file: /var/crash/vmcore.1
kgdb: kernel image: /boot/kernel/kernel
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
There is no member named pathname.
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols from /boot/kernel/geom_journal.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_journal.ko

Unread portion of the kernel message buffer:
panic: lockmgr: thread 0xc0fdf440, not exclusive lock holder 0xc0fdeaa0 unlocking
Uptime: 9h37m37s
Physical memory: 123 MB
Dumping 45 MB: 30 14

#0  doadump () at pcpu.h:195
195     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) bt
#0  doadump () at pcpu.h:195
#1  0xc0504669 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc0504820 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:572
#3  0xc04f5a67 in _lockmgr (lkp=0xc1731278, flags=6, interlkp=0xc17312a8, 
    td=0xc0fdf440, file=0x0, line=0) at /usr/src/sys/kern/kern_lock.c:414
#4  0xc055e519 in vop_stdunlock (ap=0xc7b6ac1c)
    at /usr/src/sys/kern/vfs_default.c:280
#5  0xc065fecb in VOP_UNLOCK_APV (vop=0xc06b5dc0, a=0xc7b6ac1c)
    at vnode_if.c:1667
#6  0xc056a350 in vput (vp=0xc1731220) at vnode_if.h:877
#7  0xc06377cd in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1025
#8  0xc04e8d7b in fork_exit (callout=0xc0636a30 <vm_pageout>, arg=0x0, 
    frame=0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
#9  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:205
(kgdb) list *0xc06377cd
0xc06377cd is in vm_pageout (/usr/src/sys/vm/vm_pageout.c:1025).
1020    unlock_and_continue:
1021                            VM_OBJECT_UNLOCK(object);
1022                            if (mp != NULL) {
1023                                    vm_page_unlock_queues();
1024                                    if (vp != NULL)
1025                                            vput(vp);
1026                                    VFS_UNLOCK_GIANT(vfslocked);
1027                                    vm_object_deallocate(object);
1028                                    vn_finished_write(mp);
1029                                    vm_page_lock_queues();
(kgdb) list *0xc056a350
0xc056a350 is in vput (/usr/src/sys/kern/vfs_subr.c:2195).
2190            error = 0;
2191
2192            if (vp->v_usecount > 1 || ((vp->v_iflag & VI_DOINGINACT) &&
2193                vp->v_usecount == 1)) {
2194                    VOP_UNLOCK(vp, 0, td);
2195                    v_decr_usecount(vp);
2196                    return;
2197            }
2198
2199            if (vp->v_usecount != 1) {
(kgdb) q
root@cakebox ~ > mount
/dev/ad0s1a on / (ufs, NFS exported, local, soft-updates)
devfs on /dev (devfs, local)
procfs on /proc (procfs, local)
devfs on /var/named/dev (devfs, local)
/dev/ad0s1d.eli.journal on /export/home (ufs, asynchronous, NFS exported, local, gjournal)

and dmesg:

Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
    root@cpqak.tis:/usr/obj/usr/src/sys/net4801
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Geode(TM) Integrated Processor by National Semi (266.63-MHz 586-class CPU)
  Origin = "Geode by NSC"  Id = 0x540  Stepping = 0
  Features=0x808131<FPU,TSC,MSR,CX8,CMOV,MMX>
real memory  = 134217728 (128 MB)
avail memory = 126025728 (120 MB)
cryptosoft0: <software crypto> on motherboard
cpu0 on motherboard
pcib0: <Host to PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
sis0: <NatSemi DP8381[56] 10/100BaseTX> port 0xe100-0xe1ff mem 0xa0000000-0xa000
0fff irq 10 at device 6.0 on pci0
sis0: Silicon Revision: DP83816A
miibus0: <MII bus> on sis0
nsphyter0: <DP83815 10/100 media interface> PHY 0 on miibus0
nsphyter0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis0: Ethernet address: 00:00:24:c1:4f:c4
sis0: [ITHREAD]
sis1: <NatSemi DP8381[56] 10/100BaseTX> port 0xe200-0xe2ff mem 0xa0001000-0xa000
1fff irq 10 at device 7.0 on pci0
sis1: Silicon Revision: DP83816A
miibus1: <MII bus> on sis1
nsphyter1: <DP83815 10/100 media interface> PHY 0 on miibus1
nsphyter1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis1: Ethernet address: 00:00:24:c1:4f:c5
sis1: [ITHREAD]
sis2: <NatSemi DP8381[56] 10/100BaseTX> port 0xe300-0xe3ff mem 0xa0002000-0xa0002fff irq 10 at device 8.0 on pci0
sis2: Silicon Revision: DP83816A
miibus2: <MII bus> on sis2
nsphyter2: <DP83815 10/100 media interface> PHY 0 on miibus2
nsphyter2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis2: Ethernet address: 00:00:24:c1:4f:c6
sis2: [ITHREAD]
Geode GPIO@ = 6100
Geode Soekris net4801 comBIOS ver. 1.33 20080103 Copyright (C) 2000-2007
isab0: <PCI-ISA bridge> port 0x6100-0x613f,0x6200-0x623f at device 18.0 on pci0
isa0: <ISA bus> on isab0
pci0: <bridge> at device 18.1 (no driver attached)
atapci0: <National Geode SC1100 ATA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe000-0xe00f at device 18.2 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
Geode CBA@ 0x6000
Geode rev: 06 03
Timecounter "Geode" frequency 27000000 Hz quality 1000
pci0: <bridge> at device 18.5 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xa0003000-0xa0003fff irq 11 at device 19.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <(0x0e11) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 3 ports with 3 removable, self powered
orm0: <ISA Option ROM> at iomem 0xc8000-0xd0fff pnpid ORM0000 on isa0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio0: [FILTER]
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
sio1: [FILTER]
Timecounters tick every 1.000 msec
ad0: 152627MB <SAMSUNG HM160HC LQ100-10> at ata0-master PIO4
Trying to mount root from ufs:/dev/ad0s1a
Accounting enabled

options used to build the kernel+modules:
CPUTYPE?=pentium
CFLAGS=-O -pipe -mtune=pentium -march=pentium

>How-To-Repeat:
	Cause high network & CPU load.
	For example NFS export / mount it on a fast machine and do a
	portupgrade or any other io intensive operation on the nfs mount.
	I can also reproduce the problem using a bittorrent client provided
	there is enough traffic.
	It typically takes between 1-12 hours for the box to crash.
	I am also using geli+gjournal for one filesystem on the box but not /.
	I am unsure if it relevant.
	
	my kernconf:

cpu             I586_CPU
ident           NET4801

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols

options         CPU_GEODE               # Geode support
options         CPU_SOEKRIS             # Soekris support

options         SCHED_ULE               # ULE scheduler
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking
options         INET6                   # IPv6 communications protocols
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         GEOM_ELI                # encryption on GEOM providers
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big directories
options         UFS_GJOURNAL            # Enable gjournal-based UFS journaling.
options         NFSSERVER               # Network Filesystem Server
#options        MSDOSFS                 # MSDOS Filesystem
options         PROCFS                  # Process filesystem (requires PSEUDOFS)options         PSEUDOFS                # Pseudo-filesystem framework
options         COMPAT_43TTY            # BSD 4.3 TTY compat [KEEP THIS!]
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options         ADAPTIVE_GIANT          # Giant mutex is adaptive.

# Bus support.
device          eisa
device          pci

# ATA and ATAPI devices
device          ata
device          atadisk         # ATA disk drives
options         ATA_STATIC_ID   # Static device numbering

# SCSI peripherals
device          scbus           # SCSI bus (required for SCSI)
device          da              # Direct Access (disks)

# Serial (COM) ports
device          sio             # 8250, 16[45]50 based serial ports

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus          # MII bus support
device          sis             # Silicon Integrated Systems SiS 900/SiS 7016

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
device          ether           # Ethernet support
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf             # Berkeley packet filter

device          pf              # Packet Filter
device          pflog           # Packet Filter logging

# USB support
device          ohci            # OHCI PCI->USB interface
device          usb             # USB Bus (required)
device          umass           # Disks/Mass storage - Requires scbus and da

device          crypto          # core crypto support
device          cryptodev       # /dev/crypto for access to h/w



>Fix:

	I wish I knew..
>Release-Note:
>Audit-Trail:

From: Alexander <freebsd@nagilum.org>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
	net4801 (sys/vm/vm_pageout.c)
Date: Sun, 18 May 2008 10:33:59 +0200

 I almost forgot:
 info.0:
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 52527104B (50 MB)
    Blocksize: 512
    Dumptime: Sat May 17 05:01:08 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: vn_finished_write: neg cnt
    Dump Parity: 817259864
    Bounds: 0
    Dump Status: good
 
 info.1:
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 48058368B (45 MB)
    Blocksize: 512
    Dumptime: Sat May 17 15:45:04 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20
 0xc0fdeaa0 unlocking
    Dump Parity: 1101452627
    Bounds: 1
    Dump Status: good
 
 I think something is messing up struct mount *mp; but what and where..?
 
 
 ----------------------------------------------------------------
 cakebox.homeunix.net - all the machine one needs..

From: Alexander <freebsd@nagilum.org>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
	net4801 (sys/vm/vm_pageout.c)
Date: Sun, 18 May 2008 22:22:56 +0200

 I just got another crash:
 root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.2
 kgdb: core file: /var/crash/vmcore.2
 kgdb: kernel image: /boot/kernel/kernel
 [GDB will not be able to debug user-mode threads: =20
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 There is no member named pathname.
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kernel/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20
 /boot/kernel/netgraph.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/netgraph.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 
 Unread portion of the kernel message buffer:
 lockmgr: thread 0xc0fdf440 unlocking unheld lock
 panic: vn_finished_write: neg cnt
 Uptime: 1d2h19m52s
 Physical memory: 123 MB
 Dumping 59 MB: 44 28 12
 
 #0  doadump () at pcpu.h:195
 195     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41=
 8
 #2  0xc0504820 in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc0574115 in vn_finished_write (mp=3D0xc0feb538)
      at /usr/src/sys/kern/vfs_vnops.c:1049
 #4  0xc0637816 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1028
 #5  0xc04e8d7b in fork_exit (callout=3D0xc0636a30 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
 #6  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
 05
 (kgdb) x 0xc7b6ad38
 0xc7b6ad38:     0x00000000
 
 Some time after I the machine was running again I also saw this (for =20
 the first time) on the console:
 fsync: giving up on dirty
 0xc1286550: tag devfs, type VCHR
      usecount 1, writecount 0, refcount 5 mountedhere 0xc115e300
      flags ()
      v_object 0xc12e0b24 ref 0 pages 7
       lock type devfs: EXCL (count 1) by thread 0xc11d8aa0 (pid 1071)
          dev ad0s1d.eli.journal
 GEOM_JOURNAL: Cannot suspend file system /export/home (error=3D35).
 fsync: giving up on dirty
 0xc1286550: tag devfs, type VCHR
      usecount 1, writecount 0, refcount 5 mountedhere 0xc115e300
      flags ()
      v_object 0xc12e0b24 ref 0 pages 7
       lock type devfs: EXCL (count 1) by thread 0xc11d8aa0 (pid 1071)
          dev ad0s1d.eli.journal
 GEOM_JOURNAL: Cannot suspend file system /export/home (error=3D35).
 
 I looked:
 root@cakebox ~ > ps -axp 1071
    PID  TT  STAT      TIME COMMAND
   1071  ??  DL     0:00.92 [g_journal switcher]
 
 
 
 ----------------------------------------------------------------
 cakebox.homeunix.net - all the machine one needs..

From: Alexander <freebsd@nagilum.org>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
	net4801 (sys/vm/vm_pageout.c)
Date: Tue, 20 May 2008 22:45:22 +0200

 ..the info files for dump vmcore.2:
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 62296064B (59 MB)
    Blocksize: 512
    Dumptime: Sun May 18 18:15:51 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: vn_finished_write: neg cnt
    Dump Parity: 1941630040
    Bounds: 2
    Dump Status: good
 
 and the info file for vmcore.3:
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 49664000B (47 MB)
    Blocksize: 512
    Dumptime: Mon May 19 03:03:45 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20
 0xc0fdeaa0 unlocking
    Dump Parity: 3491160915
    Bounds: 3
    Dump Status: good
 
 and the backtrace:
 kgdb -v /boot/kernel/kernel /var/crash/vmcore.3
 kgdb: core file: /var/crash/vmcore.3
 kgdb: kernel image: /boot/kernel/kernel
 [GDB will not be able to debug user-mode threads: =20
 /usr/lib/libthread_db.so: Unde
 fined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 There is no member named pathname.
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kerne
 l/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20
 /boot/kerne
 l/netgraph.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/netgraph.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/k
 ernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 
 Unread portion of the kernel message buffer:
 panic: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20
 0xc0fdeaa0 unlockin
 g
 Uptime: 8h45m36s
 Physical memory: 123 MB
 Dumping 47 MB: 32 16
 
 #0  doadump () at pcpu.h:195
 195     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41=
 8
 #2  0xc0504820 in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc04f5a67 in _lockmgr (lkp=3D0xc129b6b8, flags=3D6, interlkp=3D0xc129b6=
 e8,
      td=3D0xc0fdf440, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_lock.c=
 :414
 #4  0xc055e519 in vop_stdunlock (ap=3D0xc7b6ac1c)
      at /usr/src/sys/kern/vfs_default.c:280
 #5  0xc065fecb in VOP_UNLOCK_APV (vop=3D0xc06b5dc0, a=3D0xc7b6ac1c)
      at vnode_if.c:1667
 #6  0xc056a350 in vput (vp=3D0xc129b660) at vnode_if.h:877
 #7  0xc06377cd in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1025
 #8  0xc04e8d7b in fork_exit (callout=3D0xc0636a30 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
 #9  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
 05
 (kgdb) list *0xc056a350
 0xc056a350 is in vput (/usr/src/sys/kern/vfs_subr.c:2195).
 2190            error =3D 0;
 2191
 2192            if (vp->v_usecount > 1 || ((vp->v_iflag & VI_DOINGINACT) &&
 2193                vp->v_usecount =3D=3D 1)) {
 2194                    VOP_UNLOCK(vp, 0, td);
 2195                    v_decr_usecount(vp);
 2196                    return;
 2197            }
 2198
 2199            if (vp->v_usecount !=3D 1) {
 (kgdb) list *0xc055e519
 0xc055e519 is in vop_stdunlock (/usr/src/sys/kern/vfs_default.c:282).
 
 277     {
 278             struct vnode *vp =3D ap->a_vp;
 279
 280             return (lockmgr(vp->v_vnlock, ap->a_flags | =20
 LK_RELEASE, VI_MTX(vp),
 281                 ap->a_td));
 282     }
 283
 284     /* See above. */
 285     int
 286     vop_stdislocked(ap)
 (kgdb) list *0xc04f5a67
 0xc04f5a67 is in _lockmgr (/usr/src/sys/kern/kern_lock.c:418).
 413                                 lkp->lk_lockholder !=3D LK_KERNPROC) {
 414                                     panic("lockmgr: thread %p, not =20
 %s %p unlocking",
 415                                         thr, "exclusive lock holder",
 416                                         lkp->lk_lockholder);
 417                             }
 418                             if (lkp->lk_lockholder !=3D LK_KERNPROC)
 419                                     COUNT(td, -1);
 420                             if (lkp->lk_exclusivecount =3D=3D 1) {
 421                                     lkp->lk_flags &=3D ~LK_HAVE_EXCL;
 422                                     lkp->lk_lockholder =3D LK_NOPROC;
 (kgdb) up 3
 #3  0xc04f5a67 in _lockmgr (lkp=3D0xc129b6b8, flags=3D6, interlkp=3D0xc129b6=
 e8,
      td=3D0xc0fdf440, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_lock.c=
 :414
 414                                     panic("lockmgr: thread %p, not =20
 %s %p unlocking",
 (kgdb) frame lkp->lk_lockholder
 #0  0x00000000 in ?? ()
 
 
 
 ----------------------------------------------------------------
 cakebox.homeunix.net - all the machine one needs..

From: Alexander <freebsd@nagilum.org>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
	net4801 (sys/vm/vm_pageout.c)
Date: Sun, 01 Jun 2008 13:08:42 +0200

 This appears to be related to geli+gjournal, I managed to crash the =20
 system just by deleting a couple of huge files on the geli+gjournal =20
 volume.
 
 A few more crashes:
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 55967744B (53 MB)
    Blocksize: 512
    Dumptime: Mon May 19 08:38:28 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20
 0xc1425000 unlocking
    Dump Parity: 322179330
    Bounds: 4
    Dump Status: good
 
 root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.4
 kgdb: core file: /var/crash/vmcore.4
 kgdb: kernel image: /boot/kernel/kernel
 [GDB will not be able to debug user-mode threads: =20
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 There is no member named pathname.
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kernel/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20
 /boot/kernel/netgraph.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/netgraph.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 
 Unread portion of the kernel message buffer:
 panic: lockmgr: thread 0xc0fdf440, not exclusive lock holder =20
 0xc1425000 unlocking
 Uptime: 5h33m42s
 Physical memory: 123 MB
 Dumping 53 MB: 38 22 6
 
 #0  doadump () at pcpu.h:195
 195     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41=
 8
 #2  0xc0504820 in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc04f5a67 in _lockmgr (lkp=3D0xc129baf8, flags=3D6, interlkp=3D0xc129bb=
 28,
      td=3D0xc0fdf440, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_lock.c=
 :414
 #4  0xc055e519 in vop_stdunlock (ap=3D0xc7b6ac1c)
      at /usr/src/sys/kern/vfs_default.c:280
 #5  0xc065fecb in VOP_UNLOCK_APV (vop=3D0xc06b5dc0, a=3D0xc7b6ac1c)
      at vnode_if.c:1667
 #6  0xc056a350 in vput (vp=3D0xc129baa0) at vnode_if.h:877
 #7  0xc06377cd in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1025
 #8  0xc04e8d7b in fork_exit (callout=3D0xc0636a30 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
 #9  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
 05
 
 root@cakebox /var/crash > cat info.5
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 50630656B (48 MB)
    Blocksize: 512
    Dumptime: Mon May 19 20:03:08 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: vn_finished_write: neg cnt
    Dump Parity: 2553044824
    Bounds: 5
    Dump Status: good
 root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.5
 kgdb: core file: /var/crash/vmcore.5
 kgdb: kernel image: /boot/kernel/kernel
 [GDB will not be able to debug user-mode threads: =20
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 There is no member named pathname.
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kernel/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20
 /boot/kernel/netgraph.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/netgraph.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 
 Unread portion of the kernel message buffer:
 panic: vn_finished_write: neg cnt
 Uptime: 5h36m3s
 Physical memory: 123 MB
 Dumping 48 MB: 33 17 1
 
 #0  doadump () at pcpu.h:195
 195     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41=
 8
 #2  0xc0504820 in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc0574115 in vn_finished_write (mp=3D0xc0ff6538)
      at /usr/src/sys/kern/vfs_vnops.c:1049
 #4  0xc0637816 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1028
 #5  0xc04e8d7b in fork_exit (callout=3D0xc0636a30 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
 #6  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
 05
 root@cakebox /var/crash > cat info.6
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 54063104B (51 MB)
    Blocksize: 512
    Dumptime: Tue May 20 18:44:08 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: vn_finished_write: neg cnt
    Dump Parity: 2624807000
    Bounds: 6
    Dump Status: good
 root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.6
 kgdb: core file: /var/crash/vmcore.6
 kgdb: kernel image: /boot/kernel/kernel
 [GDB will not be able to debug user-mode threads: =20
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 There is no member named pathname.
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kernel/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20
 /boot/kernel/netgraph.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/netgraph.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 
 Unread portion of the kernel message buffer:
 lockmgr: thread 0xc0fe0440 unlocking unheld lock
 panic: vn_finished_write: neg cnt
 Uptime: 22h39m57s
 Physical memory: 123 MB
 Dumping 51 MB: 36 20 4
 
 #0  doadump () at pcpu.h:195
 195     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41=
 8
 #2  0xc0504820 in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc0574115 in vn_finished_write (mp=3D0xc0ff6538)
      at /usr/src/sys/kern/vfs_vnops.c:1049
 #4  0xc0637816 in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1028
 #5  0xc04e8d7b in fork_exit (callout=3D0xc0636a30 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
 #6  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
 05
 
 Dump7 is pretty much the same, so I'll continue with dump 8:
 root@cakebox /var/crash > cat info.8
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 55463936B (52 MB)
    Blocksize: 512
    Dumptime: Fri May 23 10:31:47 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.0-STABLE #5: Thu May 15 22:50:38 CEST 2008
      root@cpqak.tis:/usr/obj/usr/src/sys/net4801
    Panic String: lockmgr: thread 0xc0fe0440, not exclusive lock holder =20
 0xc0fdfaa0 unlocking
    Dump Parity: 3532742481
    Bounds: 8
    Dump Status: good
 root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.8
 kgdb: core file: /var/crash/vmcore.8
 kgdb: kernel image: /boot/kernel/kernel
 [GDB will not be able to debug user-mode threads: =20
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 There is no member named pathname.
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kernel/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from =20
 /boot/kernel/netgraph.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/netgraph.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 
 Unread portion of the kernel message buffer:
 panic: lockmgr: thread 0xc0fe0440, not exclusive lock holder =20
 0xc0fdfaa0 unlocking
 Uptime: 17h33m3s
 Physical memory: 123 MB
 Dumping 52 MB: 37 21 5
 
 #0  doadump () at pcpu.h:195
 195     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:195
 #1  0xc0504669 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:41=
 8
 #2  0xc0504820 in panic (fmt=3DVariable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:572
 #3  0xc04f5a67 in _lockmgr (lkp=3D0xc18ff6b8, flags=3D6, interlkp=3D0xc18ff6=
 e8,
      td=3D0xc0fe0440, file=3D0x0, line=3D0) at /usr/src/sys/kern/kern_lock.c=
 :414
 #4  0xc055e519 in vop_stdunlock (ap=3D0xc7b6ac1c)
      at /usr/src/sys/kern/vfs_default.c:280
 #5  0xc065fecb in VOP_UNLOCK_APV (vop=3D0xc06b5dc0, a=3D0xc7b6ac1c)
      at vnode_if.c:1667
 #6  0xc056a350 in vput (vp=3D0xc18ff660) at vnode_if.h:877
 #7  0xc06377cd in vm_pageout () at /usr/src/sys/vm/vm_pageout.c:1025
 #8  0xc04e8d7b in fork_exit (callout=3D0xc0636a30 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6ad38) at /usr/src/sys/kern/kern_fork.c:783
 #9  0xc0646ab0 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:2=
 05
 (kgdb) list *0xc055e519
 0xc055e519 is in vop_stdunlock (/usr/src/sys/kern/vfs_default.c:282).
 277     {
 278             struct vnode *vp =3D ap->a_vp;
 279
 280             return (lockmgr(vp->v_vnlock, ap->a_flags | =20
 LK_RELEASE, VI_MTX(vp),
 281                 ap->a_td));
 282     }
 283
 284     /* See above. */
 285     int
 286     vop_stdislocked(ap)
 
 We are now at Dump15 but it's just more of the same...
 
 
 ----------------------------------------------------------------
 cakebox.homeunix.net - all the machine one needs..

From: Alexander <freebsd@nagilum.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
	net4801 (sys/vm/vm_pageout.c)
Date: Sun, 10 Aug 2008 18:06:08 +0200

 I updated to the 7.0-STABLE August 1st version, still see the same crashes.
 I have confirmed however that the crashes are related to activity on  
 the gjournal+geli volume.
 /dev/ad0s1d.eli.journal on /export/home (ufs, asynchronous, NFS  
 exported, local, gjournal)
 
 
 ----------------------------------------------------------------
 cakebox.homeunix.net - all the machine one needs..

From: Nagilum <nagilum@nagilum.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
	net4801 (sys/vm/vm_pageout.c)
Date: Thu, 16 Oct 2008 13:49:03 +0200

 Updated to 7.1-PRERELEASE and installed a HiFn adapter, still the same issue=
 :
 
 root@cakebox /var/crash > cat info.0
 Dump header from device /dev/ad0s1b
    Architecture: i386
    Architecture Version: 2
    Dump Length: 58413056B (55 MB)
    Blocksize: 512
    Dumptime: Mon Oct 13 17:29:15 2008
    Hostname: cakebox.tis
    Magic: FreeBSD Kernel Dump
    Version String: FreeBSD 7.1-PRERELEASE #2: Fri Oct 10 14:45:18 CEST 2008
      root@cakebox.tis:/usr/obj/export/src/sys/cakebox
    Panic String: lockmgr: thread 0xc0fe8230, not exclusive lock holder =20
 0xc0fe7460 unlocking
    Dump Parity: 504547331
    Bounds: 0
    Dump Status: good
 root@cakebox /var/crash > kgdb -v /boot/kernel/kernel /var/crash/vmcore.0
 kgdb: core file: /var/crash/vmcore.0
 kgdb: kernel image: /boot/kernel/kernel
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions=
 .
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: lockmgr: thread 0xc0fe8230, not exclusive lock holder =20
 0xc0fe7460 unlocking
 Uptime: 17h19m43s
 Physical memory: 123 MB
 Dumping 55 MB: 40 24 8
 
 Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from =20
 /boot/kernel/ng_ether.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ng_ether.ko
 Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from =20
 /boot/kernel/accf_http.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/accf_http.ko
 Reading symbols from /boot/kernel/wlan_scan_sta.ko...Reading symbols =20
 from /boot/kernel/wlan_scan_sta.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/wlan_scan_sta.ko
 Reading symbols from /boot/kernel/wlan_ccmp.ko...Reading symbols from =20
 /boot/kernel/wlan_ccmp.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/wlan_ccmp.ko
 Reading symbols from /boot/kernel/wlan_tkip.ko...Reading symbols from =20
 /boot/kernel/wlan_tkip.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/wlan_tkip.ko
 Reading symbols from /boot/kernel/nfslockd.ko...Reading symbols from =20
 /boot/kernel/nfslockd.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/nfslockd.ko
 Reading symbols from /boot/kernel/nfsclient.ko...Reading symbols from =20
 /boot/kernel/nfsclient.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/nfsclient.ko
 Reading symbols from /boot/kernel/krpc.ko...Reading symbols from =20
 /boot/kernel/krpc.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/krpc.ko
 Reading symbols from /boot/kernel/geom_journal.ko...Reading symbols =20
 from /boot/kernel/geom_journal.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_journal.ko
 Reading symbols from /boot/kernel/nullfs.ko...Reading symbols from =20
 /boot/kernel/nullfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/nullfs.ko
 #0  doadump () at pcpu.h:196
 196     pcpu.h: No such file or directory.
          in pcpu.h
 (kgdb) bt
 #0  doadump () at pcpu.h:196
 #1  0xc052a659 in boot (howto=3D260) at /export/src/sys/kern/kern_shutdown.c=
 :418
 #2  0xc052a817 in panic (fmt=3DVariable "fmt" is not available.
 ) at /export/src/sys/kern/kern_shutdown.c:574
 #3  0xc051a3d7 in _lockmgr (lkp=3D0xc1758e5c, flags=3D6, interlkp=3D0xc1758e=
 8c,
      td=3D0xc0fe8230, file=3D0x0, line=3D0) at /export/src/sys/kern/kern_loc=
 k.c:414
 #4  0xc0586449 in vop_stdunlock (ap=3D0xc7b6dc14)
      at /export/src/sys/kern/vfs_default.c:319
 #5  0xc06b800b in VOP_UNLOCK_APV (vop=3D0xc0719560, a=3D0xc7b6dc14)
      at vnode_if.c:1667
 #6  0xc0592460 in vput (vp=3D0xc1758e04) at vnode_if.h:877
 #7  0xc068a99d in vm_pageout () at /export/src/sys/vm/vm_pageout.c:1028
 #8  0xc050d2ab in fork_exit (callout=3D0xc0689bd0 <vm_pageout>, arg=3D0x0,
      frame=3D0xc7b6dd38) at /export/src/sys/kern/kern_fork.c:804
 #9  0xc069e4a0 in fork_trampoline ()
      at /export/src/sys/i386/i386/exception.s:264
 
 current dmesg:
 Copyright (c) 1992-2008 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
          The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 7.1-PRERELEASE #2: Fri Oct 10 14:45:18 CEST 2008
      root@cakebox.tis:/usr/obj/export/src/sys/cakebox
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: Geode(TM) Integrated Processor by National Semi (266.66-MHz =20
 586-class CPU)
    Origin =3D "Geode by NSC"  Id =3D 0x540  Stepping =3D 0
    Features=3D0x808131<FPU,TSC,MSR,CX8,CMOV,MMX>
 real memory  =3D 134217728 (128 MB)
 avail memory =3D 126013440 (120 MB)
 cryptosoft0: <software crypto> on motherboard
 pcib0: <Host to PCI bridge> pcibus 0 on motherboard
 pci0: <PCI bus> on pcib0
 sis0: <NatSemi DP8381[56] 10/100BaseTX> port 0xe100-0xe1ff mem =20
 0xa0000000-0xa000
 0fff irq 10 at device 6.0 on pci0
 sis0: Silicon Revision: DP83816A
 miibus0: <MII bus> on sis0
 nsphyter0: <DP83815 10/100 media interface> PHY 0 on miibus0
 nsphyter0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 sis0: Ethernet address: 00:00:24:c1:4f:c4
 sis0: [ITHREAD]
 sis1: <NatSemi DP8381[56] 10/100BaseTX> port 0xe200-0xe2ff mem =20
 0xa0001000-0xa000
 1fff irq 10 at device 7.0 on pci0
 sis1: Silicon Revision: DP83816A
 miibus1: <MII bus> on sis1
 nsphyter1: <DP83815 10/100 media interface> PHY 0 on miibus1
 nsphyter1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 sis1: Ethernet address: 00:00:24:c1:4f:c5
 sis1: [ITHREAD]
 sis2: <NatSemi DP8381[56] 10/100BaseTX> port 0xe300-0xe3ff mem =20
 0xa0002000-0xa000
 2fff irq 10 at device 8.0 on pci0
 sis2: Silicon Revision: DP83816A
 miibus2: <MII bus> on sis2
 nsphyter2: <DP83815 10/100 media interface> PHY 0 on miibus2
 nsphyter2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 sis2: Ethernet address: 00:00:24:c1:4f:c6
 sis2: [ITHREAD]
 hifn0 mem =20
 0xa0003000-0xa0003fff,0xa0004000-0xa0005fff,0xa0008000-0xa000ffff irq
 11 at device 14.0 on pci0
 hifn0: [ITHREAD]
 hifn0: Hifn 7955, rev 0, 32KB dram, pll=3D0x801<ext clk, 4x mult>
 Geode GPIO@ =3D 6100
 Geode Soekris net4801 comBIOS ver. 1.33 20080103 Copyright (C) 2000-2007
 isab0: <PCI-ISA bridge> port 0x6100-0x613f,0x6200-0x623f at device =20
 18.0 on pci0
 isa0: <ISA bus> on isab0
 pci0: <bridge> at device 18.1 (no driver attached)
 atapci0: <National Geode SC1100 ATA33 controller> port =20
 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe000-0xe00f at device 18.2 on pci0
 ata0: <ATA channel 0> on atapci0
 ata0: [ITHREAD]
 ata1: <ATA channel 1> on atapci0
 ata1: [ITHREAD]
 Geode CBA@ 0x6000
 Geode rev: 06 03
 Timecounter "Geode" frequency 27000000 Hz quality 1000
 pci0: <bridge> at device 18.5 (no driver attached)
 ohci0: <OHCI (generic) USB controller> mem 0xa0010000-0xa0010fff irq 5 =20
 at device 19.0 on pci0
 ohci0: [GIANT-LOCKED]
 ohci0: [ITHREAD]
 usb0: OHCI version 1.0, legacy support
 usb0: <OHCI (generic) USB controller> on ohci0
 usb0: USB revision 1.0
 uhub0: <(0x0e11) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
 uhub0: 3 ports with 3 removable, self powered
 cpu0 on motherboard
 pmtimer0 on isa0
 orm0: <ISA Option ROM> at iomem 0xc8000-0xd0fff pnpid ORM0000 on isa0
 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 atkbd0: [GIANT-LOCKED]
 atkbd0: [ITHREAD]
 sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
 sio0: type 16550A, console
 sio0: [FILTER]
 sio1 at port 0x2f8-0x2ff irq 3 on isa0
 sio1: type 16550A
 sio1: [FILTER]
 rum0: <Ralink 802.11 bg WLAN, class 0/0, rev 2.00/0.01, addr 2> on uhub0
 rum0: MAC/BBP RT2573 (rev 0x2573a), RF RT2528
 rum0: WARNING: using obsoleted IFF_NEEDSGIANT flag
 rum0: Ethernet address: 00:1f:c6:20:c9:29
 Timecounters tick every 1.000 msec
 ad0: 152627MB <SAMSUNG HM160HC LQ100-10> at ata0-master PIO4
 Trying to mount root from ufs:/dev/ad0s1a
 WARNING: / was not properly dismounted
 /: mount pending error: blocks 4 files 1
 Accounting enabled
 sis1: promiscuous mode enabled
 sis2: promiscuous mode enabled
 GEOM_ELI: Device ad0s1d.eli created.
 GEOM_ELI: Encryption: AES-CBC 256
 GEOM_ELI:     Crypto: hardware
 GEOM_JOURNAL: Journal 2989812231: ad0s1d.eli contains data.
 GEOM_JOURNAL: Journal 2989812231: ad0s1d.eli contains journal.
 GEOM_JOURNAL: Journal ad0s1d.eli consistent.
 
 
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 #    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
 #   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
 #  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
 # /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
 #           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
 
 
 ----------------------------------------------------------------
 cakebox.homeunix.net - all the machine one needs..

From: Tor Egge <Tor.Egge@cvsup.no.freebsd.org>
To: nagilum@nagilum.org
Cc: FreeBSD-gnats-submit@freebsd.org, kib@freebsd.org, alc@freebsd.org
Subject: Re: i386/123768: [panic] [vm] 7.0-STABLE locking issue on Soekris
 net4801 (sys/vm/vm_pageout.c)
Date: Tue, 21 Oct 2008 01:51:01 +0000 (UTC)

 ----Next_Part(Tue_Oct_21_01_51_01_2008_475)--
 Content-Type: Text/Plain; charset=us-ascii
 Content-Transfer-Encoding: 7bit
 
 
 The semantics of vn_start_write() doesn't match what is expected from it
 in vm_pageout_scan().
 
 If vn_start_write() is called with V_NOWAIT or PCATCH flags and the file system
 is suspending or suspended then it can return an error without clearing *mpp.
 vm_pageout_scan() expects it to be cleared on all errors.
 
 Either mp should be set to NULL in vm_pageout.c after a vn_start_write()
 failure, or vn_start_write() should be changed to clear *mpp on all failures
 where vp is not NULL.
 
 - Tor Egge
 
 ----Next_Part(Tue_Oct_21_01_51_01_2008_475)--
 Content-Type: Text/Plain; charset=us-ascii
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline; filename="vnops2.diff"
 
 Index: vfs_vnops.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/kern/vfs_vnops.c,v
 retrieving revision 1.267
 diff -u -r1.267 vfs_vnops.c
 --- vfs_vnops.c	20 Sep 2008 19:48:24 -0000	1.267
 +++ vfs_vnops.c	21 Oct 2008 01:46:38 -0000
 @@ -963,12 +963,17 @@
  		while ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0) {
  			if (flags & V_NOWAIT) {
  				error = EWOULDBLOCK;
 +				if (vp != NULL)
 +					*mpp = NULL;
  				goto unlock;
  			}
  			error = msleep(&mp->mnt_flag, MNT_MTX(mp),
  			    (PUSER - 1) | (flags & PCATCH), "suspfs", 0);
 -			if (error)
 +			if (error) {
 +				if (vp != NULL)
 +					*mpp = NULL;
  				goto unlock;
 +			}
  		}
  	}
  	if (flags & V_XSLEEP)
 @@ -1024,6 +1029,8 @@
  	if (flags & V_NOWAIT) {
  		MNT_REL(mp);
  		MNT_IUNLOCK(mp);
 +		if (vp != NULL)
 +			*mpp = NULL;
  		return (EWOULDBLOCK);
  	}
  	/*
 @@ -1034,6 +1041,8 @@
  	vfs_rel(mp);
  	if (error == 0)
  		goto retry;
 +	if (vp != NULL)
 +		*mpp = NULL;
  	return (error);
  }
  
 
 ----Next_Part(Tue_Oct_21_01_51_01_2008_475)----
State-Changed-From-To: open->analyzed 
State-Changed-By: kib 
State-Changed-When: Tue Oct 21 09:35:51 UTC 2008 
State-Changed-Why:  
Take, I am going to commit Tor' patch. 


Responsible-Changed-From-To: freebsd-i386->kib 
Responsible-Changed-By: kib 
Responsible-Changed-When: Tue Oct 21 09:35:51 UTC 2008 
Responsible-Changed-Why:  
Take, I am going to commit Tor' patch. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=123768 
State-Changed-From-To: analyzed->closed 
State-Changed-By: kib 
State-Changed-When: Sun Nov 23 16:09:14 UTC 2008 
State-Changed-Why:  
Modified patch committed to HEAD and RELENG_7 (185012 and 185210). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=123768 
>Unformatted:
