From vampiro@rootshell.ru  Sat Aug 31 21:17:30 2002
Return-Path: <vampiro@rootshell.ru>
Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id B80FF37B400
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 31 Aug 2002 21:17:30 -0700 (PDT)
Received: from vampiro.rootshell.ru (vampiro.rootshell.ru [195.162.58.222])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 9752743E3B
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 31 Aug 2002 21:17:28 -0700 (PDT)
	(envelope-from vampiro@rootshell.ru)
Received: by vampiro.rootshell.ru (Sendmail for UK-NC (RT11-SJ), from userid 1111)
	id 7C3385430; Sun,  1 Sep 2002 11:17:22 +0700 (OMSST)
Message-Id: <20020901041722.7C3385430@vampiro.rootshell.ru>
Date: Sun,  1 Sep 2002 11:17:22 +0700 (OMSST)
From: El Vampiro <vampiro@rootshell.ru>
Reply-To: El Vampiro <vampiro@rootshell.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         42277
>Category:       kern
>Synopsis:       Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    mckusick
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Aug 31 21:20:01 PDT 2002
>Closed-Date:    Wed Oct 15 19:04:03 PDT 2003
>Last-Modified:  Wed Oct 15 19:04:03 PDT 2003
>Originator:     El Vampiro
>Release:        FreeBSD 4.6-STABLE i386
>Organization:
>Environment:
System: FreeBSD rshb.com.ru 4.6-STABLE FreeBSD 4.6-STABLE #0: Sat Aug 31 13:17:44 OMSST 2002 vampiro@vampiro.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER  i386

Hardware: new Intel brand server

Kernel:
	machine		i386
	cpu		I586_CPU
	cpu		I686_CPU
	ident		NEWMONSTER
	maxusers	128
	makeoptions	DEBUG=-g
	options		CPU_ENABLE_SSE
	options 	INET
	options 	FFS
	options 	FFS_ROOT
	options		NFS
	options 	SOFTUPDATES
	options 	PROCFS
	options		CD9660
	options 	COMPAT_43
	options 	UCONSOLE
	options 	USERCONFIG
	options 	VISUAL_USERCONFIG
	options 	P1003_1B
	options 	_KPOSIX_PRIORITY_SCHEDULING
	options		_KPOSIX_VERSION=199309L
	options		KTRACE
	options		PERFMON
	options 	SYSVSHM
	options 	SYSVMSG
	options 	SYSVSEM
	options		SHMMAXPGS=8192  # max amount of shared memory pages (4k on i386)
	options		SHMMNI=512      # max shared mem id's per system
	options		SHMSEG=256      # max shared mem id's per process
	options		MSGMNB=8192     # max # of bytes in a queue
	options		MSGMNI=256      # number of message queue identifiers
	options		MSGSEG=256      # number of message segments per queue
	options		MSGSSZ=64       # size of a message segment
	options		MSGTQL=8192     # max messages in system
	options		SEMMAP=256
	options		SEMMNI=256
	options		SEMMNS=512
	options		SEMMNU=256
	options		INCLUDE_CONFIG_FILE
	options		IPFILTER
	options		IPFILTER_LOG
	options         IPFIREWALL
	options         IPFIREWALL_VERBOSE
	options         IPFIREWALL_DEFAULT_TO_ACCEPT
	options		IPFW2
	options		RANDOM_IP_ID
	options		ICMP_BANDLIM
	options		ACCEPT_FILTER_HTTP
	options         VESA
	options         PANIC_REBOOT_WAIT_TIME=20
	options         SMBFS
	options         LIBMCHAIN
	options         LIBICONV
	options         NETSMB
	options         NETSMBCRYPTO
	options         UFS_DIRHASH
	options         SHOW_BUSYBUFS
	options		HZ=1000
	device		isa
	device		pci
	device		fdc0	at isa? port IO_FD1 irq 6 drq 2
	device		fd0	at fdc0 drive 0
	device		fd1	at fdc0 drive 1
	device		ata0	at isa? port IO_WD1 irq 14
	device		ata1	at isa? port IO_WD2 irq 15
	device		ata
	device		atadisk
	device          atapicd
	options 	ATA_STATIC_ID
	device          ahc
	device          aic0    at isa?
	device          scbus
	device          da
	device          sa
	device          cd
	device          pass
	device		atkbdc0	at isa? port IO_KBD
	device		atkbd0	at atkbdc? irq 1
	device		vga0	at isa?
	device		sc0	at isa? flags 0x100
	device		npx0	at nexus? port IO_NPX irq 13
	device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
	device		sio1	at isa? port IO_COM2 irq 3
	device 		miibus
	device 		dc
	device 		fxp
	pseudo-device	loop
	pseudo-device	ether
	pseudo-device	tun
	pseudo-device	pty
	pseudo-device	bpf	8
	pseudo-device	vn	2
	pseudo-device   gzip
	pseudo-device   splash
	device          smbus
	device          intpm
	device          alpm
	device          ichsmb
	device          viapm
	device          smb
	device          iicbus
	device          iicbb
	device          ic
	device          iic
	device          iicsmb
	device		apm0 at nexus?

Copyright (c) 1992-2002 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 4.6-STABLE #0: Sat Aug 31 13:17:44 OMSST 2002
    vampiro@vampiro.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER
Timecounter "i8254"  frequency 1193182 Hz
CPU: Pentium III/Pentium III Xeon/Celeron (999.72-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x68a  Stepping = 10
  Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
real memory  = 268369920 (262080K bytes)
avail memory = 257720320 (251680K bytes)
Preloaded elf kernel "kernel" at 0xc0360000.
VESA: v2.0, 4096k memory, flags:0x0, mode table:0xc02fe6c2 (1000022)
VESA: ATI MACH64
netsmb_dev: loaded
Pentium Pro MTRR support enabled
Using $PIR table, 268435454 entries at 0xc00fdf10
apm: protected mode connections are not supported
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GV graphics accelerator> at 2.0 irq 11
fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x5800-0x583f mem 0xfb000000-0xfb0fffff,0xfb101000-0xfb101fff irq 10 at device 3.0 on pci0
fxp0: Ethernet address 00:d0:b7:b8:ab:64
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
dc0: <Davicom DM9102A 10/100BaseTX> port 0x5400-0x54ff mem 0xfb102000-0xfb1020ff irq 7 at device 7.0 on pci0
dc0: Ethernet address: 00:80:ad:08:12:2b
miibus1: <MII bus> on dc0
ukphy0: <Generic IEEE 802.3u media interface> on miibus1
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 ATA33 controller> port 0x5840-0x584f,0x374-0x377,0x170-0x177 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
pci1: <PCI bus> on pcib1
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6000-0x60ff mem 0xfd000000-0xfd000fff irq 9 at device 4.0 on pci1
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6400-0x64ff mem 0xfd001000-0xfd001fff irq 5 at device 4.1 on pci1
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcf7ff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 on isa0
sio1: type 16550A
ipfw2 initialized, divert disabled, rule-based forwarding enabled, default to accept, logging unlimited
IP Filter: v3.4.27 initialized.  Default = pass all, Logging = enabled
acd0: CDROM <SONY CD-ROM CDU5221> at ata0-master PIO4
Waiting 2 seconds for SCSI devices to settle
sa0 at ahc1 bus 0 target 0 lun 0
sa0: <ARCHIVE Python 04106-XXX 7550> Removable Sequential Access SCSI-2 device 
sa0: 10.000MB/s transfers (10.000MHz, offset 15)
da0 at ahc0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device 
da0: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
da0: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
da1 at ahc0 bus 0 target 1 lun 0
da1: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device 
da1: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
da1: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
Mounting root from ufs:/dev/da0s1a

>Description:
	Several kernel panics per day with panicstr: softdep_lock: locki ng against myself
	Upon reboot fsck discovers many filesystem errors. File damage occurs often.
	$ gdb -k kernel.debug vmcore.0 
	GNU gdb 4.18 (FreeBSD)
	Copyright 1998 Free Software Foundation, Inc.
	GDB is free software, covered by the GNU General Public License, and you are
	welcome to change it and/or distribute copies of it under certain conditions.
	Type "show copying" to see the conditions.
	There is absolutely no warranty for GDB.  Type "show warranty" for details.
	This GDB was configured as "i386-unknown-freebsd"...

	/big1/vampiro/crashes/4-NEW/vmcore.0: Permission denied.
	(kgdb) quit
	$ gdb -k kernel.debug vmcore.0 
	GNU gdb 4.18 (FreeBSD)
	Copyright 1998 Free Software Foundation, Inc.
	GDB is free software, covered by the GNU General Public License, and you are
	welcome to change it and/or distribute copies of it under certain conditions.
	Type "show copying" to see the conditions.
	There is absolutely no warranty for GDB.  Type "show warranty" for details.
	This GDB was configured as "i386-unknown-freebsd"...
	IdlePTD at phsyical address 0x0037f000
	initial pcb at physical address 0x002ea800
	panicstr: softdep_lock: locking against myself
	panic messages:
	---
	Fatal trap 12: page fault while in kernel mode
	fault virtual address   = 0xffff000a
	fault code              = supervisor read, page not present
	instruction pointer     = 0x8:0xc022988c
	stack pointer           = 0x10:0xcd0f9d10
	frame pointer           = 0x10:0xcd0f9d10
	code segment            = base 0x0, limit 0xfffff, type 0x1b
				= DPL 0, pres 1, def32 1, gran 1
	processor eflags        = interrupt enabled, resume, IOPL = 0
	current process         = 6 (syncer)
	interrupt mask          = bio 
	trap number             = 12
	panic: page fault

	syncing disks... panic: softdep_lock: locking against myself
	Uptime: 3h29m7s

	dumping to dev #da/0x20009, offset 128
	dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
	---
	#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
	487             if (dumping++) {
	(kgdb) where
	#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
	#1  0xc01888d3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
	#2  0xc0188cf8 in poweroff_wait (junk=0xc02ae0c0, howto=-1053066752)
	    at /usr/src/sys/kern/kern_shutdown.c:595
	#3  0xc02295be in acquire_lock (lk=0xc02d8c7c)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
	#4  0xc022d6d8 in softdep_update_inodeblock (ip=0xc13b7a00, bp=0xc64ec464, 
	    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
	#5  0xc022870d in ffs_update (vp=0xcdbcefc0, waitfor=0)
	    at /usr/src/sys/ufs/ffs/ffs_inode.c:106
	#6  0xc0232105 in ffs_fsync (ap=0xcd0f9bb0)
	    at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
	#7  0xc02309e3 in ffs_sync (mp=0xc12a1a00, waitfor=2, cred=0xc0a3c880, 
	    p=0xc02ff300) at vnode_if.h:558
	#8  0xc01b8f97 in sync (p=0xc02ff300, uap=0x0)
	    at /usr/src/sys/kern/vfs_syscalls.c:576
	#9  0xc0188646 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
	#10 0xc0188cf8 in poweroff_wait (junk=0xc02b868c, howto=-1070890577)
	    at /usr/src/sys/kern/kern_shutdown.c:595
	#11 0xc0274866 in trap_fatal (frame=0xcd0f9cd0, eva=4294901770)
	    at /usr/src/sys/i386/i386/trap.c:974
	#12 0xc0274539 in trap_pfault (frame=0xcd0f9cd0, usermode=0, eva=4294901770)
	    at /usr/src/sys/i386/i386/trap.c:867
	#13 0xc02740f7 in trap (frame={tf_fs = -854654960, tf_es = -1053884400, 
	      tf_ds = 16, tf_edi = 0, tf_esi = -1053851648, tf_ebp = -854614768, 
	      tf_isp = -854614788, tf_ebx = -65536, tf_edx = 0, tf_ecx = -65536, 
	      tf_eax = -1051831424, tf_trapno = 12, tf_err = 0, tf_eip = -1071474548, 
	      tf_cs = 8, tf_eflags = 66067, tf_esp = -854614736, tf_ss = -1071458418})
	    at /usr/src/sys/i386/i386/trap.c:466
	#14 0xc022988c in worklist_remove (item=0xffff0000)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
	#15 0xc022d78e in softdep_update_inodeblock (ip=0xc12f8000, bp=0xc651ab80, 
	    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
	#16 0xc022870d in ffs_update (vp=0xcdd565c0, waitfor=0)
	    at /usr/src/sys/ufs/ffs/ffs_inode.c:106
	#17 0xc02289f1 in ffs_truncate (vp=0xcdd565c0, length=0, flags=0, cred=0x0, 
	    p=0xcc00a5e0) at /usr/src/sys/ufs/ffs/ffs_inode.c:201
	#18 0xc0232e08 in ufs_inactive (ap=0xcd0f9ed8)
	    at /usr/src/sys/ufs/ufs/ufs_inode.c:89
	#19 0xc0238301 in ufs_vnoperate (ap=0xcd0f9ed8)
	    at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
	#20 0xc01b70e8 in vput (vp=0xcdd565c0) at vnode_if.h:815
	#21 0xc022c594 in handle_workitem_remove (dirrem=0xc14034e0)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
	#22 0xc0229c0d in process_worklist_item (matchmnt=0x0, flags=0)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
	#23 0xc0229ab2 in softdep_process_worklist (matchmnt=0x0)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
	#24 0xc01b6a0f in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
	(kgdb) up 14   
	#14 0xc022988c in worklist_remove (item=0xffff0000)
	    at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
	467                     panic("worklist_remove: lock not held");
	(kgdb) l
	462     worklist_remove(item)
	463             struct worklist *item;
	464     {
	465
	466             if (lk.lkt_held == -1)
	467                     panic("worklist_remove: lock not held");
	468             if ((item->wk_state & ONWORKLIST) == 0) {
	469                     FREE_LOCK(&lk);
	470                     panic("worklist_remove: not on list");
	471             }
	(kgdb) p item
	$1 = (struct worklist *) 0x0
	(kgdb) p lk
	$2 = {lkt_spl = 0, lkt_held = -1}
	(kgdb) up
	#15 0xc022d78e in softdep_update_inodeblock (ip=0xc12f8000, bp=0xc651ab80, 
	    waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
	3847                    WORKLIST_REMOVE(wk);
	(kgdb) l
	3842             * operations dependent on the inode being written to disk
	3843             * can be moved to the id_bufwait so that they will be
	3844             * processed when the buffer I/O completes.
	3845             */
	3846            while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
	3847                    WORKLIST_REMOVE(wk);
	3848                    WORKLIST_INSERT(&inodedep->id_bufwait, wk);
	3849            }
	3850            /*
	3851             * Newly allocated inodes cannot be written until the bitmap
	(kgdb) p wk
	$3 = (struct worklist *) 0x68c040
	(kgdb) p inodedep
	$4 = (struct inodedep *) 0xc14e5380
	(kgdb) p inodedep->id_inowait
	$5 = {lh_first = 0xffff0000}
	(kgdb) p inodedep->id_bufwait
	$6 = {lh_first = 0x0}

	I can provide more info upon request.
	
>How-To-Repeat:
	I found no way to repeat this. System can panic even upon minimal activity and can work upon hard load.	
>Fix:
	The only way I found is turn softupdates off

>Release-Note:
>Audit-Trail:

From: El Vampiro <vampiro@rootshell.ru>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Mon, 9 Sep 2002 09:25:46 +0700

 Two more panisc during holidays.
 First:
 
 IdlePTD at phsyical address 0x003f0000
 initial pcb at physical address 0x0035e080
 panicstr: softdep_lock: locking against myself
 panic messages:
 ---
 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0xffff000a
 fault code              = supervisor read, page not present
 instruction pointer     = 0x8:0xc022f84c
 stack pointer           = 0x10:0xcd0c6d10
 frame pointer           = 0x10:0xcd0c6d10
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 6 (syncer)
 interrupt mask          = bio 
 trap number             = 12
 panic: page fault
 
 syncing disks... panic: softdep_lock: locking against myself
 Uptime: 1d2h45m48s
 
 dumping to dev #da/0x20009, offset 128
 dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
 ---
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 487             if (dumping++) {
 (kgdb) where
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 #1  0xc0191837 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
 #2  0xc0191c5c in poweroff_wait (junk=0xc03224c0, howto=-1052082688) at /usr/src/sys/kern/kern_shutdown.c:595
 #3  0xc022f57e in acquire_lock (lk=0xc034c6bc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
 #4  0xc0233698 in softdep_update_inodeblock (ip=0xc14a7e00, bp=0xc64dce38, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
 #5  0xc022e6cd in ffs_update (vp=0xcd869980, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
 #6  0xc02380c5 in ffs_fsync (ap=0xcd0c6bb0) at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
 #7  0xc02369a3 in ffs_sync (mp=0xc12a7200, waitfor=2, cred=0xc0a42680, p=0xc0372b60) at vnode_if.h:558
 #8  0xc01c19db in sync (p=0xc0372b60, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
 #9  0xc01915aa in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
 #10 0xc0191c5c in poweroff_wait (junk=0xc032c60c, howto=-1070415569) at /usr/src/sys/kern/kern_shutdown.c:595
 #11 0xc0278df6 in trap_fatal (frame=0xcd0c6cd0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
 #12 0xc0278ac9 in trap_pfault (frame=0xcd0c6cd0, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867
 #13 0xc02786b3 in trap (frame={tf_fs = -854851568, tf_es = -1052377072, tf_ds = 16, tf_edi = 0, tf_esi = -1052351744, 
       tf_ebp = -854823664, tf_isp = -854823684, tf_ebx = -65536, tf_edx = -1051899520, tf_ecx = -65536, 
       tf_eax = -1051899520, tf_trapno = 12, tf_err = 0, tf_eip = -1071450036, tf_cs = 8, tf_eflags = 66067, 
       tf_esp = -854823632, tf_ss = -1071433906}) at /usr/src/sys/i386/i386/trap.c:466
 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 #15 0xc023374e in softdep_update_inodeblock (ip=0xc1466300, bp=0xc64cde4c, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 #16 0xc022e6cd in ffs_update (vp=0xcdde9200, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
 #17 0xc022e9b1 in ffs_truncate (vp=0xcdde9200, length=0, flags=0, cred=0x0, p=0xcbfd75e0)
     at /usr/src/sys/ufs/ffs/ffs_inode.c:201
 #18 0xc0238dc8 in ufs_inactive (ap=0xcd0c6ed8) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
 #19 0xc023e2c1 in ufs_vnoperate (ap=0xcd0c6ed8) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
 #20 0xc01bfb2c in vput (vp=0xcdde9200) at vnode_if.h:815
 #21 0xc0232554 in handle_workitem_remove (dirrem=0xc151a960) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
 #22 0xc022fbcd in process_worklist_item (matchmnt=0x0, flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
 #23 0xc022fa72 in softdep_process_worklist (matchmnt=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
 #24 0xc01bf453 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
 (kgdb) up 14
 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 467                     panic("worklist_remove: lock not held");
 (kgdb) l
 462     worklist_remove(item)
 463             struct worklist *item;
 464     {
 465
 466             if (lk.lkt_held == -1)
 467                     panic("worklist_remove: lock not held");
 468             if ((item->wk_state & ONWORKLIST) == 0) {
 469                     FREE_LOCK(&lk);
 470                     panic("worklist_remove: not on list");
 471             }
 (kgdb) p item
 $1 = (struct worklist *) 0x0
 (kgdb) p lk
 $2 = {lkt_spl = 0, lkt_held = -1}
 (kgdb) up
 #15 0xc023374e in softdep_update_inodeblock (ip=0xc1466300, bp=0xc64cde4c, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 3847                    WORKLIST_REMOVE(wk);
 (kgdb) l
 3842             * operations dependent on the inode being written to disk
 3843             * can be moved to the id_bufwait so that they will be
 3844             * processed when the buffer I/O completes.
 3845             */
 3846            while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
 3847                    WORKLIST_REMOVE(wk);
 3848                    WORKLIST_INSERT(&inodedep->id_bufwait, wk);
 3849            }
 3850            /*
 3851             * Newly allocated inodes cannot be written until the bitmap
 (kgdb) p wk
 $3 = (struct worklist *) 0x68c040
 (kgdb) p *wk
 Cannot access memory at address 0x68c040.
 (kgdb) p *inodedep
 $4 = {id_list = {wk_list = {le_next = 0xc1943000, le_prev = 0xc64cdf8c}, wk_type = 1, wk_state = 32777}, id_hash = {
     le_next = 0x0, le_prev = 0xc1222e94}, id_fs = 0xc12c7000, id_ino = 22082, id_nlinkdelta = 0, id_savedino = 0x0, 
   id_deps = {le_next = 0x0, le_prev = 0x0}, id_buf = 0x0, id_savedsize = -1, id_pendinghd = {lh_first = 0x0}, id_bufwait = {
     lh_first = 0xc14d4c00}, id_inowait = {lh_first = 0xffff0000}, id_inoupdt = {tqh_first = 0x0, tqh_last = 0xc14d49c4}, 
   id_newinoupdt = {tqh_first = 0x0, tqh_last = 0xc14d49cc}}
 
 # vmstat -m -M vmcore.2 
 Memory statistics by bucket size
 Size   In Use   Free   Requests  HighWater  Couldfree
   16      952    328     874701    1280          0
   32      536    360    4176708     640          0
   64    16355   5917   31821882     320       9082
  128     1247    513    6295455     160      59721
  256    17477    779    4662394      80        635
  512      443    213      18611      40          0
   1K      102     18    3986474      20         19
   2K       42    138       2378      10       1463
   4K       30      2     658022       5          0
   8K        3      1         40       5          0
  16K        5      0        245       5          0
  32K        8      0         16       5          0
  64K       10      0         10       5          0
 128K        3      0          4       5          0
 
 Memory usage type by bucket size
 Size  Type(s)
   16  uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl,
           ether_multi, BPF, vnodes, mount, pcb, soname, accf, shm, rman, bus,
           sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld
   32  atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag,
           indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl,
           ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname,
           accf, taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl,
           uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld
   64  isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon,
           NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes,
           vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc,
           session, temp, devbuf, lockf, proc-args, file
  128  ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF,
           vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred,
           temp, devbuf, zombie, proc-args, dev_t, timecounter, kld
  256  UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon,
           NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes,
           ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue,
           file desc
  512  UFS dirhash, UFS mount, NFS daemon, BPF, mount, BIO buffer, ptys,
           msg, bus, uidinfo, ATA generic, temp, devbuf, prison, file desc
   1K  uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, ioctlops, bus,
           temp, devbuf, kqueue, file desc
   2K  UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver, shm,
           ioctlops, bus, proc, devbuf, file desc
   4K  memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus,
           temp, devbuf
   8K  pagedep, syncache, shm, bus, temp
  16K  VM pgdata, indirdep, sem, msg, temp, devbuf
  32K  BPF, sem, msg, temp, devbuf
  64K  ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf
 128K  vfscache, msg, SWAP
 
 Memory statistics by type                          Type  Kern
         Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
      atkbddev     1     1K      1K 42378K        1    0     0  32
    uc_devlist    14     2K      2K 42378K       14    0     0  16,1K
       memdesc     1     4K      4K 42378K        1    0     0  4K
          mbuf     1     4K      4K 42378K        1    0     0  4K
        isadev    10     1K      1K 42378K       10    0     0  64
          ZONE    16     2K      2K 42378K       16    0     0  128
     VM pgdata     1    16K     16K 42378K        1    0     0  16K
   ISOFS mount     1    64K     64K 42378K        1    0     0  64K
   UFS dirhash   776   153K    264K 42378K     2226    0     0  16,32,64,128,256,512,1K,2K,4K
     UFS mount    21    46K     46K 42378K       39    0     0  512,2K,4K
     UFS ihash     1    64K     64K 42378K        1    0     0  64K
      FFS node 17135  4284K   4485K 42378K   520818    0     0  256
        dirrem     2     1K      3K 42378K    25912    0     0  32
         mkdir     0     0K      1K 42378K      124    0     0  32
        diradd     0     0K      3K 42378K    26553    0     0  32
      freefile     0     0K      2K 42378K    13804    0     0  32
      freeblks     1     1K      5K 42378K    12336    0     0  128
      freefrag     0     0K      3K 42378K     5170    0     0  32
    allocindir     1     1K    152K 42378K    20648    0     0  64
      indirdep     1     1K     33K 42378K     1158    0     0  32,16K
   allocdirect     1     1K      6K 42378K    22316    0     0  64
     bmsafemap     3     1K      1K 42378K    11096    0     0  32
        newblk     1     1K      1K 42378K    42965    0     0  32,256
      inodedep     4    65K     77K 42378K    23254    0     0  128,64K
       pagedep     3     9K     11K 42378K    12282    0     0  64,8K
      p1003.1b     1     1K      1K 42378K        1    0     0  16
      NFS hash     1    64K     64K 42378K        1    0     0  64K
   NQNFS Lease     1     1K      1K 42378K        1    0     0  1K
 NFSV3 srvdesc     0     0K      1K 42378K    62136    0     0  16,256
    NFS daemon    71     8K      8K 42378K       71    0     0  64,256,512
       NFS req     0     0K      1K 42378K  2015016    0     0  64
   NFS srvsock     1     1K      1K 42378K        1    0     0  256
      syncache     1     8K      8K 42378K        1    0     0  8K
     tseg_qent     0     0K      2K 42378K    30284    0     0  32
   IpFw/IpAcct    79    11K     53K 42378K    14699    0     0  64,128,256,1K
      in_multi     3     1K      1K 42378K        3    0     0  32
      routetbl   163    23K    126K 42378K     3651    0     0  16,32,64,128,256
   ether_multi    12     1K      1K 42378K       12    0     0  16,32,64
        ifaddr    27     6K      6K 42378K       27    0     0  32,64,256,2K
           BPF    15   137K    266K 42378K       31    0     0  16,32,128,512,4K,32K
   Export Host     6     2K      2K 42378K        6    0     0  256
        vnodes    26     6K      6K 42378K      337    0     0  16,32,64,128,256
         mount    16     8K      8K 42378K       24    0     0  16,128,512
 cluster_save buffer     0     0K      1K 42378K     1521    0     0  32
      vfscache 15065  1073K   1332K 42378K   549206    0     0  64,128,128K
    BIO buffer     4     8K    286K 42378K     1184    0     0  512,2K
           pcb    69     6K      8K 42378K    14966    0     0  16,32,64,2K
        soname    38     4K      6K 42378K   839397    0     0  16,32,64,128
    ACD driver     2     3K      3K 42378K        2    0     0  256,2K
          accf     3     1K      1K 42378K        3    0     0  16,32
          ptys     5     3K      3K 42378K        5    0     0  512
          ttys   210    27K     47K 42378K     2355    0     0  128,256
           shm    16    37K     51K 42378K     1085    0     0  16,2K,8K
           sem     3    88K     88K 42378K        3    0     0  16K,32K,64K
           msg     4   137K    137K 42378K        4    0     0  512,16K,32K,128K
          rman    58     4K      4K 42378K      401    0     0  16,64
           iov     0     0K      1K 42378K      406    0     0  128
      ioctlops     0     0K      4K 42378K        7    0     0  1K,2K,4K
     taskqueue     1     1K      1K 42378K        1    0     0  32
          SWAP     2    73K    145K 42378K        4    0     0  32,128K
 ATAPI generic     1     1K      1K 42378K        2    0     0  32,128
          kobj     1     4K      4K 42378K        1    0     0  4K
  eventhandler    15     1K      1K 42378K       15    0     0  32,64
           bus   375    34K     37K 42378K      696    0     0  16,32,64,128,256,512,1K,2K,4K,8K
     sysctloid    10     1K      1K 42378K       10    0     0  16,64
        sysctl     0     0K      1K 42378K     4771    0     0  16,32
       uidinfo    11     1K      1K 42378K     1271    0     0  32,512
          cred    67     9K     13K 42378K   715419    0     0  128
       subproc   241    17K     26K 42378K  8001154    0     0  32,64,256
          proc     2     4K      4K 42378K        2    0     0  2K
       session    47     3K      4K 42378K     4512    0     0  64
          pgrp    47     2K      2K 42378K     4620    0     0  32
   ATA generic     2     1K      1K 42378K        2    0     0  16,512
          temp   305   138K    153K 42378K  8754546    0     0  16,32,64,128,256,512,1K,4K,8K,16K,32K,64K
        devbuf   506   412K    413K 42378K     1141    0     0  16,32,64,128,256,512,1K,2K,4K,16K,32K,64K
         lockf    42     3K      6K 42378K  1564032    0     0  64
        prison     3     2K      2K 42378K        3    0     0  512
        atexit     1     1K      1K 42378K        1    0     0  16
        zombie     0     0K      1K 42378K  3997472    0     0  128
     proc-args    81     5K      7K 42378K  4062493    0     0  16,32,64,128,256
        kqueue     5     5K     22K 42378K     7104    0     0  256,1K
         sigio     1     1K      1K 42378K      133    0     0  32
          file   728    46K     67K 42378K 17098401    0     0  64
     file desc   147    40K     62K 42378K  4000832    0     0  256,512,1K,2K
         dev_t   663    83K     83K 42378K      663    0     0  128
   timecounter    10     2K      2K 42378K       10    0     0  128
           kld     4     1K      1K 42378K       35    0     0  16,32,128
 
 Memory Totals:  In Use    Free    Requests
                  7243K   1062K    52496940
 
 
 And second panic:
 
 IdlePTD at phsyical address 0x003f0000
 initial pcb at physical address 0x0035e080
 panicstr: softdep_lock: locking against myself
 panic messages:
 ---
 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0xffff000a
 fault code              = supervisor read, page not present
 instruction pointer     = 0x8:0xc022f84c
 stack pointer           = 0x10:0xcda0ace8
 frame pointer           = 0x10:0xcda0ace8
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 28039 (nmbd)
 interrupt mask          = bio 
 trap number             = 12
 panic: page fault
 
 syncing disks... panic: softdep_lock: locking against myself
 Uptime: 1d5h27m47s
 
 dumping to dev #da/0x20009, offset 128
 dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
 ---
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 487             if (dumping++) {
 (kgdb) where
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 #1  0xc0191837 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
 #2  0xc0191c5c in poweroff_wait (junk=0xc03224c0, howto=-1052660736) at /usr/src/sys/kern/kern_shutdown.c:595
 #3  0xc022f57e in acquire_lock (lk=0xc034c6bc) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
 #4  0xc0233698 in softdep_update_inodeblock (ip=0xc141ac00, bp=0xc65508dc, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3813
 #5  0xc022e6cd in ffs_update (vp=0xcd67e780, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
 #6  0xc02380c5 in ffs_fsync (ap=0xcda0ab88) at /usr/src/sys/ufs/ffs/ffs_vnops.c:273
 #7  0xc02369a3 in ffs_sync (mp=0xc12a7400, waitfor=2, cred=0xc0a42680, p=0xc0372b60) at vnode_if.h:558
 #8  0xc01c19db in sync (p=0xc0372b60, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
 #9  0xc01915aa in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
 #10 0xc0191c5c in poweroff_wait (junk=0xc032c60c, howto=-1070415569) at /usr/src/sys/kern/kern_shutdown.c:595
 #11 0xc0278df6 in trap_fatal (frame=0xcda0aca8, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
 #12 0xc0278ac9 in trap_pfault (frame=0xcda0aca8, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867
 #13 0xc02786b3 in trap (frame={tf_fs = -845152240, tf_es = -1049231344, tf_ds = 16, tf_edi = 0, tf_esi = -1049228288, 
       tf_ebp = -845107992, tf_isp = -845108012, tf_ebx = -65536, tf_edx = -1045960576, tf_ecx = -65536, 
       tf_eax = -1045960576, tf_trapno = 12, tf_err = 0, tf_eip = -1071450036, tf_cs = 8, tf_eflags = 66071, 
       tf_esp = -845107960, tf_ss = -1071433906}) at /usr/src/sys/i386/i386/trap.c:466
 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 #15 0xc023374e in softdep_update_inodeblock (ip=0xc1760c00, bp=0xc6564930, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 #16 0xc022e6cd in ffs_update (vp=0xcdd3a440, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
 #17 0xc022e9b1 in ffs_truncate (vp=0xcdd3a440, length=0, flags=0, cred=0x0, p=0xcd93e8a0)
     at /usr/src/sys/ufs/ffs/ffs_inode.c:201
 #18 0xc0238dc8 in ufs_inactive (ap=0xcda0aeb0) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
 #19 0xc023e2c1 in ufs_vnoperate (ap=0xcda0aeb0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
 #20 0xc01bfb2c in vput (vp=0xcdd3a440) at vnode_if.h:815
 #21 0xc01c2e15 in unlink (p=0xcd93e8a0, uap=0xcda0af80) at /usr/src/sys/kern/vfs_syscalls.c:1504
 #22 0xc027901a in syscall2 (frame={tf_fs = -1078001617, tf_es = 47, tf_ds = -1078001617, tf_edi = 672627448, tf_esi = 0, 
       tf_ebp = -1077937936, tf_isp = -845107244, tf_ebx = 1, tf_edx = 672628888, tf_ecx = 135524352, tf_eax = 10, 
       tf_trapno = 12, tf_err = 2, tf_eip = 672239144, tf_cs = 31, tf_eflags = 647, tf_esp = -1077940028, tf_ss = 47})
     at /usr/src/sys/i386/i386/trap.c:1175
 #23 0xc026cd75 in Xint0x80_syscall ()
 #24 0x805d23c in ?? ()
 #25 0x804b714 in ?? ()
 #26 0x804c40d in ?? ()
 #27 0x804a871 in ?? ()
 (kgdb) up 14
 #14 0xc022f84c in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 467                     panic("worklist_remove: lock not held");
 (kgdb) l
 462     worklist_remove(item)
 463             struct worklist *item;
 464     {
 465
 466             if (lk.lkt_held == -1)
 467                     panic("worklist_remove: lock not held");
 468             if ((item->wk_state & ONWORKLIST) == 0) {
 469                     FREE_LOCK(&lk);
 470                     panic("worklist_remove: not on list");
 471             }
 (kgdb) p *item
 Cannot access memory at address 0x0.
 (kgdb) p lk
 $1 = {lkt_spl = 0, lkt_held = -1}
 (kgdb) up
 #15 0xc023374e in softdep_update_inodeblock (ip=0xc1760c00, bp=0xc6564930, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 3847                    WORKLIST_REMOVE(wk);
 (kgdb) l
 3842             * operations dependent on the inode being written to disk
 3843             * can be moved to the id_bufwait so that they will be
 3844             * processed when the buffer I/O completes.
 3845             */
 3846            while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
 3847                    WORKLIST_REMOVE(wk);
 3848                    WORKLIST_INSERT(&inodedep->id_bufwait, wk);
 3849            }
 3850            /*
 3851             * Newly allocated inodes cannot be written until the bitmap
 (kgdb) p wk
 $2 = (struct worklist *) 0x68c040
 (kgdb) p *wk
 Cannot access memory at address 0x68c040.
 
 # vmstat -m -M vmcore.3
 Memory statistics by bucket size
 Size   In Use   Free   Requests  HighWater  Couldfree
   16     1116    164     424115    1280          0
   32      488    280    4533610     640          0
   64    19590   6586   35473023     320      18958
  128     1244   4100    6899026     160     348464
  256    17591    905    5822574      80       2631
  512      548    124      15562      40        277
   1K      101      7    4403849      20         13
   2K       61     99       2956      10       1304
   4K       30      3     624293       5          0
   8K        3      1         10       5          0
  16K        5      0         85       5          0
  32K        8      0          8       5          0
  64K       10      0         10       5          0
 128K        3      0          4       5          0
  
 Memory usage type by bucket size
 Size  Type(s)
   16  uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl,
           ether_multi, BPF, vnodes, mount, pcb, soname, accf, shm, rman, bus,
           sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld
   32  atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag,
           indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl,
           ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname,
           accf, taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl,
           uidinfo, subproc, pgrp, temp, devbuf, proc-args, sigio, kld
   64  isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon,
           NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes,
           vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc,
           session, temp, devbuf, lockf, proc-args, file
  128  ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF,
           vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred,
           temp, devbuf, zombie, proc-args, dev_t, timecounter, kld
  256  UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon,
           NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes,
           ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue,
           file desc
  512  UFS dirhash, UFS mount, NFS daemon, mount, BIO buffer, ptys, msg,
           ioctlops, bus, uidinfo, ATA generic, temp, devbuf, prison, file desc
   1K  uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, ioctlops, bus,
           temp, devbuf, kqueue
   2K  UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver, shm,
           ioctlops, bus, proc, devbuf
   4K  memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus,
           temp, devbuf
   8K  pagedep, syncache, shm, bus, temp
  16K  VM pgdata, indirdep, sem, msg, devbuf
  32K  BPF, sem, msg, temp, devbuf
  64K  ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf
 128K  vfscache, msg, SWAP
 
 Memory statistics by type                          Type  Kern
         Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
      atkbddev     1     1K      1K 42378K        1    0     0  32
    uc_devlist    14     2K      2K 42378K       14    0     0  16,1K
       memdesc     1     4K      4K 42378K        1    0     0  4K
          mbuf     1     4K      4K 42378K        1    0     0  4K
        isadev    10     1K      1K 42378K       10    0     0  64
          ZONE    16     2K      2K 42378K       16    0     0  128
     VM pgdata     1    16K     16K 42378K        1    0     0  16K
   ISOFS mount     1    64K     64K 42378K        1    0     0  64K
   UFS dirhash  1088   216K    274K 42378K     4095    0     0  16,32,64,128,256,512,1K,2K,4K
     UFS mount    21    46K     46K 42378K       21    0     0  512,2K,4K
     UFS ihash     1    64K     64K 42378K        1    0     0  64K
      FFS node 17283  4321K   4487K 42378K  1282170    0     0  256
        dirrem     0     0K      3K 42378K    14638    0     0  32
         mkdir     0     0K      1K 42378K      130    0     0  32
        diradd     2     1K      3K 42378K    14962    0     0  32
      freefile     0     0K      2K 42378K     8476    0     0  32
      freeblks     1     1K      3K 42378K     6793    0     0  128
      freefrag     0     0K     10K 42378K     3611    0     0  32
    allocindir     0     0K    425K 42378K    32022    0     0  64
      indirdep     0     0K     33K 42378K      410    0     0  32,16K
   allocdirect     1     1K      4K 42378K    12040    0     0  64
     bmsafemap     1     1K      1K 42378K     7652    0     0  32
        newblk     1     1K      1K 42378K    44063    0     0  32,256
      inodedep     4    65K     78K 42378K    13879    0     0  128,64K
       pagedep     2     9K     10K 42378K     8084    0     0  64,8K
      p1003.1b     1     1K      1K 42378K        1    0     0  16
      NFS hash     1    64K     64K 42378K        1    0     0  64K
   NQNFS Lease     1     1K      1K 42378K        1    0     0  1K
 NFSV3 srvdesc     0     0K      1K 42378K        8    0     0  16,256
    NFS daemon    11     4K      4K 42378K       11    0     0  64,256,512
       NFS req     0     0K      1K 42378K  2212294    0     0  64
   NFS srvsock     1     1K      1K 42378K        1    0     0  256
      syncache     1     8K      8K 42378K        1    0     0  8K
     tseg_qent     0     0K      1K 42378K      185    0     0  32
   IpFw/IpAcct    53     8K     15K 42378K     2652    0     0  64,128,256,1K
      in_multi     3     1K      1K 42378K        3    0     0  32
      routetbl   140    20K    120K 42378K     2327    0     0  16,32,64,128,256
   ether_multi    12     1K      1K 42378K       12    0     0  16,32,64
        ifaddr    27     6K      6K 42378K       27    0     0  32,64,256,2K
           BPF    15   137K    137K 42378K       15    0     0  16,32,128,4K,32K
   Export Host     6     2K      2K 42378K        6    0     0  256
        vnodes    26     6K      6K 42378K      337    0     0  16,32,64,128,256
         mount    16     8K      8K 42378K       18    0     0  16,128,512
 cluster_save buffer     0     0K      1K 42378K     1031    0     0  32
      vfscache 18465  1287K   1696K 42378K  1339707    0     0  64,128,128K
    BIO buffer    25    50K    248K 42378K     1868    0     0  512,2K
           pcb    63     6K      6K 42378K     5919    0     0  16,32,64,2K
        soname    34     3K      5K 42378K   426863    0     0  16,32,64,128
    ACD driver     2     3K      3K 42378K        2    0     0  256,2K
          accf     3     1K      1K 42378K        3    0     0  16,32
          ptys     1     1K      1K 42378K        1    0     0  512
          ttys   210    27K     27K 42378K      639    0     0  128,256
           shm    14    33K     43K 42378K     1141    0     0  16,2K,8K
           sem     3    88K     88K 42378K        3    0     0  16K,32K,64K
           msg     4   137K    137K 42378K        4    0     0  512,16K,32K,128K
          rman    58     4K      4K 42378K      401    0     0  16,64
           iov     0     0K      1K 42378K      316    0     0  128
      ioctlops     0     0K      4K 42378K       14    0     0  512,1K,2K,4K
     taskqueue     1     1K      1K 42378K        1    0     0  32
          SWAP     2    73K    145K 42378K        4    0     0  32,128K
 ATAPI generic     1     1K      1K 42378K        2    0     0  32,128
          kobj     1     4K      4K 42378K        1    0     0  4K
  eventhandler    15     1K      1K 42378K       15    0     0  32,64
           bus   375    34K     37K 42378K      696    0     0  16,32,64,128,256,512,1K,2K,4K,8K
     sysctloid    10     1K      1K 42378K       10    0     0  16,64
        sysctl     0     0K      1K 42378K     4790    0     0  16,32
       uidinfo    10     1K      1K 42378K      708    0     0  32,512
          cred    68     9K     12K 42378K   722640    0     0  128
       subproc   210    14K     21K 42378K  8845542    0     0  32,64,256
          proc     2     4K      4K 42378K        2    0     0  2K
       session    46     3K      4K 42378K     4248    0     0  64
          pgrp    46     2K      2K 42378K     4248    0     0  32
   ATA generic     2     1K      1K 42378K        2    0     0  16,512
          temp   299   138K    152K 42378K  9568290    0     0  16,32,64,128,256,512,1K,4K,8K,32K,64K
        devbuf   506   412K    412K 42378K     1168    0     0  16,32,64,128,256,512,1K,2K,4K,16K,32K,64K
         lockf    36     3K      4K 42378K  1546084    0     0  64
        prison     3     2K      2K 42378K        3    0     0  512
        atexit     1     1K      1K 42378K        1    0     0  16
        zombie     0     0K      1K 42378K  4419948    0     0  128
     proc-args    70     4K      6K 42378K  4491591    0     0  16,32,64,128,256
        kqueue     4     4K     10K 42378K     5529    0     0  256,1K
         sigio     1     1K      1K 42378K      121    0     0  32
          file   646    41K     52K 42378K 18711429    0     0  64
     file desc   120    31K     45K 42378K  4422440    0     0  256,512
         dev_t   663    83K     83K 42378K      663    0     0  128
   timecounter    10     2K      2K 42378K       10    0     0  128
           kld     4     1K      1K 42378K       35    0     0  16,32,128
 
 Memory Totals:  In Use    Free    Requests
                  7564K   1449K    58199125
 
 Here is the kernel:
 
 machine		i386
 cpu		I586_CPU
 cpu		I686_CPU
 ident		NEWMONSTER
 
 maxusers	128
 makeoptions	DEBUG=-g
 
 options		CPU_ENABLE_SSE
 options 	INET
 options 	FFS
 options 	FFS_ROOT
 options		NFS
 options 	SOFTUPDATES
 options 	PROCFS
 options		CD9660
 options 	COMPAT_43
 options 	UCONSOLE
 options 	USERCONFIG
 options 	VISUAL_USERCONFIG
 options 	P1003_1B
 options 	_KPOSIX_PRIORITY_SCHEDULING
 options		_KPOSIX_VERSION=199309L
 options		KTRACE
 options		PERFMON
 
 options 	SYSVSHM
 options 	SYSVMSG
 options 	SYSVSEM
 
 options		SHMMAXPGS=8192  # max amount of shared memory pages (4k on i386)
 options		SHMALL=33554432 # max amount of shared memory (bytes)
 options		SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)"
 
 options		SHMMNI=256      # max shared mem id's per system
 options		SHMSEG=256      # max shared mem id's per process
 
 options		MSGMNB=8192     # max # of bytes in a queue
 options		MSGMNI=256      # number of message queue identifiers
 options		MSGSEG=256      # number of message segments per queue
 options		MSGSSZ=64       # size of a message segment
 options		MSGTQL=8192     # max messages in system
 
 options		SEMMAP=256
 options		SEMMNI=256
 options		SEMMNS=512
 options		SEMMNU=256
 
 options		INCLUDE_CONFIG_FILE
 options		IPFILTER
 options		IPFILTER_LOG
 options         IPFIREWALL
 options         IPFIREWALL_VERBOSE
 options         IPFIREWALL_DEFAULT_TO_ACCEPT
 options		IPFW2
 
 options		RANDOM_IP_ID
 options		ICMP_BANDLIM
 
 options		ACCEPT_FILTER_HTTP
 options         VESA
 options         PANIC_REBOOT_WAIT_TIME=20
 
 options         SMBFS
 options         LIBMCHAIN
 options         LIBICONV
 options         NETSMB
 options         NETSMBCRYPTO
 
 options         UFS_DIRHASH
 options         SHOW_BUSYBUFS
 options		HZ=1000
 
 device		isa
 device		pci
 
 device		fdc0	at isa? port IO_FD1 irq 6 drq 2
 device		fd0	at fdc0 drive 0
 device		fd1	at fdc0 drive 1
 
 device		ata0	at isa? port IO_WD1 irq 14
 device		ata1	at isa? port IO_WD2 irq 15
 
 device		ata
 device		atadisk
 device          atapicd
 options 	ATA_STATIC_ID
 device          ahc
 device          aic0    at isa?
 device          scbus
 device          da
 device          sa
 device          cd
 device          pass
 device		atkbdc0	at isa? port IO_KBD
 device		atkbd0	at atkbdc? irq 1
 device		vga0	at isa?
 device		sc0	at isa? flags 0x100
 device		npx0	at nexus? port IO_NPX irq 13
 device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
 device		sio1	at isa? port IO_COM2 irq 3
 # device		sio2	at isa? disable port IO_COM3 irq 5
 # device		sio3	at isa? disable port IO_COM4 irq 9
 device miibus
 device dc
 device fxp
 pseudo-device	loop
 pseudo-device	ether
 pseudo-device	tun
 pseudo-device	pty
 pseudo-device	bpf	8
 pseudo-device	vn	2
 pseudo-device   gzip
 pseudo-device   splash
 device          smbus
 device          intpm
 device          alpm
 device          ichsmb
 device          viapm
 device          smb
 device          iicbus
 device          iicbb
 device          ic
 device          iic
 device          iicsmb
 device		apm0 at nexus?
 
 This machine runs squid && pgsql - I cleared softupdates from squid and pgsql filesystems but the panic chase me.
 
 -- 
 VAMPIRO-RIPN

From: El Vampiro <vampiro@rootshell.ru>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Wed, 11 Sep 2002 09:06:16 +0700

 One more panic
 
 # gdb -k kernel.debug.4 vmcore.4
 GNU gdb 4.18 (FreeBSD)
 Copyright 1998 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-unknown-freebsd"...Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 in elfstab_build_psymtabs
 Deprecated bfd_read called at /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 933 in fill_symbuf
 
 IdlePTD at phsyical address 0x003ee000
 initial pcb at physical address 0x0035cf60
 panicstr: softdep_lock: locking against myself
 panic messages:
 ---
 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0xffff000a
 fault code              = supervisor read, page not present
 instruction pointer     = 0x8:0xc022f4ac
 stack pointer           = 0x10:0xcd774ce8
 frame pointer           = 0x10:0xcd774ce8
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 85221 (nmbd)
 interrupt mask          = bio
 trap number             = 12
 panic: page fault
 
 syncing disks... panic: softdep_lock: locking against myself
 Uptime: 1d1h33m56s
 
 dumping to dev #da/0x20009, offset 128
 dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
 ---
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 487             if (dumping++) {
 (kgdb) where
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 #1  0xc01918af in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
 #2  0xc0191cd4 in poweroff_wait (junk=0xc03213c0, howto=-968037120)
     at /usr/src/sys/kern/kern_shutdown.c:595
 #3  0xc022f1de in acquire_lock (lk=0xc034b57c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
 #4  0xc02348dc in softdep_count_dependencies (bp=0xc64ced00, wantcount=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
 #5  0xc0237b18 in ffs_fsync (ap=0xcd774b88) at /usr/src/sys/ufs/ffs/ffs_vnops.c:168
 #6  0xc0236603 in ffs_sync (mp=0xc12a6400, waitfor=2, cred=0xc0a42680, p=0xc03717e0) at vnode_if.h:558
 #7  0xc01c1a53 in sync (p=0xc03717e0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
 #8  0xc019164a in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
 #9  0xc0191cd4 in poweroff_wait (junk=0xc032b72c, howto=-1070419377)
     at /usr/src/sys/kern/kern_shutdown.c:595
 #10 0xc0279816 in trap_fatal (frame=0xcd774ca8, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
 #11 0xc02794e9 in trap_pfault (frame=0xcd774ca8, usermode=0, eva=4294901770)
     at /usr/src/sys/i386/i386/trap.c:867
 #12 0xc02790a7 in trap (frame={tf_fs = -847839216, tf_es = -1048182768, tf_ds = 16, tf_edi = 0, 
       tf_esi = -1048123136, tf_ebp = -847819544, tf_isp = -847819564, tf_ebx = -65536, tf_edx = 0, 
       tf_ecx = -65536, tf_eax = -1046726272, tf_trapno = 12, tf_err = 0, tf_eip = -1071450964, 
       tf_cs = 8, tf_eflags = 66067, tf_esp = -847819512, tf_ss = -1071434834})
     at /usr/src/sys/i386/i386/trap.c:466
 #13 0xc022f4ac in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 #14 0xc02333ae in softdep_update_inodeblock (ip=0xc186e900, bp=0xc64c5fa4, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 #15 0xc022e32d in ffs_update (vp=0xce00c880, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
 #16 0xc022e611 in ffs_truncate (vp=0xce00c880, length=0, flags=0, cred=0x0, p=0xcd693560)
     at /usr/src/sys/ufs/ffs/ffs_inode.c:201
 #17 0xc0239390 in ufs_inactive (ap=0xcd774eb0) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
 #18 0xc023e889 in ufs_vnoperate (ap=0xcd774eb0) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2422
 #19 0xc01bfba4 in vput (vp=0xce00c880) at vnode_if.h:815
 #20 0xc01c2e8d in unlink (p=0xcd693560, uap=0xcd774f80) at /usr/src/sys/kern/vfs_syscalls.c:1504
 #21 0xc0279a3a in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = -1078001617, tf_edi = 672627448, 
       tf_esi = 0, tf_ebp = -1077937936, tf_isp = -847818796, tf_ebx = 1, tf_edx = 135532544, 
       tf_ecx = 135548928, tf_eax = 10, tf_trapno = 12, tf_err = 2, tf_eip = 672239144, tf_cs = 31, 
       tf_eflags = 647, tf_esp = -1077940028, tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1175
 #22 0xc026d1e5 in Xint0x80_syscall ()
 #23 0x805d23c in ?? ()
 #24 0x804b714 in ?? ()
 #25 0x804c40d in ?? ()
 #26 0x804a871 in ?? ()
 (kgdb) up 13
 #13 0xc022f4ac in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 467                     panic("worklist_remove: lock not held");
 (kgdb) l
 462     worklist_remove(item)
 463             struct worklist *item;
 464     {
 465
 466             if (lk.lkt_held == -1)
 467                     panic("worklist_remove: lock not held");
 468             if ((item->wk_state & ONWORKLIST) == 0) {
 469                     FREE_LOCK(&lk);
 470                     panic("worklist_remove: not on list");
 471             }
 (kgdb) p lk
 $1 = {lkt_spl = 0, lkt_held = -1}
 (kgdb) up
 #14 0xc02333ae in softdep_update_inodeblock (ip=0xc186e900, bp=0xc64c5fa4, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 3847                    WORKLIST_REMOVE(wk);
 (kgdb) l
 3842             * operations dependent on the inode being written to disk
 3843             * can be moved to the id_bufwait so that they will be
 3844             * processed when the buffer I/O completes.
 3845             */
 3846            while ((wk = LIST_FIRST(&inodedep->id_inowait)) != NULL) {
 3847                    WORKLIST_REMOVE(wk);
 3848                    WORKLIST_INSERT(&inodedep->id_bufwait, wk);
 3849            }
 3850            /*
 3851             * Newly allocated inodes cannot be written until the bitmap
 (kgdb) p wk
 $2 = (struct worklist *) 0x68c040
 (kgdb) p *wk
 Cannot access memory at address 0x68c040.
 (kgdb) p *inodedep
 $3 = {id_list = {wk_list = {le_next = 0x0, le_prev = 0xc19c1400}, wk_type = 1, wk_state = 32777}, 
   id_hash = {le_next = 0x0, le_prev = 0xc1230f58}, id_fs = 0xc12d0800, id_ino = 22126, 
   id_nlinkdelta = 0, id_savedino = 0x0, id_deps = {le_next = 0x0, le_prev = 0xc1411118}, id_buf = 0x0, 
   id_savedsize = -1, id_pendinghd = {lh_first = 0x0}, id_bufwait = {lh_first = 0xc14aa380}, 
   id_inowait = {lh_first = 0xffff0000}, id_inoupdt = {tqh_first = 0x0, tqh_last = 0xc19c39c4}, 
   id_newinoupdt = {tqh_first = 0x0, tqh_last = 0xc19c39cc}}
 
 Here is the kernel:
 
 machine		i386
 cpu		I586_CPU
 cpu		I686_CPU
 ident		NEWMONSTER
 
 maxusers	128
 makeoptions	DEBUG=-g
 
 options		CPU_ENABLE_SSE
 options 	INET
 options 	FFS
 options 	FFS_ROOT
 options		MFS
 options		NFS
 options 	PROCFS
 options		CD9660
 options 	COMPAT_43
 options 	UCONSOLE
 options 	USERCONFIG
 options 	VISUAL_USERCONFIG
 options		PERFMON
 
 options 	SYSVSHM
 options 	SYSVMSG
 options 	SYSVSEM
 
 options		SHMMAXPGS=8192  # max amount of shared memory pages (4k on i386)
 options		SHMALL=33554432 # max amount of shared memory (bytes)
 
 options		MSGMNB=8192     # max # of bytes in a queue
 options		MSGMNI=256      # number of message queue identifiers per system
 options		MSGSEG=256      # number of message segments per queue
 options		MSGSSZ=64       # size of a message segment
 options		MSGTQL=8192     # max messages in system per system
 
 options		SEMMAP=512
 options		SEMMNI=512
 options		SEMMNS=1024
 options		SEMMNU=512
 
 options 	P1003_1B
 options 	_KPOSIX_PRIORITY_SCHEDULING
 options		_KPOSIX_VERSION=199309L
 
 options		INCLUDE_CONFIG_FILE
 options		IPFILTER
 options		IPFILTER_LOG
 options		IPFIREWALL
 options		IPFIREWALL_VERBOSE
 options		IPFIREWALL_DEFAULT_TO_ACCEPT
 options		IPFW2
 
 options		RANDOM_IP_ID
 options		ICMP_BANDLIM
 
 options		PANIC_REBOOT_WAIT_TIME=20
 
 options		SMBFS
 options		LIBMCHAIN
 options		LIBICONV
 options		NETSMB
 options		NETSMBCRYPTO
 
 options		UFS_DIRHASH
 options 	SOFTUPDATES
 
 device		isa
 device		pci
 
 device		fdc0	at isa? port IO_FD1 irq 6 drq 2
 device		fd0	at fdc0 drive 0
 device		fd1	at fdc0 drive 1
 
 device		ata0	at isa? port IO_WD1 irq 14
 device		ata1	at isa? port IO_WD2 irq 15
 device		ata
 device		atadisk
 device		atapicd
 options 	ATA_STATIC_ID
 
 device		ahc
 device		aic0    at isa?
 device		isp					# Qlogic family
 device		ispfw				# Firmware for QLogic HBAs
 
 device		scbus
 device		da
 device		sa
 device		cd
 device		pass
 
 device		atkbdc0	at isa? port IO_KBD
 device		atkbd0	at atkbdc? irq 1
 device		vga0	at isa?
 device		sc0	at isa? flags 0x100
 options		SC_HISTORY_SIZE=256
 
 device		npx0	at nexus? port IO_NPX irq 13
 
 device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
 device		sio1	at isa? port IO_COM2 irq 3
 # device		sio2	at isa? disable port IO_COM3 irq 5
 # device		sio3	at isa? disable port IO_COM4 irq 9
 
 device miibus
 device dc
 device fxp
 
 pseudo-device	loop
 pseudo-device	ether
 pseudo-device	tun
 pseudo-device	pty
 pseudo-device	bpf	6
 pseudo-device	vn	2
 pseudo-device   gzip
 pseudo-device   splash
 pseudo-device	snp
 
 And more info from crashdumps:
 
 # vmstat -M vmcore.4 -N kernel.debug.4 
  procs      memory      page                    disks     faults      cpu
  r b w     avm    fre  flt  re  pi  po  fr  sr da0 da1   in   sy  cs us sy id
  2 2 0  180728  26468 1058   0   0   0 1222   1   0   0  207 4001 150  4  5 91
 
 # iostat -M vmcore.4 -N kernel.debug.4 
       tty             da0              da1             acd0             cpu
  tin tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
    0  373  0.00   0  0.00   0.00   0  0.00   0.00   0  0.00   4  0  5  0 91
 
 # netstat -m -M vmcore.4 -N kernel.debug.4 
 65/1152/10240 mbufs in use (current/peak/max):
         65 mbufs allocated to data
 64/526/2560 mbuf clusters in use (current/peak/max)
 1340 Kbytes allocated to network (17% of mb_map in use)
 0 requests for memory denied
 0 requests for memory delayed
 0 calls to protocol drain routines
 
 # vmstat -m -M vmcore.4 -N kernel.debug.4 > vm4
 Memory statistics by bucket size
 Size   In Use   Free   Requests  HighWater  Couldfree
   16      984    296     936609    1280          0
   32      554    342    4261294     640          0
   64    15908   4956   30941725     320      22344
  128     1344    448    6128690     160      43544
  256    15654   2378    4975665      80       3876
  512      470    154      91754      40         54
   1K      118     22    3868477      20       6607
   2K       57    113       2138      10       1322
   4K       30      2     602308       5          0
   8K        2      1         51       5          0
  16K        6      0        250       5          0
  32K        8      0         20       5          0
  64K       10      0         10       5          0
 128K        3      0          4       5          0
 
 Memory usage type by bucket size
 Size  Type(s)
   16  uc_devlist, UFS dirhash, p1003.1b, NFSV3 srvdesc, routetbl,
 	  ether_multi, BPF, vnodes, mount, pcb, soname, shm, rman, bus,
 	  sysctloid, sysctl, ATA generic, temp, devbuf, atexit, proc-args, kld
   32  atkbddev, UFS dirhash, dirrem, mkdir, diradd, freefile, freefrag,
 	  indirdep, bmsafemap, newblk, tseg_qent, in_multi, routetbl,
 	  ether_multi, ifaddr, BPF, vnodes, cluster_save buffer, pcb, soname,
 	  taskqueue, SWAP, ATAPI generic, eventhandler, bus, sysctl, uidinfo,
 	  subproc, pgrp, temp, devbuf, proc-args, sigio, kld
   64  isadev, UFS dirhash, allocindir, allocdirect, pagedep, NFS daemon,
 	  NFS req, IpFw/IpAcct, routetbl, ether_multi, ifaddr, vnodes,
 	  vfscache, pcb, soname, rman, eventhandler, bus, sysctloid, subproc,
 	  session, temp, devbuf, lockf, proc-args, file
  128  ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, routetbl, BPF,
 	  vnodes, mount, vfscache, soname, ttys, iov, ATAPI generic, bus, cred,
 	  temp, devbuf, zombie, proc-args, dev_t, timecounter, kld
  256  UFS dirhash, FFS node, newblk, NFSV3 srvdesc, NFS daemon,
 	  NFS srvsock, IpFw/IpAcct, routetbl, ifaddr, Export Host, vnodes,
 	  ACD driver, ttys, bus, subproc, temp, devbuf, proc-args, kqueue,
 	  file desc
  512  UFS dirhash, UFS mount, NFSV3 diroff, NFS daemon, BPF, mount,
 	  BIO buffer, ptys, msg, ioctlops, bus, uidinfo, ATA generic, temp,
 	  devbuf, prison, file desc
   1K  uc_devlist, UFS dirhash, NQNFS Lease, IpFw/IpAcct, shm, ioctlops,
 	  bus, temp, devbuf, kqueue, file desc
   2K  UFS dirhash, UFS mount, ifaddr, BIO buffer, pcb, ACD driver,
 	  ioctlops, bus, proc, devbuf, file desc
   4K  memdesc, mbuf, UFS dirhash, UFS mount, BPF, ioctlops, kobj, bus,
 	  temp, devbuf
   8K  pagedep, syncache, bus, temp
  16K  VM pgdata, indirdep, shm, sem, msg, devbuf
  32K  BPF, sem, msg, temp, devbuf
  64K  ISOFS mount, UFS ihash, inodedep, NFS hash, sem, temp, devbuf
 128K  vfscache, msg, SWAP
 
 Memory statistics by type                          Type  Kern
         Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
      atkbddev     1     1K      1K 42379K        1    0     0  32
    uc_devlist    14     2K      2K 42379K       14    0     0  16,1K
       memdesc     1     4K      4K 42379K        1    0     0  4K
          mbuf     1     4K      4K 42379K        1    0     0  4K
        isadev    10     1K      1K 42379K       10    0     0  64
          ZONE    16     2K      2K 42379K       16    0     0  128
     VM pgdata     1    16K     16K 42379K        1    0     0  16K
   ISOFS mount     1    64K     64K 42379K        1    0     0  64K
   UFS dirhash   848   164K    253K 42379K     2445    0     0  16,32,64,128,256,512,1K,2K,4K
     UFS mount    21    46K     46K 42379K       21    0     0  512,2K,4K
     UFS ihash     1    64K     64K 42379K        1    0     0  64K
      FFS node 15321  3831K   4424K 42379K   531382    0     0  256
        dirrem     5     1K      5K 42379K    24227    0     0  32
         mkdir     0     0K      1K 42379K      126    0     0  32
        diradd     1     1K      3K 42379K    24703    0     0  32
      freefile     1     1K      3K 42379K    13050    0     0  32
      freeblks     2     1K     10K 42379K    11708    0     0  128
      freefrag     0     0K      3K 42379K     5387    0     0  32
    allocindir     0     0K    154K 42379K    24140    0     0  64
      indirdep     0     0K     33K 42379K     1103    0     0  32,16K
   allocdirect     2     1K      5K 42379K    21386    0     0  64
     bmsafemap     2     1K      1K 42379K    10251    0     0  32
        newblk     1     1K      1K 42379K    45527    0     0  32,256
      inodedep     9    65K     77K 42379K    22480    0     0  128,64K
       pagedep    10     9K     10K 42379K    11889    0     0  64,8K
      p1003.1b     1     1K      1K 42379K        1    0     0  16
      NFS hash     1    64K     64K 42379K        1    0     0  64K
   NQNFS Lease     1     1K      1K 42379K        1    0     0  1K
 NFSV3 srvdesc     0     0K      1K 42379K    63902    0     0  16,256
  NFSV3 diroff     1     1K      1K 42379K        1    0     0  512
    NFS daemon    71     8K      8K 42379K       71    0     0  64,256,512
       NFS req     0     0K      1K 42379K  1950920    0     0  64
   NFS srvsock     1     1K      1K 42379K        1    0     0  256
      syncache     1     8K      8K 42379K        1    0     0  8K
     tseg_qent     0     0K      3K 42379K    19470    0     0  32
   IpFw/IpAcct    88    13K     59K 42379K    12062    0     0  64,128,256,1K
      in_multi     3     1K      1K 42379K        3    0     0  32
      routetbl   152    21K    135K 42379K   425559    0     0  16,32,64,128,256
   ether_multi    12     1K      1K 42379K       12    0     0  16,32,64
        ifaddr    27     6K      6K 42379K       27    0     0  32,64,256,2K
           BPF    15   137K    266K 42379K       39    0     0  16,32,128,512,4K,32K
   Export Host     6     2K      2K 42379K        6    0     0  256
        vnodes    27     7K      7K 42379K      359    0     0  16,32,64,128,256
         mount    16     8K      8K 42379K       18    0     0  16,128,512
 cluster_save buffer     0     0K      1K 42379K     1595    0     0  32
      vfscache 14754  1058K   1299K 42379K   559532    0     0  64,128,128K
    BIO buffer    34    68K    290K 42379K     2145    0     0  512,2K
           pcb    64     6K      8K 42379K    15315    0     0  16,32,64,2K
        soname    38     4K      5K 42379K   900210    0     0  16,32,64,128
    ACD driver     2     3K      3K 42379K        2    0     0  256,2K
          ptys     6     3K      3K 42379K        6    0     0  512
          ttys   210    27K     52K 42379K     2862    0     0  128,256
           shm    12    23K     31K 42379K     1068    0     0  16,1K,16K
           sem     3    88K     88K 42379K        3    0     0  16K,32K,64K
           msg     4   137K    137K 42379K        4    0     0  512,16K,32K,128K
          rman    58     4K      4K 42379K      401    0     0  16,64
           iov     0     0K      1K 42379K      366    0     0  128
      ioctlops     0     0K      4K 42379K       14    0     0  512,1K,2K,4K
     taskqueue     1     1K      1K 42379K        1    0     0  32
          SWAP     2    73K    145K 42379K        4    0     0  32,128K
 ATAPI generic     1     1K      1K 42379K        2    0     0  32,128
          kobj     1     4K      4K 42379K        1    0     0  4K
  eventhandler    15     1K      1K 42379K       15    0     0  32,64
           bus   375    34K     38K 42379K      690    0     0  16,32,64,128,256,512,1K,2K,4K,8K
     sysctloid    10     1K      1K 42379K       10    0     0  16,64
        sysctl     0     0K      1K 42379K   218127    0     0  16,32
       uidinfo    11     1K      1K 42379K     1060    0     0  32,512
          cred    76    10K     15K 42379K   712896    0     0  128
       subproc   232    17K     27K 42379K  7763310    0     0  32,64,256
          proc     2     4K      4K 42379K        2    0     0  2K
       session    47     3K      4K 42379K     5890    0     0  64
          pgrp    47     2K      2K 42379K     6119    0     0  32
   ATA generic     2     1K      1K 42379K        2    0     0  16,512
          temp   311   136K    151K 42379K  8534373    0     0  16,32,64,128,256,512,1K,4K,8K,32K,64K
        devbuf   522   419K    419K 42379K     1185    0     0  16,32,64,128,256,512,1K,2K,4K,16K,32K,64K
         lockf    43     3K      6K 42379K  1517843    0     0  64
        prison     3     2K      2K 42379K        3    0     0  512
        atexit     1     1K      1K 42379K        1    0     0  16
        zombie     0     0K      2K 42379K  3877840    0     0  128
     proc-args    78     4K      7K 42379K  3943027    0     0  16,32,64,128,256
        kqueue     7     7K     24K 42379K     7347    0     0  256,1K
         sigio     1     1K      1K 42379K      127    0     0  32
          file   675    43K     72K 42379K 16631410    0     0  64
     file desc   131    36K     66K 42379K  3881154    0     0  256,512,1K,2K
         dev_t   663    83K     83K 42379K      663    0     0  128
   timecounter    10     2K      2K 42379K       10    0     0  128
           kld     4     1K      1K 42379K       35    0     0  16,32,128
 
 Memory Totals:  In Use    Free    Requests
                  6836K   1317K    51808995
 
 -- 
 VAMPIRO-RIPN
 http://vampiro.rootshell.ru

From: "Evgueni V. Gavrilov" <aquatique@rusunix.org>
To: freebsd-gnats-submit@FreeBSD.org, mckusick@mckusick.com
Cc:  
Subject: Re: kern/42277: still climbing (6 months later)
Date: Mon, 24 Feb 2003 11:56:45 +0600

 6 months after PR submitting the bug is still climbing...
 please, turn some FS-guys to look it - the only way to avoid yhe bug - to turn softupdates off.
 :-(
 
 I can provide crashdumps and more on that box.
 
 IdlePTD at phsyical address 0x00386000
 initial pcb at physical address 0x002f0820
 panicstr: softdep_lock: locking against myself
 panic messages:
 ---
 Fatal trap 12: page fault while in kernel mode
 fault virtual address   = 0xffff000a
 fault code              = supervisor read, page not present
 instruction pointer     = 0x8:0xc022b510
 stack pointer           = 0x10:0xcd114d10
 frame pointer           = 0x10:0xcd114d10
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, def32 1, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 5 (syncer)
 interrupt mask          = bio
 trap number             = 12
 panic: page fault
 
 syncing disks... panic: softdep_lock: locking against myself
 Uptime: 19h3m41s
 
 dumping to dev #da/0x20009, offset 128
 dump 255 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 235 234 233 232 231 230 229 228 227 226 225 224 223 222 221 220 219 218 217 216 215 214 213 212 211 210 209 208 207 206 205 204 203 202 201 200 199 198 197 196 195 194 193 192 191 190 189 188 187 186 185 184 183 182 181 180 179 178 177 176 175 174 173 172 171 170 169 168 167 166 165 164 163 162 161 160 159 158 157 156 155 154 153 152 151 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 12 9 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
 ---
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 487             if (dumping++) {
 (kgdb) where
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
 #1  0xc0188d8f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
 #2  0xc01891b4 in poweroff_wait (junk=0xc02add80, howto=-967414684) at /usr/src/sys/kern/kern_shutdown.c:595
 #3  0xc022b242 in acquire_lock (lk=0xc02deb9c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
 #4  0xc0230940 in softdep_count_dependencies (bp=0xc6566c64, wantcount=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
 #5  0xc0233b84 in ffs_fsync (ap=0xcd114bb0) at /usr/src/sys/ufs/ffs/ffs_vnops.c:168
 #6  0xc0232667 in ffs_sync (mp=0xc12e8600, waitfor=2, cred=0xc0a60580, p=0xc03050e0) at vnode_if.h:558
 #7  0xc01b94bf in sync (p=0xc03050e0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
 #8  0xc0188b2a in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
 #9  0xc01891b4 in poweroff_wait (junk=0xc02beb8c, howto=-1070864721) at /usr/src/sys/kern/kern_shutdown.c:595
 #10 0xc02769a6 in trap_fatal (frame=0xcd114cd0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
 #11 0xc0276679 in trap_pfault (frame=0xcd114cd0, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867
 #12 0xc0276263 in trap (frame={tf_fs = -854523888, tf_es = -1046544368, tf_ds = 16, tf_edi = 0, tf_esi = -1046538240,
       tf_ebp = -854504176, tf_isp = -854504196, tf_ebx = -65536, tf_edx = -1048670080, tf_ecx = -65536, tf_eax = -1048670080,
       tf_trapno = 12, tf_err = 0, tf_eip = -1071467248, tf_cs = 8, tf_eflags = 66071, tf_esp = -854504144,
       tf_ss = -1071451118}) at /usr/src/sys/i386/i386/trap.c:466
 #13 0xc022b510 in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
 #14 0xc022f412 in softdep_update_inodeblock (ip=0xc19f1800, bp=0xc658d5b4, waitfor=0)
     at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
 #15 0xc022a391 in ffs_update (vp=0xcd8e1380, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
 #16 0xc022a675 in ffs_truncate (vp=0xcd8e1380, length=0, flags=0, cred=0x0, p=0xcc008780)
     at /usr/src/sys/ufs/ffs/ffs_inode.c:201
 #17 0xc02353fc in ufs_inactive (ap=0xcd114ed8) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
 #18 0xc023a825 in ufs_vnoperate (ap=0xcd114ed8) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
 #19 0xc01b7610 in vput (vp=0xcd8e1380) at vnode_if.h:815
 #20 0xc022e218 in handle_workitem_remove (dirrem=0xc18b3780) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
 #21 0xc022b891 in process_worklist_item (matchmnt=0x0, flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
 #22 0xc022b736 in softdep_process_worklist (matchmnt=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
 #23 0xc01b6f37 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
 (kgdb) up 4
 #4  0xc0230940 in softdep_count_dependencies (bp=0xc6566c64, wantcount=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
 4792            ACQUIRE_LOCK(&lk);
 (kgdb) p lk
 $1 = {lkt_spl = 0, lkt_held = -1}
 
 $ uname -a
 FreeBSD ns2.rsb.local 4.8-PRERELEASE FreeBSD 4.8-PRERELEASE #0: Sun Feb 23 11:04:58 OMST 2003     aquatique@ns2.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER  i386
 
 kernel is:
 
 machine         i386
 cpu             I686_CPU
 ident           NEWMONSTER
 
 maxusers        128
 
 makeoptions     DEBUG=-g
 
 options         CPU_ENABLE_SSE
 options         INET
 options         FFS
 options         FFS_ROOT
 options         MFS
 options         NFS
 options         PROCFS
 options         CD9660
 options         COMPAT_43
 options         UCONSOLE
 options         USERCONFIG
 options         VISUAL_USERCONFIG
 options         PERFMON
 
 options         SYSVSHM
 options         SYSVMSG
 options         SYSVSEM
 
 options         SHMMAXPGS=16384 # max amount of shared memory pages (4k on i386)
 options         SHMALL=16384    # max amount of shared memory (bytes)
 options         SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)"
 
 options         MSGMNB=8192     # max # of bytes in a queue
 options         MSGMNI=256      # number of message queue identifiers per system
 options         MSGSEG=2048             # number of message segments per queue
 options         MSGSSZ=64       # size of a message segment
 options         MSGTQL=8192     # max messages in system per system
 
 options         SEMMAP=512
 options         SEMMNI=512
 options         SEMMNS=1024
 options         SEMMNU=512
 
 options         P1003_1B
 options         _KPOSIX_PRIORITY_SCHEDULING
 options         _KPOSIX_VERSION=199309L
 
 options         INCLUDE_CONFIG_FILE
 
 options         IPFILTER
 options         IPFILTER_LOG
 options         IPFIREWALL
 options         IPFIREWALL_VERBOSE
 options         IPFIREWALL_DEFAULT_TO_ACCEPT
 options         IPFW2
 
 options         RANDOM_IP_ID
 options         ICMP_BANDLIM
 
 options         PANIC_REBOOT_WAIT_TIME=20
 
 options         SMBFS
 options         LIBMCHAIN
 options         LIBICONV
 options         NETSMB
 options         NETSMBCRYPTO
 
 options         UFS_DIRHASH
 options         SOFTUPDATES
 
 device          isa
 device          pci
 
 device          fdc0    at isa? port IO_FD1 irq 6 drq 2
 device          fd0     at fdc0 drive 0
 device          fd1     at fdc0 drive 1
 
 device          ata0    at isa? port IO_WD1 irq 14
 device          ata1    at isa? port IO_WD2 irq 15
 device          ata
 device          atadisk
 device          atapicd
 options         ATA_STATIC_ID
 
 device          ahc
 device          aic0    at isa?
 
 device          scbus
 device          da
 device          sa
 device          cd
 device          pass
 
 device          atkbdc0 at isa? port IO_KBD
 device          atkbd0  at atkbdc? irq 1
 device          vga0    at isa?
 device          sc0     at isa? flags 0x100
 options         SC_HISTORY_SIZE=1024
 
 device          npx0    at nexus? port IO_NPX irq 13
 
 device          sio0    at isa? port IO_COM1 flags 0x10 irq 4
 device          sio1    at isa? port IO_COM2 irq 3
 device          sio2    at isa? disable port IO_COM3 irq 5
 device          sio3    at isa? disable port IO_COM4 irq 9
 
 device          puc
 options         PUC_FASTINTR
 
 device          miibus
 device          fxp
 device          dc
 
 pseudo-device   loop
 pseudo-device   ether
 pseudo-device   tun     4
 pseudo-device   ppp     4
 pseudo-device   pty
 pseudo-device   bpf     4
 pseudo-device   vn      2
 pseudo-device   gzip
 pseudo-device   splash
 pseudo-device   snp
 
 dmesg are:
 
 Copyright (c) 1992-2003 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
 FreeBSD 4.8-PRERELEASE #0: Sun Feb 23 11:04:58 OMST 2003
     aquatique@ns2.rsb.local:/usr/obj/usr/src/sys/NEWMONSTER
 Timecounter "i8254"  frequency 1193182 Hz
 Timecounter "TSC"  frequency 999724552 Hz
 CPU: Intel Pentium III (999.72-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0x68a  Stepping = 10
   Features=0x387fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,PN,MMX,FXSR,SSE>
 real memory  = 268369920 (262080K bytes)
 avail memory = 257544192 (251508K bytes)
 Preloaded elf kernel "kernel" at 0xc0367000.
 Preloaded elf module "accf_http.ko" at 0xc036709c.
 netsmb_dev: loaded
 Pentium Pro MTRR support enabled
 Using $PIR table, 268435454 entries at 0xc00fdf10
 npx0: <math processor> on motherboard
 npx0: INT 16 interface
 pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
 pci0: <PCI bus> on pcib0
 pci0: <ATI Mach64-GV graphics accelerator> at 2.0 irq 11
 fxp0: <Intel Pro 10/100B/100+ Ethernet> port 0x5880-0x58bf mem 0xfb000000-0xfb0fffff,0xfb101000-0xfb101fff irq 10 at device 3.0 on pci0
 fxp0: Ethernet address 00:d0:b7:b8:ab:64
 inphy0: <i82555 10/100 media interface> on miibus0
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 dc0: <Davicom DM9102A 10/100BaseTX> port 0x5400-0x54ff mem 0xfb102000-0xfb1020ff irq 5 at device 7.0 on pci0
 dc0: Ethernet address: 00:80:ad:08:12:2b
 miibus1: <MII bus> on dc0
 ukphy0: <Generic IEEE 802.3u media interface> on miibus1
 ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 puc0: <Moxa Technologies, C168H/PCI> port 0x5c00-0x5c0f,0x58c0-0x58ff,0x5800-0x587f irq 12 at device 8.0 on pci0
 sio4: type 16550A
 sio5: type 16550A
 sio6: type 16550A
 sio7: type 16550A
 sio8: type 16550A
 sio9: type 16550A
 sio10: type 16550A
 sio11: type 16550A
 isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
 isa0: <ISA bus> on isab0
 atapci0: <ServerWorks ROSB4 ATA33 controller> port 0x5c10-0x5c1f,0x374-0x377,0x170-0x177 at device 15.1 on pci0
 ata0: at 0x1f0 irq 14 on atapci0
 ata1: at 0x170 irq 15 on atapci0
 pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
 pci1: <PCI bus> on pcib1
 ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6000-0x60ff mem 0xfd000000-0xfd000fff irq 9 at device 4.0 on pci1
 aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
 ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6400-0x64ff mem 0xfd001000-0xfd001fff irq 9 at device 4.1 on pci1
 aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
 orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcf7ff on isa0
 fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
 fdc0: FIFO enabled, 8 bytes threshold
 fd0: <1440-KB 3.5" drive> on fdc0 drive 0
 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
 sio0: type 16550A
 sio1 at port 0x2f8-0x2ff irq 3 on isa0
 sio1: type 16550A
 ipfw2 initialized, divert disabled, rule-based forwarding enabled, default to accept, logging unlimited
 IP Filter: v3.4.29 initialized.  Default = pass all, Logging = enabled
 acd0: CDROM <SONY CD-ROM CDU5221> at ata0-master PIO4
 Waiting 2 seconds for SCSI devices to settle
 sa0 at ahc1 bus 0 target 0 lun 0
 sa0: <ARCHIVE Python 04106-XXX 7550> Removable Sequential Access SCSI-2 device
 sa0: 10.000MB/s transfers (10.000MHz, offset 15)
 da0 at ahc0 bus 0 target 0 lun 0
 da0: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device
 da0: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
 da0: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
 da1 at ahc0 bus 0 target 1 lun 0
 da1: <QUANTUM ATLAS10K3_18_WLS 020W> Fixed Direct Access SCSI-3 device
 da1: 160.000MB/s transfers (80.000MHz, offset 127, 16bit), Tagged Queueing Enabled
 da1: 17537MB (35916548 512 byte sectors: 255H 63S/T 2235C)
 Mounting root from ufs:/dev/da0s1a
 

From: "Evgueni V. Gavrilov" <aquatique@rusunix.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Mon, 24 Feb 2003 12:47:35 +0600

 where full follows
 
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
         error = 0
 #1  0xc0188d8f in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316
         howto = 260
 #2  0xc01891b4 in poweroff_wait (junk=0xc02add80, howto=-967414684) at /usr/src/sys/kern/kern_shutdown.c:595
         fmt = 0xc02add80 "softdep_lock: locking against myself"
         bootopt = 260
         buf = "softdep_lock: locking against myself", '\000' <repeats 219 times>
 #3  0xc022b242 in acquire_lock (lk=0xc02deb9c) at /usr/src/sys/ufs/ffs/ffs_softdep.c:261
         lk = (struct lockit *) 0xc02add80
         holder = 260
 #4  0xc0230940 in softdep_count_dependencies (bp=0xc6566c64, wantcount=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792
         bp = (struct buf *) 0xc6566c64
         wantcount = 0
         wk = (struct worklist *) 0xc6566c64
         inodedep = (struct inodedep *) 0x0
         aip = (struct allocindir *) 0x0
         pagedep = (struct pagedep *) 0x0
         dap = (struct diradd *) 0x0
         i = 0
         retval = 0
 #5  0xc0233b84 in ffs_fsync (ap=0xcd114bb0) at /usr/src/sys/ufs/ffs/ffs_vnops.c:168
         vp = (struct vnode *) 0xcd6e9540
         bp = (struct buf *) 0xc6566c64
         nbp = (struct buf *) 0x0
         s = 6864960
         error = 0
         wait = 0
         passes = 4
         skipmeta = 0
         lbn = 1
 #6  0xc0232667 in ffs_sync (mp=0xc12e8600, waitfor=2, cred=0xc0a60580, p=0xc03050e0) at vnode_if.h:558
         a = {a_desc = 0xc02c8380, a_vp = 0xcd6e9540, a_cred = 0xc0a60580, a_waitfor = 2, a_p = 0xc03050e0}
         vp = (struct vnode *) 0xcd6e9540
         cred = (struct ucred *) 0xc0a60580
         waitfor = 2
         p = (struct proc *) 0xc03050e0
         p = (struct proc *) 0xc03050e0
         nvp = (struct vnode *) 0xcd6e9480
         vp = (struct vnode *) 0xcd6e9540
         ip = (struct inode *) 0x0
         ump = (struct ufsmount *) 0xc12f8c00
         fs = (struct fs *) 0xc130f000
         error = 0
         allerror = 0
 #7  0xc01b94bf in sync (p=0xc03050e0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:576
         p = (struct proc *) 0xc03050e0
         mp = (struct mount *) 0xc12e8600
         nmp = (struct mount *) 0x0
         asyncflag = 0
 #8  0xc0188b2a in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235
         bp = (struct buf *) 0x0
         iter = 5
         nbusy = -1070863476
         pbusy = -854504396
         howto = 256
 #9  0xc01891b4 in poweroff_wait (junk=0xc02beb8c, howto=-1070864721) at /usr/src/sys/kern/kern_shutdown.c:595
         fmt = 0xc02beb8c "%s"
         bootopt = 256
         buf = "softdep_lock: locking against myself", '\000' <repeats 219 times>
 #10 0xc02769a6 in trap_fatal (frame=0xcd114cd0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:974
         frame = (struct trapframe *) 0x100
         code = -1070863476
         type = 12
         ss = -1070863476
         esp = 0
         softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27, ssd_dpl = 0, ssd_p = 1, ssd_xx = 6, ssd_xx1 = 2,
   ssd_def32 = 1, ssd_gran = 1}
 #11 0xc0276679 in trap_pfault (frame=0xcd114cd0, usermode=0, eva=4294901770) at /usr/src/sys/i386/i386/trap.c:867
         va = 4294901760
         vm = (struct vmspace *) 0x0
         map = 0xc02f6f2c
         rv = 0
         ftype = 1 '\001'
         p = (struct proc *) 0xcc008780
 #12 0xc0276263 in trap (frame={tf_fs = -854523888, tf_es = -1046544368, tf_ds = 16, tf_edi = 0, tf_esi = -1046538240,
       tf_ebp = -854504176, tf_isp = -854504196, tf_ebx = -65536, tf_edx = -1048670080, tf_ecx = -65536, tf_eax = -1048670080,
       tf_trapno = 12, tf_err = 0, tf_eip = -1071467248, tf_cs = 8, tf_eflags = 66071, tf_esp = -854504144, tf_ss = -1071451118})
     at /usr/src/sys/i386/i386/trap.c:466
         p = (struct proc *) 0xcc008780
         sticks = 3327710644
         i = 0
         ucode = 0
         type = 12
         code = 0
         eva = 4294901770
 #13 0xc022b510 in worklist_remove (item=0xffff0000) at /usr/src/sys/ufs/ffs/ffs_softdep.c:467
         item = (struct worklist *) 0x0
 #14 0xc022f412 in softdep_update_inodeblock (ip=0xc19f1800, bp=0xc658d5b4, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:3847
         ip = (struct inode *) 0x68c040
         inodedep = (struct inodedep *) 0xc17e9080
         wk = (struct worklist *) 0x68c040
         gotit = 6864960
 #15 0xc022a391 in ffs_update (vp=0xcd8e1380, waitfor=0) at /usr/src/sys/ufs/ffs/ffs_inode.c:106
         fs = (struct fs *) 0xc130f000
         bp = (struct buf *) 0xc658d5b4
         ip = (struct inode *) 0xc19f1800
         error = 0
 #16 0xc022a675 in ffs_truncate (vp=0xcd8e1380, length=0, flags=0, cred=0x0, p=0xcc008780) at /usr/src/sys/ufs/ffs/ffs_inode.c:201
         flags = 0
         ovp = (struct vnode *) 0xcd8e1380
         lastblock = -1050765568
         oip = (struct inode *) 0xc19f1800
         bn = -872380544
         lbn = -1053816608
         lastiblock = {16842754, -872380544, -854503788}
         indir_lbn = {-1053762688, -1070823872, -846326912}
         oldblks = {5, -854503868, -1071954580, -1046538240, 16842754, -846326804, -872380544, -854503856, -1071405019,
   -854503832, -854503816, -1071913279, -854503832, -846326912, 65538}
         newblks = {-1071355862, -1066948224, -1072155537, 0, -1046538240, 16777280, 0, -854503892, -1072154947, -1046538240,
   16777280, 1048832, 65538, -846326912, -872380544}
         fs = (struct fs *) 0xc130f000
         bp = (struct buf *) 0xc01927c9
         offset = -1053816832
         size = 0
         level = 0
         count = -854503996
         nblocks = 6864960
         blocksreleased = 0
         i = -1046538240
         aflags = -1053878016
         error = 0
         allerror = -1072519760
         osize = -3670066803013795776
 #17 0xc02353fc in ufs_inactive (ap=0xcd114ed8) at /usr/src/sys/ufs/ufs/ufs_inode.c:89
         ap = (struct vop_inactive_args *) 0x0
         vp = (struct vnode *) 0xcd8e1380
         ip = (struct inode *) 0xc19f1800
         p = (struct proc *) 0xcc008780
         mode = 0
         error = 0
 #18 0xc023a825 in ufs_vnoperate (ap=0xcd114ed8) at /usr/src/sys/ufs/ufs/ufs_vnops.c:2376
         ap = (struct vop_generic_args *) 0x0
 #19 0xc01b7610 in vput (vp=0xcd8e1380) at vnode_if.h:815
         a = {a_desc = 0xc02c85c0, a_vp = 0xcd8e1380, a_p = 0xcc008780}
         vp = (struct vnode *) 0xcd8e1380
         p = (struct proc *) 0x0
         vp = (struct vnode *) 0xcd8e1380
         p = (struct proc *) 0x0
 #20 0xc022e218 in handle_workitem_remove (dirrem=0xc18b3780) at /usr/src/sys/ufs/ffs/ffs_softdep.c:2852
         dirrem = (struct dirrem *) 0xc18b3780
         p = (struct proc *) 0xcc008780
         inodedep = (struct inodedep *) 0xc17e9080
         vp = (struct vnode *) 0xcd8e1380
         ip = (struct inode *) 0x0
         oldinum = 0
         error = 0
 #21 0xc022b891 in process_worklist_item (matchmnt=0x0, flags=0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:716
         flags = 0
         wk = (struct worklist *) 0xc18b3780
         matchfs = (struct fs *) 0x0
         vp = (struct vnode *) 0x0
         matchcnt = 0
 #22 0xc022b736 in softdep_process_worklist (matchmnt=0x0) at /usr/src/sys/ufs/ffs/ffs_softdep.c:622
         matchmnt = (struct mount *) 0x0
         p = (struct proc *) 0xcc008780
         matchcnt = 0
         loopcount = 1
         starttime = 1046045855
 #23 0xc01b6f37 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1177
         slp = (struct synclist *) 0xc1219a0c
         vp = (struct vnode *) 0x0
         starttime = 1046045855
         s = 0
         p = (struct proc *) 0xcc008780
 
Responsible-Changed-From-To: freebsd-bugs->mckusic 
Responsible-Changed-By: dwmalone 
Responsible-Changed-When: Thu Mar 13 13:46:44 PST 2003 
Responsible-Changed-Why:  
Maybe Kirk can have a look at this PR - Evgueni is eager to provide more 
debugging information if it is needed. 

David. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=42277 
Responsible-Changed-From-To: mckusic->mckusick 
Responsible-Changed-By: dwmalone 
Responsible-Changed-When: Thu Mar 13 13:47:53 PST 2003 
Responsible-Changed-Why:  
(Retransmit, this time spelling Kirk's name correctly...) 

Maybe Kirk can have a look at this softupdates panic - Evgueni 
is able to provide more debugging information if needed. 

David. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=42277 

From: "Evgueni V. Gavrilov" <aquatique@rusunix.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/42277: Several kernel panics per day with panicstr: softdep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Wed, 25 Jun 2003 20:07:25 +0700

 The trouble occurs due to corrupting inodedep structure @64 byte offset 
 by another structure of 128 byte, MALLOCed with one of the following 
 malloc type: ZONE, UFS dirhash, freeblks, inodedep, IpFw/IpAcct, 
 routetbl, BPF, vnodes, mount, vfscache, soname, ttys, zombie, proc-args, 
 dev_t, timecounter, kld, iov, ATAPI generic, bus, cred, temp, devbuf.
 
 It can be reproduced by hitting/breaking limit rule of IPFW2.
 
 Following patch by Kirk McKusick adds an unused short @64 byte offset to 
 inodedep structure and upon initialization sets it to 0x12345678. Once 
 trashing occurs - value of the unused short prints.
 
 This patch prevents a box from the panic and shows possible changes. In 
 the most cases unused field being decremented, but sometimes it turns to 
 0x12325678 or 0x12305678. So I could suppose that trashing caused by 
 shift operators.
 
 http://aquatique.rusunix.org/mckusick.2.patch
 
 The only possible candiate I found is struct ip_fw, but it uses 256-byte 
 buckets instead of 128.
 
 

From: Jim Kuhn <jkuhn@sandvine.com>
To: "'freebsd-gnats-submit@FreeBSD.org'" <freebsd-gnats-submit@FreeBSD.org>,
	"'vampiro@rootshell.ru'" <vampiro@rootshell.ru>
Cc:  
Subject: Re: kern/42277: Several kernel panics per day with panicstr: soft
	dep_lock: locking against myself. Further filesystems damage guaranteed.
Date: Mon, 6 Oct 2003 10:05:49 -0400 

 (Try again, sorry about the bad first submission)
 
 I've tracked the problem down to ipfw limit rule processing.  In the
 expiry of dynamic rules, it is possible for an O_LIMIT_PARENT rule to be
 removed when it still has live children.  When the children eventually
 do expire, a pointer to the (long gone) parent is dereferenced and a
 count decremented.  Since this memory can, and is, allocated for other
 purposes, chaos ensues.
 
 The offset in question in inodedep is the offset of the 16 bit count
 field in the ipfw2 ipfw_dyn_rule.
 
 Note that this problem exists on current and stable.
 
 Suggested fix (diff excerpt):
 
 Index: src/sys/netinet/ip_fw2.c
 diff -c src/sys/netinet/ip_fw2.c:1.6.2.3.1000.9
 src/sys/netinet/ip_fw2.c:1.6.2.3
 *** src/sys/netinet/ip_fw2.c:1.6.2.3.1000.9     Mon Oct  6 09:42:33 2003
 --- src/sys/netinet/ip_fw2.c    Wed Aug 21 01:34:07 2002
 ***************
 *** 849,860 ****
                                     !TIME_LEQ( q->expire, time_second ))
                                         goto next;
                         }
 !                       if (q->dyn_type != O_LIMIT_PARENT || !q->count)
 !                       {
 !
 !                               UNLINK_DYN_RULE(prev, ipfw_dyn_v[i], q);
 !                               continue;
 !                       }
   next:
                         prev=q;
                         q=q->next;
 --- 693,700 ----
                                     !TIME_LEQ( q->expire, time_second ))
                                         goto next;
                         }
 !                       UNLINK_DYN_RULE(prev, ipfw_dyn_v[i], q);
 !                       continue;
   next:
                         prev=q;
                         q=q->next;
 
 
 Jim
State-Changed-From-To: open->closed 
State-Changed-By: mckusick 
State-Changed-When: Wed Oct 15 19:02:23 PDT 2003 
State-Changed-Why:  
This bug is resolved with update 1.40 to sys/netinet/ip_fw2.c. 
The same patch needs to be applied to -stable. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=42277 
>Unformatted:
