From vangyzen@vangyzen.net  Tue Mar 14 02:34:11 2006
Return-Path: <vangyzen@vangyzen.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 6519716A427
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 14 Mar 2006 02:34:11 +0000 (UTC)
	(envelope-from vangyzen@vangyzen.net)
Received: from smtp.vangyzen.net (70-96-250-22.br1.lkv.mn.frontiernet.net [70.96.250.22])
	by mx1.FreeBSD.org (Postfix) with ESMTP id A336A43D6B
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 14 Mar 2006 02:34:08 +0000 (GMT)
	(envelope-from vangyzen@vangyzen.net)
Received: by smtp.vangyzen.net (Postfix, from userid 1001)
	id CB9156D41E; Mon, 13 Mar 2006 20:34:07 -0600 (CST)
Message-Id: <20060314023407.CB9156D41E@smtp.vangyzen.net>
Date: Mon, 13 Mar 2006 20:34:07 -0600 (CST)
From: Eric van Gyzen <eric@vangyzen.net>
Reply-To: Eric van Gyzen <eric@vangyzen.net>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: [panic] panic: sbdrop
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         94433
>Category:       kern
>Synopsis:       [panic] panic: sbdrop
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    sam
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Mar 14 02:40:18 GMT 2006
>Closed-Date:    Mon Jun 12 21:07:09 GMT 2006
>Last-Modified:  Mon Jun 12 21:07:09 GMT 2006
>Originator:     Eric van Gyzen
>Release:        FreeBSD 6.0-RELEASE-p4 i386
>Organization:
>Environment:
System: FreeBSD host.example.com 6.0-RELEASE-p4 FreeBSD 6.0-RELEASE-p4 #6: Sat Feb 25 15:37:23 CST 2006 vangyzen@host.example.com:/freebsd/obj/freebsd/src/sys/KERNCONF i386

--- KERNCONF begins here ---
machine		i386
cpu		I686_CPU

options 	SCHED_4BSD
options 	PREEMPTION
options 	INET
options 	INET6
options 	FFS
options 	SOFTUPDATES
options 	UFS_ACL
options 	UFS_DIRHASH
options 	MSDOSFS
options 	CD9660
options 	GEOM_GPT
options 	COMPAT_43
options 	COMPAT_FREEBSD4
options 	COMPAT_FREEBSD5
options 	SCSI_DELAY=1000
options 	KTRACE
options 	SYSVSHM
options 	SYSVMSG
options 	SYSVSEM
options 	_KPOSIX_PRIORITY_SCHEDULING
options 	KBD_INSTALL_CDEV
options 	ADAPTIVE_GIANT

device		apic

device		isa
device		pci

device		ata
device		atadisk
options 	ATA_STATIC_ID

device		scbus
device		da
device		cd
device		pass

device		atkbdc
device		atkbd
device		psm

device		vga

device		splash

device		sc

device		agp

device		npx

device		pmtimer

device		ppc
device		ppbus
device		lpt
device		plip
device		ppi

device		loop
device		mem
device		io
device		random
device		ether
device		pty
device		md

device		bpf

device		uhci
device		ohci
device		ehci
device		usb
device		ugen
device		uhid
device		ukbd
device		ulpt
device		umass
device		ums

options 	INCLUDE_CONFIG_FILE

makeoptions	DEBUG=-g

options 	KDB
options 	DDB
options 	GDB

device		fdc

device		atapicd
device		atapicam

device		sym

device		sio

device		miibus
device		rl
device		fxp

device		wlan
device		wlan_wep
device		wlan_ccmp
device		wlan_tkip
device		wlan_xauth
device		wlan_acl

device		ath
device		ath_hal
device		ath_rate_sample

options		IPFIREWALL
options		IPFIREWALL_VERBOSE
options		IPFIREWALL_VERBOSE_LIMIT=1024
options		IPFIREWALL_DEFAULT_TO_ACCEPT
options		IPDIVERT
--- KERNCONF ends here ---

--- dmesg.boot begins here ---
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 6.0-RELEASE-p4 #5: Sat Feb 25 15:33:28 CST 2006
    vangyzen@host.example.com:/freebsd/obj/freebsd/src/sys/KERNCONF
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) Processor (1399.76-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x644  Stepping = 4
  Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
  AMD Features=0xc0440800<SYSCALL,<b18>,MMX+,3DNow+,3DNow>
real memory  = 268369920 (255 MB)
avail memory = 253153280 (241 MB)
wlan: mac acl policy registered
ath_hal: 0.9.14.9 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413)
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <AMIINT > on motherboard
acpi0: Power Button (fixed)
pci_link0: <ACPI PCI Link LNKA> irq 11 on acpi0
pci_link1: <ACPI PCI Link LNKB> irq 12 on acpi0
pci_link2: <ACPI PCI Link LNKC> irq 5 on acpi0
pci_link3: <ACPI PCI Link LNKD> irq 10 on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8367 (KT266/KY266x/KT333) host to PCI bridge> mem 0xe0000000-0xe1ffffff at device 0.0 on pci0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
fxp0: <Intel 82559 Pro/100 Ethernet> port 0xdc00-0xdc3f mem 0xdfffe000-0xdfffefff,0xdfe00000-0xdfefffff irq 12 at device 6.0 on pci0
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:d0:b7:5a:ce:f6
ath0: <Atheros 5212> mem 0xdffd0000-0xdffdffff irq 5 at device 7.0 on pci0
ath0: Ethernet address: 00:11:95:91:32:f2
ath0: mac 7.9 phy 4.5 radio 5.6
sym0: <1010-33> port 0xd400-0xd4ff mem 0xdffff800-0xdffffbff,0xdfffa000-0xdfffbfff irq 10 at device 8.0 on pci0
sym0: Symbios NVRAM, ID 7, Fast-80, LVD, parity checking
sym0: open drain IRQ line driver, using on-chip SRAM
sym0: using LOAD/STORE-based firmware.
sym0: handling phase mismatch from SCRIPTS.
sym0: [GIANT-LOCKED]
sym1: <1010-33> port 0xd800-0xd8ff mem 0xdffffc00-0xdfffffff,0xdfffc000-0xdfffdfff irq 10 at device 8.1 on pci0
sym1: Symbios NVRAM, ID 7, Fast-80, SE, parity checking
sym1: open drain IRQ line driver, using on-chip SRAM
sym1: using LOAD/STORE-based firmware.
sym1: handling phase mismatch from SCRIPTS.
sym1: [GIANT-LOCKED]
rl0: <RealTek 8139 10/100BaseTX> port 0xd000-0xd0ff mem 0xdffff700-0xdffff7ff irq 11 at device 9.0 on pci0
miibus1: <MII bus> on rl0
rlphy0: <RealTek internal media interface> on miibus1
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
rl0: Ethernet address: 00:40:f4:2d:96:ec
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 8233 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 17.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
uhci0: <VIA 83C572 USB controller> port 0xc400-0xc41f irq 10 at device 17.2 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xc800-0xc81f irq 10 at device 17.3 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xcc00-0xcc1f irq 10 at device 17.4 on pci0
uhci2: [GIANT-LOCKED]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
acpi_button1: <Sleep Button> on acpi0
fdc0: <floppy drive controller> port 0x3f2-0x3f3,0x3f4-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <ECP parallel printer port> port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcbfff,0xcc000-0xccfff,0xcd000-0xd0fff,0xd1000-0xd3fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ugen0: vendor 0x05e3 USB Host To Host Bridge, rev 1.00/1.80, addr 2
Timecounter "TSC" frequency 1399762999 Hz quality 800
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to accept, logging limited to 1024 packets/entry by default
ad0: 39266MB <IC35L040AVER07 0 ER4OA44A> at ata0-master UDMA100
acd0: CDRW <SONY CD-RW CRX120E/1.0j> at ata1-master PIO4
(noperiph:sym0:0:-1:-1): SCSI BUS reset delivered.
(noperiph:sym1:0:-1:-1): SCSI BUS reset delivered.
da0 at sym0 bus 0 target 0 lun 0
da0: <IBM DDYS-T18350N S96H> Fixed Direct Access SCSI-3 device 
da0: 160.000MB/s transfers (80.000MHz, offset 62, 16bit), Tagged Queueing Enabled
da0: 17501MB (35843670 512 byte sectors: 255H 63S/T 2231C)
cd0 at ata1 bus 0 target 0 lun 0
cd0: <SONY CD-RW  CRX120E 1.0j> Removable CD-ROM SCSI-0 device 
cd0: 16.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from ufs:/dev/da0s1a
--- dmesg.boot ends here ---

>Description:

panic: sbdrop

This happened twice.  I have the dumps and will gladly
help anyone who is interested.  Right now, I'm not sure
what other information would be helpful.

--- panic-sbdrop-2006-03-09 begins here ---
$ kgdb kernel.debug /var/crash/vmcore-panic-sbdrop-2006-03-09
GNU gdb 6.1.1 [FreeBSD]
[...]

Unread portion of the kernel message buffer:

[not ascii; here is a hexdump]

                     c1  20 33 70 c0 00 00 00 00
00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
29 00 00 00 00 00 00 00  00 90 02 00 00 00 00 00
00 00 00 00 6c 0b 05 c1  00 00 00 00 09 00 01 00
00 00 00 00 00 00 00 00  00 00 00 00 88 14 05 c1
30 33 70 c0 00 00 00 00  00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00  2a 00 00 00 00 00 00 00
00 a0 02 00 00 00 00 00  00 00 00 00 b4 0b 05 c1
00 00 00 00 0d 0a 00 01  00 00 00 00 00 00 00 00
00 00 00 00 00 d0 14 05  c1 40 33 70 c0 18 0c 05
c1 90 a3 4b c1 88 a3 4b  c1 00 00 00 00 6c 7f 4f
c1 f8 15 00 00 00 00 00  00 00 b0 02 00 00 00 00

#0  doadump () at pcpu.h:165
165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt f
#0  doadump () at pcpu.h:165
No locals.
#1  0xc04fae3e in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:399
	first_buf_printf = 1
#2  0xc04fb104 in panic (fmt=0xc069ed40 "sbdrop")
    at /freebsd/src/sys/kern/kern_shutdown.c:555
	td = (struct thread *) 0xc190e480
	bootopt = 260
	newpanic = 1
	ap = 0xc190e480 "[binary]"
	buf = "sbdrop", '\0' <repeats 249 times>
#3  0xc05378b8 in sbdrop_locked (sb=0xcf603b50, len=940)
    at /freebsd/src/sys/kern/uipc_socket2.c:1157
	m = (struct mbuf *) 0x0
	next = (struct mbuf *) 0x0
#4  0xc05377ce in sbflush_locked (sb=0xcf603b50)
    at /freebsd/src/sys/kern/uipc_socket2.c:1124
No locals.
#5  0xc0536d49 in sbrelease_locked (sb=0xcf603b50, so=0x0)
    at /freebsd/src/sys/kern/uipc_socket2.c:559
No locals.
#6  0xc0536db1 in sbrelease (sb=0xcf603b50, so=0xc19c2c84)
    at /freebsd/src/sys/kern/uipc_socket2.c:572
No locals.
#7  0xc0534921 in sorflush (so=0xc19c2c84)
    at /freebsd/src/sys/kern/uipc_socket.c:1480
	sb = (struct sockbuf *) 0xc19c2cd4
	pr = (struct protosw *) 0xc06d46a0
	asb = {sb_sel = {si_thrlist = {tqe_next = 0x0, tqe_prev = 0x0}, 
    si_thread = 0x0, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0, 
      kl_unlock = 0, kl_locked = 0, kl_lockarg = 0x0}, si_flags = 0}, 
  sb_mtx = {mtx_object = {lo_class = 0xc06cf004, 
      lo_name = 0xc069ecad "so_rcv", lo_type = 0xc069ecad "so_rcv", 
      lo_flags = 196608, lo_list = {tqe_next = 0x0, tqe_prev = 0x0}, 
      lo_witness = 0x0}, mtx_lock = 3247498368, mtx_recurse = 0}, 
  sb_state = 0, sb_mb = 0xc29af800, sb_mbtail = 0xc29af800, 
  sb_lastrecord = 0xc29af800, sb_cc = 940, sb_hiwat = 8192, sb_mbcnt = 2048, 
  sb_mbmax = 65536, sb_ctl = 0, sb_lowat = 1, sb_timeo = 0, sb_flags = 64}
#8  0xc0532cbb in sofree (so=0xc19c2c84)
    at /freebsd/src/sys/kern/uipc_socket.c:406
	head = (struct socket *) 0x0
#9  0xc0532fe9 in soclose (so=0xc19c2c84)
    at /freebsd/src/sys/kern/uipc_socket.c:484
	error = 0
#10 0xc0522e6b in soo_close (fp=0xc1eac870, td=0xc190e480)
    at /freebsd/src/sys/kern/sys_socket.c:317
	error = 0
	so = (struct socket *) 0x0
#11 0xc04dc0d4 in fdrop_locked (fp=0xc1eac870, td=0xc190e480) at file.h:289
	error = 0
#12 0xc04dc025 in fdrop (fp=0xc1eac870, td=0xc190e480)
    at /freebsd/src/sys/kern/kern_descrip.c:2101
No locals.
#13 0xc04da653 in closef (fp=0xc1eac870, td=0xc190e480)
    at /freebsd/src/sys/kern/kern_descrip.c:1921
	vp = (struct vnode *) 0xc1eac870
	lf = {l_start = 4294967295, l_len = -4495592928909675680, l_pid = 0, 
  l_type = -7040, l_whence = -15984}
	fdtol = (struct filedesc_to_leader *) 0xcf603ca0
	fdp = (struct filedesc *) 0xc2ce5200
#14 0xc04d7a81 in close (td=0xc190e480, uap=0x0)
    at /freebsd/src/sys/kern/kern_descrip.c:1004
	fdp = (struct filedesc *) 0xc2ce5200
	fp = (struct file *) 0xc1eac870
	fd = 1
	error = -1047468928
	holdleaders = 0
#15 0xc0662dbb in syscall (frame=
      {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 1, tf_esi = 134613344, tf_ebp = -1077941400, tf_isp = -815776412, tf_ebx = 134729728, tf_edx = 0, tf_ecx = 1, tf_eax = 6, tf_trapno = 22, tf_err = 2, tf_eip = 169785299, tf_cs = 51, tf_eflags = 642, tf_esp = -1077941428, tf_ss = 59})
    at /freebsd/src/sys/i386/i386/trap.c:976
	params = 0xbfbfeb50 <Address 0xbfbfeb50 out of bounds>
	callp = (struct sysent *) 0xc06ca6e8
	td = (struct thread *) 0xc190e480
	p = (struct proc *) 0xc19c720c
	orig_tf_eflags = 642
	sticks = 4436
	error = 0
	narg = 1
	args = {1, -815776464, -1067048045, 0, 0, 0, 4436, -1046711796}
	code = 6
#16 0xc06520cf in Xint0x80_syscall ()
    at /freebsd/src/sys/i386/i386/exception.s:200
No locals.
#17 0x00000033 in ?? ()
No symbol table info available.
Previous frame inner to this frame (corrupt stack?)
--- panic-sbdrop-2006-03-09 ends here ---

--- panic-sbdrop-2006-03-12 begins here ---
$ kgdb kernel.debug /var/crash/vmcore-panic-sbdrop-2006-03-12
GNU gdb 6.1.1 [FreeBSD]
[...]

Unread portion of the kernel message buffer:

[not ascii; here is a hexdump]

                     51  c1 00 40 09 28 18 6c 03
c1 08 3c 03 c1 38 01 03  c1 50 95 03 c1 44 44 51
c1 00 a0 06 08 00 00 00  00 74 07 37 c1 d0 e5 02
c1 28 f9 02 c1 4c 4b 51  c1 00 00 12 28 d0 52 03
c1 f4 48 3c c1 c0 7c 03  c1 a0 f6 02 c1 18 43 51
c1 00 20 07 28 00 00 00  00 ac 37 35 c1 08 0d 0a
03 c1 88 40 03 c1 78 4c  51 c1 00 30 0f 28 d8 c3
03 c1 18 e8 02 c1 e0 e4  02 c1 20 02 03 c1 c8 47
51 c1 00 a0 13 08 00 00  00 00 4c 13 3b c1 a0 28
03 c1 28 60 03 c1 18 43  51 c1 00 b0 bf bf 00 00
00 00 5c 0c 35 c1 58 0b  03 c1 d0 43 03 c1 c8 47
51 c1 00 50 17 08 00 00  00 00 e4 d0 36 c1 d0 4b
03 c1 00 1d 03 c1 a4 dd  94 c1 00

#0  doadump () at pcpu.h:165
165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) bt f
#0  doadump () at pcpu.h:165
No locals.
#1  0xc04fae3e in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:399
	first_buf_printf = 1
#2  0xc04fb104 in panic (fmt=0xc069ed40 "sbdrop")
    at /freebsd/src/sys/kern/kern_shutdown.c:555
	td = (struct thread *) 0xc194ac00
	bootopt = 260
	newpanic = 1
	ap = 0xc194ac00 "[binary]"
	buf = "sbdrop", '\0' <repeats 249 times>
#3  0xc05378b8 in sbdrop_locked (sb=0xcf612b50, len=17)
    at /freebsd/src/sys/kern/uipc_socket2.c:1157
	m = (struct mbuf *) 0x0
	next = (struct mbuf *) 0x0
#4  0xc05377ce in sbflush_locked (sb=0xcf612b50)
    at /freebsd/src/sys/kern/uipc_socket2.c:1124
No locals.
#5  0xc0536d49 in sbrelease_locked (sb=0xcf612b50, so=0x0)
    at /freebsd/src/sys/kern/uipc_socket2.c:559
No locals.
#6  0xc0536db1 in sbrelease (sb=0xcf612b50, so=0xc1908b20)
    at /freebsd/src/sys/kern/uipc_socket2.c:572
No locals.
#7  0xc0534921 in sorflush (so=0xc1908b20)
    at /freebsd/src/sys/kern/uipc_socket.c:1480
	sb = (struct sockbuf *) 0xc1908b70
	pr = (struct protosw *) 0xc06d8b14
	asb = {sb_sel = {si_thrlist = {tqe_next = 0x0, tqe_prev = 0x0}, 
    si_thread = 0x0, si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0, 
      kl_unlock = 0, kl_locked = 0, kl_lockarg = 0x0}, si_flags = 0}, 
  sb_mtx = {mtx_object = {lo_class = 0xc06cf004, 
      lo_name = 0xc069ecad "so_rcv", lo_type = 0xc069ecad "so_rcv", 
      lo_flags = 196608, lo_list = {tqe_next = 0x0, tqe_prev = 0x0}, 
      lo_witness = 0x0}, mtx_lock = 3247746048, mtx_recurse = 0}, 
  sb_state = 0, sb_mb = 0x0, sb_mbtail = 0x0, sb_lastrecord = 0x0, sb_cc = 17, 
  sb_hiwat = 42080, sb_mbcnt = 4294964992, sb_mbmax = 262144, 
  sb_ctl = 4294967280, sb_lowat = 1, sb_timeo = 0, sb_flags = 64}
#8  0xc0532cbb in sofree (so=0xc1908b20)
    at /freebsd/src/sys/kern/uipc_socket.c:406
	head = (struct socket *) 0x0
#9  0xc0532fe9 in soclose (so=0xc1908b20)
    at /freebsd/src/sys/kern/uipc_socket.c:484
	error = 0
#10 0xc0522e6b in soo_close (fp=0xc198ea20, td=0xc194ac00)
    at /freebsd/src/sys/kern/sys_socket.c:317
	error = 0
	so = (struct socket *) 0x0
#11 0xc04dc0d4 in fdrop_locked (fp=0xc198ea20, td=0xc194ac00) at file.h:289
	error = 0
#12 0xc04dc025 in fdrop (fp=0xc198ea20, td=0xc194ac00)
    at /freebsd/src/sys/kern/kern_descrip.c:2101
No locals.
#13 0xc04da653 in closef (fp=0xc198ea20, td=0xc194ac00)
    at /freebsd/src/sys/kern/kern_descrip.c:1921
	vp = (struct vnode *) 0xc198ea20
	lf = {l_start = -4580996068436530020, l_len = 23122899, 
  l_pid = -370322744, l_type = 599, l_whence = 0}
	fdtol = (struct filedesc_to_leader *) 0xbe24ecff
	fdp = (struct filedesc *) 0xc1ca2400
#14 0xc04d7a81 in close (td=0xc194ac00, uap=0x0)
    at /freebsd/src/sys/kern/kern_descrip.c:1004
	fdp = (struct filedesc *) 0xc1ca2400
	fp = (struct file *) 0xc198ea20
	fd = 3
	error = -1047221248
	holdleaders = 0
#15 0xc0662dbb in syscall (frame=
      {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 0, tf_esi = 673886912, tf_ebp = -1077941608, tf_isp = -815714972, tf_ebx = 673809636, tf_edx = 0, tf_ecx = 0, tf_eax = 6, tf_trapno = 0, tf_err = 2, tf_eip = 673286099, tf_cs = 51, tf_eflags = 534, tf_esp = -1077941636, tf_ss = 59})
    at /freebsd/src/sys/i386/i386/trap.c:976
	params = 0xbfbfea80 <Address 0xbfbfea80 out of bounds>
	callp = (struct sysent *) 0xc06ca6e8
	td = (struct thread *) 0xc194ac00
	p = (struct proc *) 0xc1a7f624
	orig_tf_eflags = 534
	sticks = 12
	error = 0
	narg = 1
	args = {3, -1066484000, 152949657, -815715028, -1067035982, 
  -1066484000, -815715020, 672605572}
	code = 6
#16 0xc06520cf in Xint0x80_syscall ()
    at /freebsd/src/sys/i386/i386/exception.s:200
No locals.
#17 0x00000033 in ?? ()
No symbol table info available.
Previous frame inner to this frame (corrupt stack?)
--- panic-sbdrop-2006-03-12 ends here ---


>How-To-Repeat:

unknown

>Fix:

unknown


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Tue Mar 14 12:20:30 UTC 2006 
Responsible-Changed-Why:  
Grab ownership of this PR.  I've been tracking a number of related reports 
for a bug that has the same symptoms as this for over a year now, and it 
has proved very hard to track down. I have a fairly large set of changes 
outstanding against CVS HEAD that likely resolve this, or at least, make 
it easier to track down, but it will probably be a while before they hit 
the RELENG_6 branch, as they will require extensive testing.  The problem 
is that by the time an assertion fires, the socket buffer memory 
corruption has long since occurred. 

If you're able to (relatively) easily reproduce the panic, we can likely 
make some progress.  The first thing to do, performance permitting, is 
to turn on INVARIANTS, INVARIANT_SUPPORT, and SOCKBUF_DEBUG options in 
the kernel.  These will have a noticeable performance impact, and I don't 
know if that is compatible with your workload.  Also, they change the 
timing of the socket code significantly, so may cause the race condition 
to close, meaning we can't reproduce it with the debugging options on. 
However, it's worth a try.  BTW, are you using IPv6 at all on the box? 

Sorry I can't be more helpful, other than to say that we know there's an 
issue, and we've invested quite a bit of time trying to track it down 
(and thus far failing), and now in redesigning the code in quetion to 
avoid the problems that are likely the cause, but that will take a bit 
to come to fruition.  Further debugging to see if we can identify the 
specific cause would be good, if it's possible! 


http://www.freebsd.org/cgi/query-pr.cgi?pr=94433 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Eric van Gyzen <eric@vangyzen.net>
Cc: FreeBSD-gnats-submit@FreeBSD.org, rwatson@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Wed, 15 Mar 2006 19:00:15 +0300

   Eric,
 
   just for the record... Is ng_ksocket(4) used in any way
 on the box in question?
 
 -- 
 Totus tuus, Glebius.
 GLEBIUS-RIPN GLEB-RIPE

From: Eric van Gyzen <eric@vangyzen.net>
To: Gleb Smirnoff <glebius@FreeBSD.org>
Cc: FreeBSD-gnats-submit@FreeBSD.org,  rwatson@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Wed, 15 Mar 2006 11:04:15 -0600

 Gleb Smirnoff wrote:
 > 
 >   just for the record... Is ng_ksocket(4) used in any way
 > on the box in question?
 
 No.  Netgraph is not used at all.

From: Eric van Gyzen <eric@vangyzen.net>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Sat, 18 Mar 2006 10:41:42 -0600

 For the record:
 
 --------------------------------------------------------------
 
 Robert Watson wrote:
 > If you're able to (relatively) easily reproduce the panic, we can likely
 > make some progress.
 
 Unfortunately, I have no idea how to reproduce the panic.  All I can do
 is wait.
 
 > The first thing to do, performance permitting, is
 > to turn on INVARIANTS, INVARIANT_SUPPORT, and SOCKBUF_DEBUG options in
 > the kernel.  These will have a noticeable performance impact, and I don't
 > know if that is compatible with your workload.
 
 Done.  This machine has plenty of power for its purpose.
 
 > BTW, are you using IPv6 at all on the box?
 
 No.  The kernel has IPv6 support, and every interface has an
 auto-configured link-local address, but I'm not actively using IPv6.
 

From: Eric van Gyzen <eric@vangyzen.net>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Sat, 18 Mar 2006 10:51:09 -0600

 I have never been so happy to see a kernel panic.  :)
 
 Here is a backtrace after enabling INVARIANTS, INVARIANT_SUPPORT,
 and SOCKBUF_DEBUG.  I hope it helps.
 
 Of course, let me know what else you need.
 
 ###################################################################
 
 $ kgdb kernel.debug /var/crash/vmcore-m_tag_delete_chain-2006-03-18
 GNU gdb 6.1.1 [FreeBSD]
 [...]
 
 Unread portion of the kernel message buffer:
 [not ascii]
 
 #0  doadump () at pcpu.h:165
 165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt f
 #0  doadump () at pcpu.h:165
 No locals.
 #1  0xc04f53f0 in boot (howto=260) at 
 /freebsd/src/sys/kern/kern_shutdown.c:399
 	first_buf_printf = 1
 #2  0xc04f569b in panic (fmt=0xc0677f20 "%s")
      at /freebsd/src/sys/kern/kern_shutdown.c:555
 	td = (struct thread *) 0xc110ea80
 	bootopt = 260
 	newpanic = 1
 	ap = 0xcbdf9b40 ""
 	buf = "page fault", '\0' <repeats 245 times>
 #3  0xc064af42 in trap_fatal (frame=0xcbdf9bc4, eva=3735929054)
      at /freebsd/src/sys/i386/i386/trap.c:831
 	code = 40
 	type = 12
 	ss = 40
 	esp = 0
 	softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27,
                     ssd_dpl = 0, ssd_p = 1, ssd_xx = 0, ssd_xx1 = 0,
                     ssd_def32 = 1, ssd_gran = 1}
 #4  0xc064ac77 in trap_pfault (frame=0xcbdf9bc4, usermode=0,
                                 eva=3735929054)
      at /freebsd/src/sys/i386/i386/trap.c:742
 	va = 3735928832
 	vm = (struct vmspace *) 0x0
 	map = 0xc0c43000
 	rv = 1
 	ftype = 1 '\001'
 	td = (struct thread *) 0xc110ea80
 	p = (struct proc *) 0xc11a1624
 #5  0xc064a8e1 in trap (frame=
        {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -1060839808,
         tf_esi = -1053859584, tf_ebp = -874537972, tf_isp = -874538000,
         tf_ebx = -559038242, tf_edx = -1053859584, tf_ecx = 4,
         tf_eax = 0,
         tf_trapno = 12, tf_err = 0, tf_eip = -1068325284, tf_cs = 32,
         tf_eflags = 590470, tf_esp = -1053859584, tf_ss = -1054635716})
      at /freebsd/src/sys/i386/i386/trap.c:432
 	td = (struct thread *) 0xc110ea80
 	p = (struct proc *) 0xc11a1624
 	sticks = 2
 	i = 0
 	ucode = 0
 	type = 12
 	code = 0
 	eva = 3735929054
 #6  0xc063abea in calltrap ()
      at /freebsd/src/sys/i386/i386/exception.s:139
 No locals.
 #7  0xc052a65c in m_tag_delete_chain (m=0xc12f6100, t=0x0)
      at /freebsd/src/sys/kern/uipc_mbuf2.c:354
 	p = (struct m_tag *) 0xdeadc0de
 	q = (struct m_tag *) 0x0
 #8  0xc04eca55 in mb_dtor_mbuf (mem=0xc12f6100, size=256, arg=0x0)
      at /freebsd/src/sys/kern/kern_mbuf.c:244
 No locals.
 #9  0xc05e978c in uma_zfree_arg (zone=0xc0c4de80, item=0xc12f6100,
                                   udata=0x0)
      at /freebsd/src/sys/vm/uma_core.c:2279
 	keg = 0xc0c293c0
 	cache = 0xc123893c
 	bucket = 0xdeadc0de
 	bflags = 0
 	cpu = 0
 #10 0xc0528762 in m_freem (mb=0xc12f6100) at uma.h:303
 No locals.
 #11 0xc046a3a0 in ath_tx_processq (sc=0xc122f000, txq=0xc1230170)
      at /freebsd/src/sys/dev/ath/if_ath.c:3692
 	ah = (struct ath_hal *) 0xc1231000
 	ic = (struct ieee80211com *) 0xc122f1ac
 	bf = (struct ath_buf *) 0xc123893c
 	ds = (struct ath_desc *) 0xcea97c68
 	ds0 = (struct ath_desc *) 0xcea97c68
 	ni = (struct ieee80211_node *) 0xc191d000
 	an = (struct ath_node *) 0xc191d000
 	sr = 0
 	lr = -1053859584
 	pri = 0
 	status = 3247558656
 	__func__ = "ath_tx_processq"
 #12 0xc046a48c in ath_tx_proc_q0123 (arg=0xc122f000, npending=1)
      at /freebsd/src/sys/dev/ath/if_ath.c:3737
 	ifp = (struct ifnet *) 0xc122b400
 #13 0xc0513262 in taskqueue_run (queue=0xc11a6380)
      at /freebsd/src/sys/kern/subr_taskqueue.c:217
 	task = (struct task *) 0xc123040c
 	owned = 0
 	pending = 1
 #14 0xc051334a in taskqueue_swi_run (dummy=0x0)
      at /freebsd/src/sys/kern/subr_taskqueue.c:252
 No locals.
 #15 0xc04e30b0 in ithread_loop (arg=0xc11a6300)
      at /freebsd/src/sys/kern/kern_intr.c:547
 	ithd = (struct ithd *) 0xc11a6300
 	ih = (struct intrhand *) 0xc119ee00
 	td = (struct thread *) 0xc110ea80
 	p = (struct proc *) 0xc11a1624
 	count = 0
 	warned = 0
 	__func__ = "ithread_loop"
 #16 0xc04e2520 in fork_exit (callout=0xc04e2f6c <ithread_loop>,
                               arg=0xc11a6300, frame=0xcbdf9d38)
      at /freebsd/src/sys/kern/kern_fork.c:789
 	p = (struct proc *) 0xc11a1624
 	td = (struct thread *) 0xc12f6100
 #17 0xc063ac4c in fork_trampoline ()
      at /freebsd/src/sys/i386/i386/exception.s:208
 No locals.

From: Eric van Gyzen <eric@vangyzen.net>
To: Robert Watson <rwatson@FreeBSD.org>, 
 freebsd-gnats-submit@FreeBSD.org
Cc: freebsd-bugs@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Sun, 26 Mar 2006 09:18:44 -0600

 Here is a backtrace from another panic with INVARIANTS, 
 INVARIANT_SUPPORT, and SOCKBUF_DEBUG in the kernel.
 
 Please feel free to ask for more information.
 
 =============================================================
 
 $ kgdb kernel.debug /var/crash/vmcore.3
 GNU gdb 6.1.1 [FreeBSD]
 [...]
 
 Unread portion of the kernel message buffer:
 [not ascii]
 
 #0  doadump () at pcpu.h:165
 165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt f
 #0  doadump () at pcpu.h:165
 No locals.
 #1  0xc04f53f0 in boot (howto=260)
      at /freebsd/src/sys/kern/kern_shutdown.c:399
          first_buf_printf = 1
 #2  0xc04f569b in panic (fmt=0xc0677f20 "%s")
      at /freebsd/src/sys/kern/kern_shutdown.c:555
          td = (struct thread *) 0xc110ea80
          bootopt = 260
          newpanic = 1
          ap = 0xcbdf9b40 ""
          buf = "page fault", '\0' <repeats 245 times>
 #3  0xc064af42 in trap_fatal (frame=0xcbdf9bc4, eva=3735929054)
      at /freebsd/src/sys/i386/i386/trap.c:831
          code = 40
          type = 12
          ss = 40
          esp = 0
          softseg = {ssd_base = 0, ssd_limit = 1048575, ssd_type = 27,
                     ssd_dpl = 0, ssd_p = 1, ssd_xx = 0, ssd_xx1 = 0,
                     ssd_def32 = 1, ssd_gran = 1}
 #4  0xc064ac77 in trap_pfault (frame=0xcbdf9bc4, usermode=0,
                                 eva=3735929054)
      at /freebsd/src/sys/i386/i386/trap.c:742
          va = 3735928832
          vm = (struct vmspace *) 0x0
          map = 0xc0c43000
          rv = 1
          ftype = 1 '\001'
          td = (struct thread *) 0xc110ea80
          p = (struct proc *) 0xc11a1624
 #5  0xc064a8e1 in trap (frame=
        {tf_fs = 8, tf_es = 40, tf_ds = 40, tf_edi = -1060839808,
         tf_esi = -1050224640, tf_ebp = -874537972, tf_isp = -874538000,
         tf_ebx = -559038242, tf_edx = -1050224640, tf_ecx = 4,
         tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1068325284,
         tf_cs = 32, tf_eflags = 590470, tf_esp = -1050224640,
         tf_ss = -1054635368})
      at /freebsd/src/sys/i386/i386/trap.c:432
          td = (struct thread *) 0xc110ea80
          p = (struct proc *) 0xc11a1624
          sticks = 2
          i = 0
          ucode = 0
          type = 12
          code = 0
          eva = 3735929054
 #6  0xc063abea in calltrap ()
      at /freebsd/src/sys/i386/i386/exception.s:139
 No locals.
 #7  0xc052a65c in m_tag_delete_chain (m=0xc166d800, t=0x0)
      at /freebsd/src/sys/kern/uipc_mbuf2.c:354
          p = (struct m_tag *) 0xdeadc0de
          q = (struct m_tag *) 0x0
 #8  0xc04eca55 in mb_dtor_mbuf (mem=0xc166d800, size=256, arg=0x0)
      at /freebsd/src/sys/kern/kern_mbuf.c:244
 No locals.
 #9  0xc05e978c in uma_zfree_arg (zone=0xc0c4de80, item=0xc166d800,
                                   udata=0x0)
      at /freebsd/src/sys/vm/uma_core.c:2279
          keg = 0xc0c293c0
          cache = 0xc1238a98
          bucket = 0xdeadc0de
          bflags = 0
          cpu = 0
 #10 0xc0528762 in m_freem (mb=0xc166d800) at uma.h:303
 No locals.
 #11 0xc046a3a0 in ath_tx_processq (sc=0xc122f000, txq=0xc1230170)
      at /freebsd/src/sys/dev/ath/if_ath.c:3692
          ah = (struct ath_hal *) 0xc1231000
          ic = (struct ieee80211com *) 0xc122f1ac
          bf = (struct ath_buf *) 0xc1238a98
          ds = (struct ath_desc *) 0xcea98190
          ds0 = (struct ath_desc *) 0xcea98190
          ni = (struct ieee80211_node *) 0xc173c000
          an = (struct ath_node *) 0xc173c000
          sr = 0
          lr = -1050224640
          pri = 0
          status = 3245588480
          __func__ = "ath_tx_processq"
 #12 0xc046a48c in ath_tx_proc_q0123 (arg=0xc122f000, npending=1)
      at /freebsd/src/sys/dev/ath/if_ath.c:3737
          ifp = (struct ifnet *) 0xc122b400
 #13 0xc0513262 in taskqueue_run (queue=0xc11a6380)
      at /freebsd/src/sys/kern/subr_taskqueue.c:217
          task = (struct task *) 0xc123040c
          owned = 0
          pending = 1
 #14 0xc051334a in taskqueue_swi_run (dummy=0x0)
      at /freebsd/src/sys/kern/subr_taskqueue.c:252
 No locals.
 #15 0xc04e30b0 in ithread_loop (arg=0xc11a6300)
      at /freebsd/src/sys/kern/kern_intr.c:547
          ithd = (struct ithd *) 0xc11a6300
          ih = (struct intrhand *) 0xc119ee00
          td = (struct thread *) 0xc110ea80
          p = (struct proc *) 0xc11a1624
          count = 0
          warned = 0
          __func__ = "ithread_loop"
 #16 0xc04e2520 in fork_exit (callout=0xc04e2f6c <ithread_loop>,
                               arg=0xc11a6300, frame=0xcbdf9d38)
      at /freebsd/src/sys/kern/kern_fork.c:789
          p = (struct proc *) 0xc11a1624
          td = (struct thread *) 0xc166d800
 #17 0xc063ac4c in fork_trampoline ()
      at /freebsd/src/sys/i386/i386/exception.s:208
 No locals.

From: Sam Leffler <sam@errno.com>
To: bug-followup@FreeBSD.org, eric@vangyzen.net
Cc:  
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Sun, 26 Mar 2006 10:07:03 -0800

 Robert asked me to look at this since the panic involves ath.  It's 
 unclear if this could be driver-related but I'd note that there has been 
 much change in the ath+net80211 code, including fixes for some race 
 conditions that might be related.  The trace looks as though the mbuf 
 (chain) was free'd out from underneath ath while the frame sat on the tx 
 queue waiting for tx complete processing.  This processing may be 
 deferred for a long time because it happens via a taskqueue and in the 
 code being used the q is managed by the swi thread which may be blocked 
 by other (slow) operations.
 
 If you can characterize when this panic occurs (e.g. idle, high volume 
 wireless traffic, heavy concurrent system activity) it might be useful. 
   Also no network configuration has been provided.
 
 I would also try to simplify the kernel config if possible.  I note in 
 particular that ipfw is being used with divert sockets and that 
 combination has always worried me wrt locking.  There is also IPv6 
 configured which should mean the network stack is running w/ Giant which 
 makes this again more difficult to analyze.
 
 
 	Sam

From: Eric van Gyzen <eric@vangyzen.net>
To: Sam Leffler <sam@errno.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Mon, 27 Mar 2006 10:08:59 -0600

 Sam Leffler wrote:
 > Robert asked me to look at this since the panic involves ath.  It's 
 > unclear if this could be driver-related but I'd note that there has been 
 > much change in the ath+net80211 code, including fixes for some race 
 > conditions that might be related.
 
 I noticed.  I'll update to RELENG_6 soon (unless you want to keep 
 hunting this bug).
 
 > If you can characterize when this panic occurs (e.g. idle, high volume 
 > wireless traffic, heavy concurrent system activity) it might be useful. 
 
 I wish I could.
 
 >  Also no network configuration has been provided.
 
 Interfaces fxp0 (public), rl0 (private), and ath0 (private).  I am NOT 
 bridging anything.  ath0 is configured with hostap, pureg, and WEP 
 (because WPA wasn't reliable).  Note that this panic did not occur while 
 using WPA; it has only happened after switching to WEP.  (But since OS X 
 now supports WPA2, I'm going to try WPA2.)
 
 > I would also try to simplify the kernel config if possible.  I note in 
 > particular that ipfw is being used with divert sockets and that 
 > combination has always worried me wrt locking.
 
 I've been wanting to try pf anyway.
 
 > There is also IPv6 
 > configured which should mean the network stack is running w/ Giant which 
 > makes this again more difficult to analyze.
 
 I'll remove it.
 
 I have just said that I'm going to make four changes to this system, 
 which could confound this bug-hunt.  I'll first upgrade to RELENG_6 and 
 wait to see if the changes in ath and net80211 fix the problem.
 
 Thanks for your help.
 
 Eric

From: Sam Leffler <sam@errno.com>
To: Eric van Gyzen <eric@vangyzen.net>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Mon, 27 Mar 2006 09:30:44 -0800

 Eric van Gyzen wrote:
 > Sam Leffler wrote:
 >> Robert asked me to look at this since the panic involves ath.  It's 
 >> unclear if this could be driver-related but I'd note that there has 
 >> been much change in the ath+net80211 code, including fixes for some 
 >> race conditions that might be related.
 > 
 > I noticed.  I'll update to RELENG_6 soon (unless you want to keep 
 > hunting this bug).
 
 I would prefer you update but Robert may feel otherwise.
 
 > 
 >> If you can characterize when this panic occurs (e.g. idle, high volume 
 >> wireless traffic, heavy concurrent system activity) it might be useful. 
 > 
 > I wish I could.
 > 
 >>  Also no network configuration has been provided.
 > 
 > Interfaces fxp0 (public), rl0 (private), and ath0 (private).  I am NOT 
 > bridging anything.  ath0 is configured with hostap, pureg, and WEP 
 > (because WPA wasn't reliable).  Note that this panic did not occur while 
 > using WPA; it has only happened after switching to WEP.  (But since OS X 
 > now supports WPA2, I'm going to try WPA2.)
 
 ifconfig output is more useful that a general description.
 
 > 
 >> I would also try to simplify the kernel config if possible.  I note in 
 >> particular that ipfw is being used with divert sockets and that 
 >> combination has always worried me wrt locking.
 > 
 > I've been wanting to try pf anyway.
 > 
 >> There is also IPv6 configured which should mean the network stack is 
 >> running w/ Giant which makes this again more difficult to analyze.
 > 
 > I'll remove it.
 > 
 > I have just said that I'm going to make four changes to this system, 
 > which could confound this bug-hunt.  I'll first upgrade to RELENG_6 and 
 > wait to see if the changes in ath and net80211 fix the problem.
 > 
 > Thanks for your help.
 > 
 > Eric
 > 
 > 
 

From: Eric van Gyzen <eric@vangyzen.net>
To: Sam Leffler <sam@errno.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Mon, 27 Mar 2006 11:50:39 -0600

 Sam Leffler wrote:
 > Eric van Gyzen wrote:
 > >Sam Leffler wrote:
 > >> Also no network configuration has been provided.
 > >
 > >Interfaces fxp0 (public), rl0 (private), and ath0 (private).  I am NOT 
 > >bridging anything.  ath0 is configured with hostap, pureg, and WEP 
 > >(because WPA wasn't reliable).  Note that this panic did not occur while 
 > >using WPA; it has only happened after switching to WEP.  (But since OS X 
 > >now supports WPA2, I'm going to try WPA2.)
 > 
 > ifconfig output is more useful that a general description.
 
 fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 	options=8<VLAN_MTU>
 	inet (public) netmask .... broadcast ....
 	inet6 fe80::2d0:b7ff:fe5a:cef6%fxp0 prefixlen 64 scopeid 0x1 
 	ether 00:d0:b7:5a:ce:f6
 	media: Ethernet autoselect (100baseTX <full-duplex>)
 	status: active
 ath0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 	inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
 	inet6 fe80::211:95ff:fe91:32f2%ath0 prefixlen 64 scopeid 0x2 
 	ether 00:11:95:91:32:f2
 	media: IEEE 802.11 Wireless Ethernet autoselect mode 11g <hostap>
 	status: associated
 	ssid MySSID channel 1 bssid 00:11:95:91:32:f2
 	authmode SHARED privacy ON deftxkey 1 wepkey 1:104-bit txpowmax 30 pureg
 	protmode CTS dtimperiod 1 bintval 100
 rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
 	options=8<VLAN_MTU>
 	inet 10.2.2.1 netmask 0xffffff00 broadcast 10.2.2.255
 	inet6 fe80::240:f4ff:fe2d:96ec%rl0 prefixlen 64 scopeid 0x3 
 	ether 00:40:f4:2d:96:ec
 	media: Ethernet autoselect (100baseTX <full-duplex>)
 	status: active

From: Robert Watson <rwatson@FreeBSD.org>
To: Sam Leffler <sam@errno.com>
Cc: Eric van Gyzen <eric@vangyzen.net>, bug-followup@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Mon, 27 Mar 2006 18:08:53 +0000 (GMT)

 On Mon, 27 Mar 2006, Sam Leffler wrote:
 
 > Eric van Gyzen wrote:
 >> Sam Leffler wrote:
 >>> Robert asked me to look at this since the panic involves ath.  It's 
 >>> unclear if this could be driver-related but I'd note that there has been 
 >>> much change in the ath+net80211 code, including fixes for some race 
 >>> conditions that might be related.
 >> 
 >> I noticed.  I'll update to RELENG_6 soon (unless you want to keep hunting 
 >> this bug).
 >
 > I would prefer you update but Robert may feel otherwise.
 
 Ditto.  I'd much rather be working from the latest source, as it both means we 
 won't be chasing a bug someone has already fixed, or generating fixes that 
 can't be applied to an active branch.
 
 Thanks,
 
 Robert N M Watson

From: Eric van Gyzen <eric@vangyzen.net>
To: Sam Leffler <sam@errno.com>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>,
	Eric van Gyzen <eric@vangyzen.net>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Fri, 7 Apr 2006 15:26:50 -0500

 Sam Leffler wrote:
 > Robert asked me to look at this since the panic involves ath.  It's 
 > unclear if this could be driver-related but I'd note that there has been 
 > much change in the ath+net80211 code, including fixes for some race 
 > conditions that might be related.
 
 Here is a panic dump after upgrading to RELENG_6 (and switching to WPA2-PSK).
 
 Apr 1 11:00 -- upgraded to RELENG_6
 Apr 5 21:00 -- switched from WEP to WPA2-PSK
 Apr 6 17:47 -- panic: ndx is 0
 
 No other changes were made to the machine.
 
 > I would also try to simplify the kernel config if possible.  I note in 
 > particular that ipfw is being used with divert sockets and that 
 > combination has always worried me wrt locking.
 
 Indeed, divert is involved in the following stack trace (FWIW).
 I'll switch to pf at my earliest convenience.
 
 > There is also IPv6 
 > configured which should mean the network stack is running w/ Giant which 
 > makes this again more difficult to analyze.
 
 I'll also remove IPv6 then, if you think multiple simultaneous changes are OK
 (though Murphy has taught me otherwise).
 
 I note that debug.mpsafenet=1.  Does "options INET6" imply running with Giant,
 regardless of debug.mpsafenet?
 
 As always, thanks for your help.
 
 -Eric
 
 
 [Non-ASCII characters were stripped from this dump with iconv(1).]
 
 $ kgdb kernel.debug /var/crash/vmcore.4
 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 [...]
 
 Unread portion of the kernel message buffer:
 panic: ndx is 0
 Uptime: 4d19h19m31s
 Dumping 255 MB (2 chunks)
   chunk 0: 1MB (159 pages) ... ok
   chunk 1: 255MB (65264 pages) 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
 
 #0  doadump () at pcpu.h:165
 165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt f
 #0  doadump () at pcpu.h:165
 No locals.
 #1  0xc04fa240 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 	first_buf_printf = 1
 #2  0xc04fa4eb in panic (fmt=0xc068be44 "ndx is %d")
     at /freebsd/src/sys/kern/kern_shutdown.c:558
 	td = (struct thread *) 0xc22bd300
 	bootopt = 260
 	newpanic = 1
 	ap = 0xcec85900 ""
 	buf = "ndx is 0", '\0' <repeats 247 times>
 #3  0xc0466bcb in ath_rate_findrate (sc=0xc2281000, an=0xc254a000, 
     shortPreamble=0, frameLen=104, rix=0x0, try0=0xcec859b0, 
     txrate=0xcec859b7 "t'")
     at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 	sn = (struct sample_node *) 0xc254a23c
 	ssc = (struct sample_softc *) 0xc2222250
 	ic = (struct ieee80211com *) 0x0
 	ndx = 0
 	size_bin = 0
 	mrr = 1
 	best_ndx = -825730640
 	change_rates = -1037511384
 	average_tx_time = 3260326460
 	__func__ = "ath_rate_findrate"
 #4  0xc046c7b8 in ath_tx_start (sc=0xc2281000, ni=0xc254a000, bf=0xc228d528, 
     m0=0xc2e99e00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 	ic = (struct ieee80211com *) 0xc2281230
 	ah = (struct ath_hal *) 0xc2283000
 	ifp = (struct ifnet *) 0xc2277400
 	cap = (const struct chanAccParams *) 0xc2281c5a
 	i = -1066478400
 	error = -1066478400
 	iswep = 64
 	ismcast = 0
 	ismrr = 0
 	keyix = 1
 	hdrlen = 32
 	pktlen = 104
 	try0 = 11
 	rix = 161 ''
 	txrate = 242 ''
 	ctsrate = 194 ''
 	cix = 255 ''
 	ds = (struct ath_desc *) 0xcec3f9c0
 	ds0 = (struct ath_desc *) 0xc057b062
 	txq = (struct ath_txq *) 0x0
 	wh = (struct ieee80211_frame *) 0xc2e99ea4
 	subtype = 0
 	flags = 1
 	ctsduration = 3469236592
 	atype = HAL_PKT_TYPE_NORMAL
 	rt = (const HAL_RATE_TABLE *) 0xc06ed4c0
 	shortPreamble = AH_FALSE
 	an = (struct ath_node *) 0xc254a000
 	m = (struct mbuf *) 0x0
 	pri = 0
 	__func__ = "ath_tx_start"
 #5  0xc046971c in ath_start (ifp=0xc2277400)
     at /freebsd/src/sys/dev/ath/if_ath.c:1268
 	sc = (struct ath_softc *) 0xc2281000
 	ah = (struct ath_hal *) 0xc2283000
 	ic = (struct ieee80211com *) 0xc2281230
 	ni = (struct ieee80211_node *) 0xc254a000
 	bf = (struct ath_buf *) 0xc228d528
 	m = (struct mbuf *) 0xc2e99e00
 	wh = (struct ieee80211_frame *) 0xc22774f8
 	__func__ = "ath_start"
 #6  0xc056001f in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 No locals.
 #7  0xc05612dc in ether_output_frame (ifp=0xc2277400, m=0xc2e99e00)
     at /freebsd/src/sys/net/if_ethersubr.c:406
 	len = 66
 	mflags = 2
 	rule = (struct ip_fw *) 0x0
 	error = 0
 #8  0xc05610f8 in ether_output (ifp=0xc2277400, m=0xc2e99e00, dst=0xcec85aac, 
     rt0=0x0) at /freebsd/src/sys/net/if_ethersubr.c:359
 	type = 8
 	error = -1024876866
 	hdrcmplt = 0
 	esrc = "\000\000jK"
 	edst = "\000\021$\235"
 	eh = (struct ether_header *) 0xc2e99ebe
 	loop_copy = 1
 	__func__ = "ether_output"
 #9  0xc058fcf8 in ip_output (m=0xc2e99e00, opt=0xc2e99ecc, ro=0xcec85aa8, 
     flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 	ip = (struct ip *) 0xc2e99ecc
 	ifp = (struct ifnet *) 0xc2277400
 	m0 = (struct mbuf *) 0xc2e99ecc
 	hlen = 20
 	len = -825730368
 	error = 0
 	dst = (struct sockaddr_in *) 0xcec85aac
 	ia = (struct in_ifaddr *) 0xc2626800
 	isbroadcast = 0
 	sw_csum = 1
 	iproute = {ro_rt = 0xc243e084, ro_dst = {sa_len = 16 '\020', 
     sa_family = 2 '\002', 
     sa_data = "\000\000\n\001\001\003\000\000\000\000\000\000\000"}}
 	odst = {s_addr = 1}
 	__func__ = "ip_output"
 #10 0xc058f228 in ip_forward (m=0xc2e99e00, srcrt=0)
     at /freebsd/src/sys/netinet/ip_input.c:1907
 	ip = (struct ip *) 0xc2e99ecc
 	ia = (struct in_ifaddr *) 0xc2626800
 	mcopy = (struct mbuf *) 0xc254f500
 	dest = {s_addr = 0}
 	error = 0
 	type = 0
 	code = 0
 	mtu = 0
 #11 0xc058de23 in ip_input (m=0xc2e99e00)
     at /freebsd/src/sys/netinet/ip_input.c:689
 	ip = (struct ip *) 0xc2e99ecc
 	ia = (struct in_ifaddr *) 0xc23b4200
 	ifa = (struct ifaddr *) 0x0
 	checkif = 0
 	hlen = 20
 	sum = 0
 	dchg = 0
 	odst = {s_addr = 50397450}
 	__func__ = "ip_input"
 #12 0xc0584257 in div_output (so=0xc243a9bc, m=0xc2e99e00, sin=0xc30ef210, 
     control=0x0) at /freebsd/src/sys/netinet/ip_divert.c:381
 	mtag = (struct m_tag *) 0x0
 	dt = (struct divert_tag *) 0xc2783870
 	error = 0
 #13 0xc05845e3 in div_send (so=0xc243a9bc, flags=0, m=0x0, nam=0xc30ef210, 
     control=0x0, td=0xc22bd300) at /freebsd/src/sys/netinet/ip_divert.c:506
 No locals.
 #14 0xc0530f87 in sosend (so=0xc243a9bc, addr=0xc30ef210, uio=0xcec85c40, 
     top=0xc2e99e00, control=0x0, flags=0, td=0xc22bd300)
     at /freebsd/src/sys/kern/uipc_socket.c:836
 	mp = (struct mbuf **) 0xc2e99e00
 	m = (struct mbuf *) 0xc2e99e00
 	space = 65584
 	len = 52
 	resid = 0
 	clen = 52
 	error = 0
 	dontroute = 0
 	atomic = 1
 #15 0xc0536480 in kern_sendit (td=0xc22bd300, s=3, mp=0xcec85cbc, flags=0, 
     control=0x0, segflg=UIO_USERSPACE)
     at /freebsd/src/sys/kern/uipc_syscalls.c:772
 	fp = (struct file *) 0xc23dce58
 	auio = {uio_iov = 0xcec85cb4, uio_iovcnt = 1, uio_offset = 52, 
   uio_resid = 0, uio_segflg = UIO_USERSPACE, uio_rw = UIO_WRITE, 
   uio_td = 0xc22bd300}
 	iov = (struct iovec *) 0x0
 	so = (struct socket *) 0xc243a9bc
 	i = 0
 	len = 52
 	error = 0
 	ktruio = (struct uio *) 0x0
 #16 0xc0536353 in sendit (td=0xc22bd300, s=3, mp=0xcec85cbc, flags=0)
     at /freebsd/src/sys/kern/uipc_syscalls.c:712
 	control = (struct mbuf *) 0x0
 	to = (struct sockaddr *) 0xc30ef210
 	error = 0
 #17 0xc05365b1 in sendto (td=0xc22bd300, uap=0x0)
     at /freebsd/src/sys/kern/uipc_syscalls.c:830
 	msg = {msg_name = 0xc30ef210, msg_namelen = 16, msg_iov = 0xcec85cb4, 
   msg_iovlen = 1, msg_control = 0x0, msg_controllen = 2, msg_flags = 0}
 	aiov = {iov_base = 0xbfbdebd4, iov_len = 0}
 	error = 0
 #18 0xc0659a5f in syscall (frame=
       {tf_fs = 59, tf_es = 59, tf_ds = 59, tf_edi = 1, tf_esi = 52,
        tf_ebp = -1078006856, tf_isp = -825729692, tf_ebx = -1078072416,
        tf_edx = -1078132736, tf_ecx = -1078072416, tf_eax = 133,
        tf_trapno = 0, tf_err = 2, tf_eip = 672268455, tf_cs = 51,
        tf_eflags = 662, tf_esp = -1078072564, tf_ss = 59})
     at /freebsd/src/sys/i386/i386/trap.c:981
 	params = 0xbfbdeb10 <Address 0xbfbdeb10 out of bounds>
 	callp = (struct sysent *) 0xc06d193c
 	td = (struct thread *) 0xc22bd300
 	p = (struct proc *) 0xc239520c
 	orig_tf_eflags = 662
 	sticks = 3639
 	error = 0
 	narg = 6
 	args = {3, -1078072416, 52, 0, -1078072432, 16, -825729740, 1}
 	code = 133
 #19 0xc064929f in Xint0x80_syscall ()
     at /freebsd/src/sys/i386/i386/exception.s:200
 No locals.
 #20 0x00000033 in ?? ()
 No symbol table info available.
 Previous frame inner to this frame (corrupt stack?)
 (kgdb) f 3
 #3  0xc0466bcb in ath_rate_findrate (sc=0xc2281000, an=0xc254a000, 
     shortPreamble=0, frameLen=104, rix=0x0, try0=0xcec859b0, 
     txrate=0xcec859b7 "t'")
     at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 350		KASSERT(ndx >= 0 && ndx < sn->num_rates, ("ndx is %d", ndx));
 (kgdb) p *sn
 $1 = {static_rate_ndx = 0, num_rates = 0, rates = {{rate = 0, rix = 0, 
       rateCode = 0, shortPreambleRateCode = 0} <repeats 15 times>}, stats = {{{
         average_tx_time = 0, successive_failures = 0, tries = 0, 
         total_packets = 0, packets_acked = 0, perfect_tx_time = 0, 
         last_tx = 0} <repeats 15 times>}, {{average_tx_time = 0, 
         successive_failures = 0, tries = 0, total_packets = 0, 
         packets_acked = 0, perfect_tx_time = 0, 
         last_tx = 0} <repeats 15 times>}, {{average_tx_time = 0, 
         successive_failures = 0, tries = 0, total_packets = 0, 
         packets_acked = 0, perfect_tx_time = 0, 
         last_tx = 0} <repeats 15 times>}}, last_sample_ndx = {0, 0, 0}, 
   current_sample_ndx = {0, 0, 0}, packets_sent = {0, 0, 0}, current_rate = {0, 
     0, 0}, packets_since_switch = {0, 0, 0}, ticks_since_switch = {0, 0, 0}, 
   packets_since_sample = {0, 0, 0}, sample_tt = {0, 0, 0}}

From: Eric van Gyzen <eric@vangyzen.net>
To: Sam Leffler <sam@errno.com>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Mon, 24 Apr 2006 20:38:16 -0500

 Eric van Gyzen wrote:
 > Sam Leffler wrote:
 > > I would also try to simplify the kernel config if possible.  I note in 
 > > particular that ipfw is being used with divert sockets and that 
 > > combination has always worried me wrt locking.
 > 
 > Indeed, divert is involved in the following stack trace (FWIW).
 > I'll switch to pf at my earliest convenience.
 > 
 > > There is also IPv6 
 > > configured which should mean the network stack is running w/ Giant which 
 > > makes this again more difficult to analyze.
 > 
 > I'll also remove IPv6 then,
 
 Here is another crash dump after removing IPv6 and switching
 from ipfw+divert to pf.
 
 Sources are RELENG_6_1 from 20 April.
 
 FWIW: I get sporadic "ath0: device timeout" messages, though I do not have
 enough data to correlate them with any known events (such as heavy usage).
 
 How can I help?
 
 ==============================================================================
 
 $ kgdb kernel.debug /var/crash/vmcore.8
 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 [...]
 
 Unread portion of the kernel message buffer:
 panic: ndx is 0
 Uptime: 21h21m50s
 Dumping 255 MB (2 chunks)
   chunk 0: 1MB (159 pages) ... ok
   chunk 1: 255MB (65264 pages) 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
 
 #0  doadump () at pcpu.h:165
 165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt
 #0  doadump () at pcpu.h:165
 #1  0xc0515a24 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 #2  0xc0515ccf in panic (fmt=0xc067d3d3 "ndx is %d")
     at /freebsd/src/sys/kern/kern_shutdown.c:558
 #3  0xc0485a6b in ath_rate_findrate (sc=0xc2281000, an=0xc25c4000, 
     shortPreamble=0, frameLen=92, rix=0x0, try0=0xcbf9faf0, 
     txrate=0xcbf9faf7 "")
     at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 #4  0xc048ae10 in ath_tx_start (sc=0xc2281000, ni=0xc25c4000, bf=0xc228c9d4, 
     m0=0xc26f3d00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 #5  0xc0488390 in ath_start (ifp=0xc2277400)
     at /freebsd/src/sys/dev/ath/if_ath.c:1268
 #6  0xc057b483 in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 #7  0xc057c738 in ether_output_frame (ifp=0xc2277400, m=0xc2743b00)
     at /freebsd/src/sys/net/if_ethersubr.c:406
 #8  0xc057c554 in ether_output (ifp=0xc2277400, m=0xc2743b00, dst=0xcbf9fbec, 
     rt0=0xc2431528) at /freebsd/src/sys/net/if_ethersubr.c:359
 #9  0xc05a2d14 in ip_output (m=0xc2743b00, opt=0xc28d6820, ro=0xcbf9fbe8, 
     flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 #10 0xc05a2244 in ip_forward (m=0xc2743b00, srcrt=1)
     at /freebsd/src/sys/netinet/ip_input.c:1907
 #11 0xc05a0e3f in ip_input (m=0xc2743b00)
     at /freebsd/src/sys/netinet/ip_input.c:689
 #12 0xc057dff6 in netisr_processqueue (ni=0xc06f6a98)
     at /freebsd/src/sys/net/netisr.c:236
 #13 0xc057e1de in swi_net (dummy=0x0) at /freebsd/src/sys/net/netisr.c:349
 #14 0xc0502d6e in ithread_execute_handlers (p=0xc216a830, ie=0xc2168500)
     at /freebsd/src/sys/kern/kern_intr.c:684
 #15 0xc0502e9e in ithread_loop (arg=0xc21556d0)
     at /freebsd/src/sys/kern/kern_intr.c:767
 #16 0xc0501fc0 in fork_exit (callout=0xc0502e38 <ithread_loop>, 
     arg=0xc21556d0, frame=0xcbf9fd38) at /freebsd/src/sys/kern/kern_fork.c:805
 #17 0xc063a6ac in fork_trampoline ()
     at /freebsd/src/sys/i386/i386/exception.s:208
 (kgdb) bt f
 #0  doadump () at pcpu.h:165
 No locals.
 #1  0xc0515a24 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 	first_buf_printf = 1
 #2  0xc0515ccf in panic (fmt=0xc067d3d3 "ndx is %d")
     at /freebsd/src/sys/kern/kern_shutdown.c:558
 	td = (struct thread *) 0xc216b900
 	bootopt = 260
 	newpanic = 1
 	ap = 0xcbf9fa40 ""
 	buf = "ndx is 0", '\0' <repeats 247 times>
 #3  0xc0485a6b in ath_rate_findrate (sc=0xc2281000, an=0xc25c4000, 
     shortPreamble=0, frameLen=92, rix=0x0, try0=0xcbf9faf0, 
     txrate=0xcbf9faf7 "")
     at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 	sn = (struct sample_node *) 0xc25c423c
 	ssc = (struct sample_softc *) 0xc2221090
 	ic = (struct ieee80211com *) 0x0
 	ndx = 0
 	size_bin = 0
 	mrr = 1
 	best_ndx = -872809744
 	change_rates = -826029344
 	average_tx_time = 3260826172
 	__func__ = "ath_rate_findrate"
 #4  0xc048ae10 in ath_tx_start (sc=0xc2281000, ni=0xc25c4000, bf=0xc228c9d4, 
     m0=0xc26f3d00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 	ic = (struct ieee80211com *) 0xc2281230
 	ah = (struct ath_hal *) 0xc2283000
 	ifp = (struct ifnet *) 0xc2277400
 	cap = (const struct chanAccParams *) 0xc2281c5a
 	i = -1066561312
 	error = -1066561312
 	iswep = 64
 	ismcast = 0
 	ismrr = 0
 	keyix = 2
 	hdrlen = 32
 	pktlen = 92
 	try0 = 11
 	rix = 145 '\221'
 	txrate = 242 ''
 	ctsrate = 194 ''
 	cix = 255 ''
 	ds = (struct ath_desc *) 0xcec3cae0
 	ds0 = (struct ath_desc *) 0xc0595d92
 	txq = (struct ath_txq *) 0x0
 	wh = (struct ieee80211_frame *) 0xc26f3d36
 	subtype = 0
 	flags = 1
 	ctsduration = 3227081275
 	atype = HAL_PKT_TYPE_NORMAL
 	rt = (const HAL_RATE_TABLE *) 0xc06d90e0
 	shortPreamble = AH_FALSE
 	an = (struct ath_node *) 0xc25c4000
 	m = (struct mbuf *) 0x0
 	pri = 0
 	__func__ = "ath_tx_start"
 #5  0xc0488390 in ath_start (ifp=0xc2277400)
     at /freebsd/src/sys/dev/ath/if_ath.c:1268
 	sc = (struct ath_softc *) 0xc2281000
 	ah = (struct ath_hal *) 0xc2283000
 	ic = (struct ieee80211com *) 0xc2281230
 	ni = (struct ieee80211_node *) 0xc25c4000
 	bf = (struct ath_buf *) 0xc228c9d4
 	m = (struct mbuf *) 0xc26f3d00
 	wh = (struct ieee80211_frame *) 0xc22774f8
 	__func__ = "ath_start"
 #6  0xc057b483 in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 No locals.
 #7  0xc057c738 in ether_output_frame (ifp=0xc2277400, m=0xc2743b00)
     at /freebsd/src/sys/net/if_ethersubr.c:406
 	len = 54
 	mflags = 3
 	rule = (struct ip_fw *) 0x0
 	error = 0
 #8  0xc057c554 in ether_output (ifp=0xc2277400, m=0xc2743b00, dst=0xcbf9fbec, 
     rt0=0xc2431528) at /freebsd/src/sys/net/if_ethersubr.c:359
 	type = 8
 	error = -1030920174
 	hdrcmplt = 0
 	esrc = "\000\000iK"
 	edst = "\000\021\225\2212"
 	eh = (struct ether_header *) 0xc28d6812
 	loop_copy = 1
 	__func__ = "ether_output"
 #9  0xc05a2d14 in ip_output (m=0xc2743b00, opt=0xc28d6820, ro=0xcbf9fbe8, 
     flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 	ip = (struct ip *) 0xc28d6820
 	ifp = (struct ifnet *) 0xc2277400
 	m0 = (struct mbuf *) 0xc28d6820
 	hlen = 20
 	len = -872809472
 	error = 0
 	dst = (struct sockaddr_in *) 0xcbf9fbec
 	ia = (struct in_ifaddr *) 0xc23a9c00
 	isbroadcast = 0
 	sw_csum = 3073
 	iproute = {ro_rt = 0xc2431528, ro_dst = {sa_len = 16 '\020', 
     sa_family = 2 '\002', 
     sa_data = "\000\000\n\001\001\002\000\000\000\000\000\000\000"}}
 	odst = {s_addr = 3073}
 	__func__ = "ip_output"
 #10 0xc05a2244 in ip_forward (m=0xc2743b00, srcrt=1)
     at /freebsd/src/sys/netinet/ip_input.c:1907
 	ip = (struct ip *) 0xc28d6820
 	ia = (struct in_ifaddr *) 0x0
 	mcopy = (struct mbuf *) 0xc28cc300
 	dest = {s_addr = 0}
 	error = 0
 	type = 0
 	code = 0
 	mtu = 0
 #11 0xc05a0e3f in ip_input (m=0xc2743b00)
     at /freebsd/src/sys/netinet/ip_input.c:689
 	ip = (struct ip *) 0xc28d6820
 	ia = (struct in_ifaddr *) 0xc23bcc00
 	ifa = (struct ifaddr *) 0x0
 	checkif = 0
 	hlen = 20
 	sum = 0
 	dchg = 1
 	odst = {s_addr = 385507398}
 	__func__ = "ip_input"
 #12 0xc057dff6 in netisr_processqueue (ni=0xc06f6a98)
     at /freebsd/src/sys/net/netisr.c:236
 	m = (struct mbuf *) 0xc2743b00
 #13 0xc057e1de in swi_net (dummy=0x0) at /freebsd/src/sys/net/netisr.c:349
 	ni = (struct netisr *) 0xc06f6a98
 	bits = 0
 	i = 0
 #14 0xc0502d6e in ithread_execute_handlers (p=0xc216a830, ie=0xc2168500)
     at /freebsd/src/sys/kern/kern_intr.c:684
 	ih = (struct intr_handler *) 0xc2172040
 	ihn = (struct intr_handler *) 0x0
 #15 0xc0502e9e in ithread_loop (arg=0xc21556d0)
     at /freebsd/src/sys/kern/kern_intr.c:767
 	intr_event = (struct intr_thread *) 0xc21556d0
 	ie = (struct intr_event *) 0xc2168500
 	td = (struct thread *) 0xc216b900
 	p = (struct proc *) 0xc216a830
 	__func__ = "ithread_loop"
 #16 0xc0501fc0 in fork_exit (callout=0xc0502e38 <ithread_loop>, 
     arg=0xc21556d0, frame=0xcbf9fd38) at /freebsd/src/sys/kern/kern_fork.c:805
 	p = (struct proc *) 0xc216a830
 	td = (struct thread *) 0x0
 #17 0xc063a6ac in fork_trampoline ()
     at /freebsd/src/sys/i386/i386/exception.s:208
 No locals.

From: Robert Watson <rwatson@FreeBSD.org>
To: Eric van Gyzen <eric@vangyzen.net>
Cc: Sam Leffler <sam@errno.com>, bug-followup@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Tue, 25 Apr 2006 13:02:34 +0100 (BST)

 On Mon, 24 Apr 2006, Eric van Gyzen wrote:
 
 > Eric van Gyzen wrote:
 >> Sam Leffler wrote:
 >>> I would also try to simplify the kernel config if possible.  I note in
 >>> particular that ipfw is being used with divert sockets and that
 >>> combination has always worried me wrt locking.
 >>
 >> Indeed, divert is involved in the following stack trace (FWIW).
 >> I'll switch to pf at my earliest convenience.
 >>
 >>> There is also IPv6
 >>> configured which should mean the network stack is running w/ Giant which
 >>> makes this again more difficult to analyze.
 >>
 >> I'll also remove IPv6 then,
 >
 > Here is another crash dump after removing IPv6 and switching from 
 > ipfw+divert to pf.
 >
 > Sources are RELENG_6_1 from 20 April.
 >
 > FWIW: I get sporadic "ath0: device timeout" messages, though I do not have 
 > enough data to correlate them with any known events (such as heavy usage).
 >
 > How can I help?
 
 Hmm.  It's not impossible that a device driver bug could result in the panic 
 mentioned below -- specifically, if an mbuf is passed up the network stack, 
 but the device driver references, and in particular, modifies it after calling 
 the stack input routine.  These sorts of bugs are really quite difficult to 
 find, as the bug is only detected significantly later (on socket close).  Sam, 
 is it possible there's a latent bug in one of the error handling paths that 
 results in an mbuf going to the input routine but the device driver frobbing 
 it after that point?
 
 Robert N M Watson
 
 >
 > ==============================================================================
 >
 > $ kgdb kernel.debug /var/crash/vmcore.8
 > [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 > GNU gdb 6.1.1 [FreeBSD]
 > [...]
 >
 > Unread portion of the kernel message buffer:
 > panic: ndx is 0
 > Uptime: 21h21m50s
 > Dumping 255 MB (2 chunks)
 >  chunk 0: 1MB (159 pages) ... ok
 >  chunk 1: 255MB (65264 pages) 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
 >
 > #0  doadump () at pcpu.h:165
 > 165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
 > (kgdb) bt
 > #0  doadump () at pcpu.h:165
 > #1  0xc0515a24 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 > #2  0xc0515ccf in panic (fmt=0xc067d3d3 "ndx is %d")
 >    at /freebsd/src/sys/kern/kern_shutdown.c:558
 > #3  0xc0485a6b in ath_rate_findrate (sc=0xc2281000, an=0xc25c4000,
 >    shortPreamble=0, frameLen=92, rix=0x0, try0=0xcbf9faf0,
 >    txrate=0xcbf9faf7 "")
 >    at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 > #4  0xc048ae10 in ath_tx_start (sc=0xc2281000, ni=0xc25c4000, bf=0xc228c9d4,
 >    m0=0xc26f3d00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 > #5  0xc0488390 in ath_start (ifp=0xc2277400)
 >    at /freebsd/src/sys/dev/ath/if_ath.c:1268
 > #6  0xc057b483 in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 > #7  0xc057c738 in ether_output_frame (ifp=0xc2277400, m=0xc2743b00)
 >    at /freebsd/src/sys/net/if_ethersubr.c:406
 > #8  0xc057c554 in ether_output (ifp=0xc2277400, m=0xc2743b00, dst=0xcbf9fbec,
 >    rt0=0xc2431528) at /freebsd/src/sys/net/if_ethersubr.c:359
 > #9  0xc05a2d14 in ip_output (m=0xc2743b00, opt=0xc28d6820, ro=0xcbf9fbe8,
 >    flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 > #10 0xc05a2244 in ip_forward (m=0xc2743b00, srcrt=1)
 >    at /freebsd/src/sys/netinet/ip_input.c:1907
 > #11 0xc05a0e3f in ip_input (m=0xc2743b00)
 >    at /freebsd/src/sys/netinet/ip_input.c:689
 > #12 0xc057dff6 in netisr_processqueue (ni=0xc06f6a98)
 >    at /freebsd/src/sys/net/netisr.c:236
 > #13 0xc057e1de in swi_net (dummy=0x0) at /freebsd/src/sys/net/netisr.c:349
 > #14 0xc0502d6e in ithread_execute_handlers (p=0xc216a830, ie=0xc2168500)
 >    at /freebsd/src/sys/kern/kern_intr.c:684
 > #15 0xc0502e9e in ithread_loop (arg=0xc21556d0)
 >    at /freebsd/src/sys/kern/kern_intr.c:767
 > #16 0xc0501fc0 in fork_exit (callout=0xc0502e38 <ithread_loop>,
 >    arg=0xc21556d0, frame=0xcbf9fd38) at /freebsd/src/sys/kern/kern_fork.c:805
 > #17 0xc063a6ac in fork_trampoline ()
 >    at /freebsd/src/sys/i386/i386/exception.s:208
 > (kgdb) bt f
 > #0  doadump () at pcpu.h:165
 > No locals.
 > #1  0xc0515a24 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 > 	first_buf_printf = 1
 > #2  0xc0515ccf in panic (fmt=0xc067d3d3 "ndx is %d")
 >    at /freebsd/src/sys/kern/kern_shutdown.c:558
 > 	td = (struct thread *) 0xc216b900
 > 	bootopt = 260
 > 	newpanic = 1
 > 	ap = 0xcbf9fa40 ""
 > 	buf = "ndx is 0", '\0' <repeats 247 times>
 > #3  0xc0485a6b in ath_rate_findrate (sc=0xc2281000, an=0xc25c4000,
 >    shortPreamble=0, frameLen=92, rix=0x0, try0=0xcbf9faf0,
 >    txrate=0xcbf9faf7 "")
 >    at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 > 	sn = (struct sample_node *) 0xc25c423c
 > 	ssc = (struct sample_softc *) 0xc2221090
 > 	ic = (struct ieee80211com *) 0x0
 > 	ndx = 0
 > 	size_bin = 0
 > 	mrr = 1
 > 	best_ndx = -872809744
 > 	change_rates = -826029344
 > 	average_tx_time = 3260826172
 > 	__func__ = "ath_rate_findrate"
 > #4  0xc048ae10 in ath_tx_start (sc=0xc2281000, ni=0xc25c4000, bf=0xc228c9d4,
 >    m0=0xc26f3d00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 > 	ic = (struct ieee80211com *) 0xc2281230
 > 	ah = (struct ath_hal *) 0xc2283000
 > 	ifp = (struct ifnet *) 0xc2277400
 > 	cap = (const struct chanAccParams *) 0xc2281c5a
 > 	i = -1066561312
 > 	error = -1066561312
 > 	iswep = 64
 > 	ismcast = 0
 > 	ismrr = 0
 > 	keyix = 2
 > 	hdrlen = 32
 > 	pktlen = 92
 > 	try0 = 11
 > 	rix = 145 '\221'
 > 	txrate = 242 ''
 > 	ctsrate = 194 ''
 > 	cix = 255 ''
 > 	ds = (struct ath_desc *) 0xcec3cae0
 > 	ds0 = (struct ath_desc *) 0xc0595d92
 > 	txq = (struct ath_txq *) 0x0
 > 	wh = (struct ieee80211_frame *) 0xc26f3d36
 > 	subtype = 0
 > 	flags = 1
 > 	ctsduration = 3227081275
 > 	atype = HAL_PKT_TYPE_NORMAL
 > 	rt = (const HAL_RATE_TABLE *) 0xc06d90e0
 > 	shortPreamble = AH_FALSE
 > 	an = (struct ath_node *) 0xc25c4000
 > 	m = (struct mbuf *) 0x0
 > 	pri = 0
 > 	__func__ = "ath_tx_start"
 > #5  0xc0488390 in ath_start (ifp=0xc2277400)
 >    at /freebsd/src/sys/dev/ath/if_ath.c:1268
 > 	sc = (struct ath_softc *) 0xc2281000
 > 	ah = (struct ath_hal *) 0xc2283000
 > 	ic = (struct ieee80211com *) 0xc2281230
 > 	ni = (struct ieee80211_node *) 0xc25c4000
 > 	bf = (struct ath_buf *) 0xc228c9d4
 > 	m = (struct mbuf *) 0xc26f3d00
 > 	wh = (struct ieee80211_frame *) 0xc22774f8
 > 	__func__ = "ath_start"
 > #6  0xc057b483 in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 > No locals.
 > #7  0xc057c738 in ether_output_frame (ifp=0xc2277400, m=0xc2743b00)
 >    at /freebsd/src/sys/net/if_ethersubr.c:406
 > 	len = 54
 > 	mflags = 3
 > 	rule = (struct ip_fw *) 0x0
 > 	error = 0
 > #8  0xc057c554 in ether_output (ifp=0xc2277400, m=0xc2743b00, dst=0xcbf9fbec,
 >    rt0=0xc2431528) at /freebsd/src/sys/net/if_ethersubr.c:359
 > 	type = 8
 > 	error = -1030920174
 > 	hdrcmplt = 0
 > 	esrc = "\000\000iK"
 > 	edst = "\000\021\225\2212"
 > 	eh = (struct ether_header *) 0xc28d6812
 > 	loop_copy = 1
 > 	__func__ = "ether_output"
 > #9  0xc05a2d14 in ip_output (m=0xc2743b00, opt=0xc28d6820, ro=0xcbf9fbe8,
 >    flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 > 	ip = (struct ip *) 0xc28d6820
 > 	ifp = (struct ifnet *) 0xc2277400
 > 	m0 = (struct mbuf *) 0xc28d6820
 > 	hlen = 20
 > 	len = -872809472
 > 	error = 0
 > 	dst = (struct sockaddr_in *) 0xcbf9fbec
 > 	ia = (struct in_ifaddr *) 0xc23a9c00
 > 	isbroadcast = 0
 > 	sw_csum = 3073
 > 	iproute = {ro_rt = 0xc2431528, ro_dst = {sa_len = 16 '\020',
 >    sa_family = 2 '\002',
 >    sa_data = "\000\000\n\001\001\002\000\000\000\000\000\000\000"}}
 > 	odst = {s_addr = 3073}
 > 	__func__ = "ip_output"
 > #10 0xc05a2244 in ip_forward (m=0xc2743b00, srcrt=1)
 >    at /freebsd/src/sys/netinet/ip_input.c:1907
 > 	ip = (struct ip *) 0xc28d6820
 > 	ia = (struct in_ifaddr *) 0x0
 > 	mcopy = (struct mbuf *) 0xc28cc300
 > 	dest = {s_addr = 0}
 > 	error = 0
 > 	type = 0
 > 	code = 0
 > 	mtu = 0
 > #11 0xc05a0e3f in ip_input (m=0xc2743b00)
 >    at /freebsd/src/sys/netinet/ip_input.c:689
 > 	ip = (struct ip *) 0xc28d6820
 > 	ia = (struct in_ifaddr *) 0xc23bcc00
 > 	ifa = (struct ifaddr *) 0x0
 > 	checkif = 0
 > 	hlen = 20
 > 	sum = 0
 > 	dchg = 1
 > 	odst = {s_addr = 385507398}
 > 	__func__ = "ip_input"
 > #12 0xc057dff6 in netisr_processqueue (ni=0xc06f6a98)
 >    at /freebsd/src/sys/net/netisr.c:236
 > 	m = (struct mbuf *) 0xc2743b00
 > #13 0xc057e1de in swi_net (dummy=0x0) at /freebsd/src/sys/net/netisr.c:349
 > 	ni = (struct netisr *) 0xc06f6a98
 > 	bits = 0
 > 	i = 0
 > #14 0xc0502d6e in ithread_execute_handlers (p=0xc216a830, ie=0xc2168500)
 >    at /freebsd/src/sys/kern/kern_intr.c:684
 > 	ih = (struct intr_handler *) 0xc2172040
 > 	ihn = (struct intr_handler *) 0x0
 > #15 0xc0502e9e in ithread_loop (arg=0xc21556d0)
 >    at /freebsd/src/sys/kern/kern_intr.c:767
 > 	intr_event = (struct intr_thread *) 0xc21556d0
 > 	ie = (struct intr_event *) 0xc2168500
 > 	td = (struct thread *) 0xc216b900
 > 	p = (struct proc *) 0xc216a830
 > 	__func__ = "ithread_loop"
 > #16 0xc0501fc0 in fork_exit (callout=0xc0502e38 <ithread_loop>,
 >    arg=0xc21556d0, frame=0xcbf9fd38) at /freebsd/src/sys/kern/kern_fork.c:805
 > 	p = (struct proc *) 0xc216a830
 > 	td = (struct thread *) 0x0
 > #17 0xc063a6ac in fork_trampoline ()
 >    at /freebsd/src/sys/i386/i386/exception.s:208
 > No locals.
 >

From: Sam Leffler <sam@errno.com>
To: Robert Watson <rwatson@FreeBSD.org>
Cc: Eric van Gyzen <eric@vangyzen.net>, bug-followup@FreeBSD.org
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Tue, 25 Apr 2006 09:21:50 -0700

 Robert Watson wrote:
 
 > Hmm.  It's not impossible that a device driver bug could result in the 
 > panic mentioned below -- specifically, if an mbuf is passed up the 
 > network stack, but the device driver references, and in particular, 
 > modifies it after calling the stack input routine.  These sorts of bugs 
 > are really quite difficult to find, as the bug is only detected 
 > significantly later (on socket close).  Sam, is it possible there's a 
 > latent bug in one of the error handling paths that results in an mbuf 
 > going to the input routine but the device driver frobbing it after that 
 > point?
 
 This panic looks like a problem in the net80211 or ath tx rate control 
 layers; I'm working with Eric on it.
 
 	Sam

From: Sam Leffler <sam@errno.com>
To: Eric van Gyzen <eric@vangyzen.net>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>,
        Tai-hwa Liang <avatar@mmlab.cse.yzu.edu.tw>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Thu, 27 Apr 2006 15:17:54 -0700

 This is a multi-part message in MIME format.
 --------------060109080100080100080602
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 Eric van Gyzen wrote:
 > Eric van Gyzen wrote:
 >> Sam Leffler wrote:
 >>> I would also try to simplify the kernel config if possible.  I note in 
 >>> particular that ipfw is being used with divert sockets and that 
 >>> combination has always worried me wrt locking.
 >> Indeed, divert is involved in the following stack trace (FWIW).
 >> I'll switch to pf at my earliest convenience.
 >>
 >>> There is also IPv6 
 >>> configured which should mean the network stack is running w/ Giant which 
 >>> makes this again more difficult to analyze.
 >> I'll also remove IPv6 then,
 > 
 > Here is another crash dump after removing IPv6 and switching
 > from ipfw+divert to pf.
 > 
 > Sources are RELENG_6_1 from 20 April.
 > 
 > FWIW: I get sporadic "ath0: device timeout" messages, though I do not have
 > enough data to correlate them with any known events (such as heavy usage).
 > 
 > How can I help?
 > 
 > ==============================================================================
 > 
 > $ kgdb kernel.debug /var/crash/vmcore.8
 > [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 > GNU gdb 6.1.1 [FreeBSD]
 > [...]
 > 
 > Unread portion of the kernel message buffer:
 > panic: ndx is 0
 > Uptime: 21h21m50s
 > Dumping 255 MB (2 chunks)
 >   chunk 0: 1MB (159 pages) ... ok
 >   chunk 1: 255MB (65264 pages) 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15
 > 
 > #0  doadump () at pcpu.h:165
 > 165		__asm __volatile("movl %%fs:0,%0" : "=r" (td));
 > (kgdb) bt
 > #0  doadump () at pcpu.h:165
 > #1  0xc0515a24 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 > #2  0xc0515ccf in panic (fmt=0xc067d3d3 "ndx is %d")
 >     at /freebsd/src/sys/kern/kern_shutdown.c:558
 > #3  0xc0485a6b in ath_rate_findrate (sc=0xc2281000, an=0xc25c4000, 
 >     shortPreamble=0, frameLen=92, rix=0x0, try0=0xcbf9faf0, 
 >     txrate=0xcbf9faf7 "")
 >     at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 > #4  0xc048ae10 in ath_tx_start (sc=0xc2281000, ni=0xc25c4000, bf=0xc228c9d4, 
 >     m0=0xc26f3d00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 > #5  0xc0488390 in ath_start (ifp=0xc2277400)
 >     at /freebsd/src/sys/dev/ath/if_ath.c:1268
 > #6  0xc057b483 in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 > #7  0xc057c738 in ether_output_frame (ifp=0xc2277400, m=0xc2743b00)
 >     at /freebsd/src/sys/net/if_ethersubr.c:406
 > #8  0xc057c554 in ether_output (ifp=0xc2277400, m=0xc2743b00, dst=0xcbf9fbec, 
 >     rt0=0xc2431528) at /freebsd/src/sys/net/if_ethersubr.c:359
 > #9  0xc05a2d14 in ip_output (m=0xc2743b00, opt=0xc28d6820, ro=0xcbf9fbe8, 
 >     flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 > #10 0xc05a2244 in ip_forward (m=0xc2743b00, srcrt=1)
 >     at /freebsd/src/sys/netinet/ip_input.c:1907
 > #11 0xc05a0e3f in ip_input (m=0xc2743b00)
 >     at /freebsd/src/sys/netinet/ip_input.c:689
 > #12 0xc057dff6 in netisr_processqueue (ni=0xc06f6a98)
 >     at /freebsd/src/sys/net/netisr.c:236
 > #13 0xc057e1de in swi_net (dummy=0x0) at /freebsd/src/sys/net/netisr.c:349
 > #14 0xc0502d6e in ithread_execute_handlers (p=0xc216a830, ie=0xc2168500)
 >     at /freebsd/src/sys/kern/kern_intr.c:684
 > #15 0xc0502e9e in ithread_loop (arg=0xc21556d0)
 >     at /freebsd/src/sys/kern/kern_intr.c:767
 > #16 0xc0501fc0 in fork_exit (callout=0xc0502e38 <ithread_loop>, 
 >     arg=0xc21556d0, frame=0xcbf9fd38) at /freebsd/src/sys/kern/kern_fork.c:805
 > #17 0xc063a6ac in fork_trampoline ()
 >     at /freebsd/src/sys/i386/i386/exception.s:208
 > (kgdb) bt f
 > #0  doadump () at pcpu.h:165
 > No locals.
 > #1  0xc0515a24 in boot (howto=260) at /freebsd/src/sys/kern/kern_shutdown.c:402
 > 	first_buf_printf = 1
 > #2  0xc0515ccf in panic (fmt=0xc067d3d3 "ndx is %d")
 >     at /freebsd/src/sys/kern/kern_shutdown.c:558
 > 	td = (struct thread *) 0xc216b900
 > 	bootopt = 260
 > 	newpanic = 1
 > 	ap = 0xcbf9fa40 ""
 > 	buf = "ndx is 0", '\0' <repeats 247 times>
 > #3  0xc0485a6b in ath_rate_findrate (sc=0xc2281000, an=0xc25c4000, 
 >     shortPreamble=0, frameLen=92, rix=0x0, try0=0xcbf9faf0, 
 >     txrate=0xcbf9faf7 "")
 >     at /freebsd/src/sys/dev/ath/ath_rate/sample/sample.c:350
 > 	sn = (struct sample_node *) 0xc25c423c
 > 	ssc = (struct sample_softc *) 0xc2221090
 > 	ic = (struct ieee80211com *) 0x0
 > 	ndx = 0
 > 	size_bin = 0
 > 	mrr = 1
 > 	best_ndx = -872809744
 > 	change_rates = -826029344
 > 	average_tx_time = 3260826172
 > 	__func__ = "ath_rate_findrate"
 > #4  0xc048ae10 in ath_tx_start (sc=0xc2281000, ni=0xc25c4000, bf=0xc228c9d4, 
 >     m0=0xc26f3d00) at /freebsd/src/sys/dev/ath/if_ath.c:3505
 > 	ic = (struct ieee80211com *) 0xc2281230
 > 	ah = (struct ath_hal *) 0xc2283000
 > 	ifp = (struct ifnet *) 0xc2277400
 > 	cap = (const struct chanAccParams *) 0xc2281c5a
 > 	i = -1066561312
 > 	error = -1066561312
 > 	iswep = 64
 > 	ismcast = 0
 > 	ismrr = 0
 > 	keyix = 2
 > 	hdrlen = 32
 > 	pktlen = 92
 > 	try0 = 11
 > 	rix = 145 '\221'
 > 	txrate = 242 ''
 > 	ctsrate = 194 ''
 > 	cix = 255 ''
 > 	ds = (struct ath_desc *) 0xcec3cae0
 > 	ds0 = (struct ath_desc *) 0xc0595d92
 > 	txq = (struct ath_txq *) 0x0
 > 	wh = (struct ieee80211_frame *) 0xc26f3d36
 > 	subtype = 0
 > 	flags = 1
 > 	ctsduration = 3227081275
 > 	atype = HAL_PKT_TYPE_NORMAL
 > 	rt = (const HAL_RATE_TABLE *) 0xc06d90e0
 > 	shortPreamble = AH_FALSE
 > 	an = (struct ath_node *) 0xc25c4000
 > 	m = (struct mbuf *) 0x0
 > 	pri = 0
 > 	__func__ = "ath_tx_start"
 > #5  0xc0488390 in ath_start (ifp=0xc2277400)
 >     at /freebsd/src/sys/dev/ath/if_ath.c:1268
 > 	sc = (struct ath_softc *) 0xc2281000
 > 	ah = (struct ath_hal *) 0xc2283000
 > 	ic = (struct ieee80211com *) 0xc2281230
 > 	ni = (struct ieee80211_node *) 0xc25c4000
 > 	bf = (struct ath_buf *) 0xc228c9d4
 > 	m = (struct mbuf *) 0xc26f3d00
 > 	wh = (struct ieee80211_frame *) 0xc22774f8
 > 	__func__ = "ath_start"
 > #6  0xc057b483 in if_start (ifp=0xc2277400) at /freebsd/src/sys/net/if.c:2234
 > No locals.
 > #7  0xc057c738 in ether_output_frame (ifp=0xc2277400, m=0xc2743b00)
 >     at /freebsd/src/sys/net/if_ethersubr.c:406
 > 	len = 54
 > 	mflags = 3
 > 	rule = (struct ip_fw *) 0x0
 > 	error = 0
 > #8  0xc057c554 in ether_output (ifp=0xc2277400, m=0xc2743b00, dst=0xcbf9fbec, 
 >     rt0=0xc2431528) at /freebsd/src/sys/net/if_ethersubr.c:359
 > 	type = 8
 > 	error = -1030920174
 > 	hdrcmplt = 0
 > 	esrc = "\000\000iK"
 > 	edst = "\000\021\225\2212"
 > 	eh = (struct ether_header *) 0xc28d6812
 > 	loop_copy = 1
 > 	__func__ = "ether_output"
 > #9  0xc05a2d14 in ip_output (m=0xc2743b00, opt=0xc28d6820, ro=0xcbf9fbe8, 
 >     flags=1, imo=0x0, inp=0x0) at /freebsd/src/sys/netinet/ip_output.c:777
 > 	ip = (struct ip *) 0xc28d6820
 > 	ifp = (struct ifnet *) 0xc2277400
 > 	m0 = (struct mbuf *) 0xc28d6820
 > 	hlen = 20
 > 	len = -872809472
 > 	error = 0
 > 	dst = (struct sockaddr_in *) 0xcbf9fbec
 > 	ia = (struct in_ifaddr *) 0xc23a9c00
 > 	isbroadcast = 0
 > 	sw_csum = 3073
 > 	iproute = {ro_rt = 0xc2431528, ro_dst = {sa_len = 16 '\020', 
 >     sa_family = 2 '\002', 
 >     sa_data = "\000\000\n\001\001\002\000\000\000\000\000\000\000"}}
 > 	odst = {s_addr = 3073}
 > 	__func__ = "ip_output"
 > #10 0xc05a2244 in ip_forward (m=0xc2743b00, srcrt=1)
 >     at /freebsd/src/sys/netinet/ip_input.c:1907
 > 	ip = (struct ip *) 0xc28d6820
 > 	ia = (struct in_ifaddr *) 0x0
 > 	mcopy = (struct mbuf *) 0xc28cc300
 > 	dest = {s_addr = 0}
 > 	error = 0
 > 	type = 0
 > 	code = 0
 > 	mtu = 0
 > #11 0xc05a0e3f in ip_input (m=0xc2743b00)
 >     at /freebsd/src/sys/netinet/ip_input.c:689
 > 	ip = (struct ip *) 0xc28d6820
 > 	ia = (struct in_ifaddr *) 0xc23bcc00
 > 	ifa = (struct ifaddr *) 0x0
 > 	checkif = 0
 > 	hlen = 20
 > 	sum = 0
 > 	dchg = 1
 > 	odst = {s_addr = 385507398}
 > 	__func__ = "ip_input"
 > #12 0xc057dff6 in netisr_processqueue (ni=0xc06f6a98)
 >     at /freebsd/src/sys/net/netisr.c:236
 > 	m = (struct mbuf *) 0xc2743b00
 > #13 0xc057e1de in swi_net (dummy=0x0) at /freebsd/src/sys/net/netisr.c:349
 > 	ni = (struct netisr *) 0xc06f6a98
 > 	bits = 0
 > 	i = 0
 > #14 0xc0502d6e in ithread_execute_handlers (p=0xc216a830, ie=0xc2168500)
 >     at /freebsd/src/sys/kern/kern_intr.c:684
 > 	ih = (struct intr_handler *) 0xc2172040
 > 	ihn = (struct intr_handler *) 0x0
 > #15 0xc0502e9e in ithread_loop (arg=0xc21556d0)
 >     at /freebsd/src/sys/kern/kern_intr.c:767
 > 	intr_event = (struct intr_thread *) 0xc21556d0
 > 	ie = (struct intr_event *) 0xc2168500
 > 	td = (struct thread *) 0xc216b900
 > 	p = (struct proc *) 0xc216a830
 > 	__func__ = "ithread_loop"
 > #16 0xc0501fc0 in fork_exit (callout=0xc0502e38 <ithread_loop>, 
 >     arg=0xc21556d0, frame=0xcbf9fd38) at /freebsd/src/sys/kern/kern_fork.c:805
 > 	p = (struct proc *) 0xc216a830
 > 	td = (struct thread *) 0x0
 > #17 0xc063a6ac in fork_trampoline ()
 >     at /freebsd/src/sys/i386/i386/exception.s:208
 > No locals.
 > 
 > 
 
 This crash turns out to be a bug in the net80211 layer.  The issue is 
 that data frames are allowed to be xmit to stations prior to their being 
 associated.  I believe the following is happening:
 
 1. station associates and fills an arp/bridge entry in the ap machine
 2. station is disassociated, e.g. because it is idle
 3. station starts to reassociate but before this completes a data packet 
 comes in on the wired interface for it; the packet will be forwarded by 
 the upper layer and sent to the wireless interface where the driver will 
 try to transmit it; in doing the xmit the ath driver asks the rate 
 control module for a tx rate to use in sending the frame but because the 
 station is not yet associated the rate control module hasn't been asked 
 to setup the rate set (this depends on the negotiated rate set 
 established on associate)
 
 This explains the assertion failure in the rate control module.  The 
 station was in an authorized state but not yet associated.  Consequently 
 the rate control data was zero'd and the assertion fired.
 
 Attached is a change to make ieee80211_find_txnode return NULL for this 
 case (operating in hostap mode, sending a directed packet, and station 
 not associated).  I believe this will fix your problem.  The change 
 would be better placed in each driver's start routine but this would 
 require changing all drivers.
 
 Tai-hwa if you can review I'd appreciate it.
 
 	Sam
 
 
 --------------060109080100080100080602
 Content-Type: text/plain;
  name="ieee80211_node.c.patch"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline;
  filename="ieee80211_node.c.patch"
 
 Index: ieee80211_node.c
 ===================================================================
 RCS file: /usr/ncvs/src/sys/net80211/ieee80211_node.c,v
 retrieving revision 1.73
 diff -u -r1.73 ieee80211_node.c
 --- ieee80211_node.c	6 Mar 2006 17:23:26 -0000	1.73
 +++ ieee80211_node.c	27 Apr 2006 21:58:51 -0000
 @@ -1454,8 +1454,19 @@
  	IEEE80211_NODE_LOCK(nt);
  	if (ic->ic_opmode == IEEE80211_M_STA || IEEE80211_IS_MULTICAST(macaddr))
  		ni = ieee80211_ref_node(ic->ic_bss);
 -	else
 +	else {
  		ni = _ieee80211_find_node(nt, macaddr);
 +		if (ic->ic_opmode == IEEE80211_M_HOSTAP && 
 +		    (ni != NULL && ni->ni_associd == 0)) {
 +			/*
 +			 * Station is not associated; don't permit the
 +			 * data frame to be sent by returning NULL.  This
 +			 * is kinda a kludge but the least intrusive way
 +			 * to add this check into all drivers.
 +			 */
 +			ieee80211_unref_node(&ni);	/* NB: null's ni */
 +		}
 +	}
  	IEEE80211_NODE_UNLOCK(nt);
  
  	if (ni == NULL) {
 
 --------------060109080100080100080602--

From: Tai-hwa Liang <avatar@mmlab.cse.yzu.edu.tw>
To: Sam Leffler <sam@errno.com>
Cc: Eric van Gyzen <eric@vangyzen.net>, bug-followup@FreeBSD.org, 
    Robert Watson <rwatson@FreeBSD.org>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Fri, 28 Apr 2006 09:01:39 +0800 (CST)

 On Thu, 27 Apr 2006, Sam Leffler wrote:
 [...]
 > This crash turns out to be a bug in the net80211 layer.  The issue is that 
 > data frames are allowed to be xmit to stations prior to their being 
 > associated.  I believe the following is happening:
 >
 > 1. station associates and fills an arp/bridge entry in the ap machine
 > 2. station is disassociated, e.g. because it is idle
 > 3. station starts to reassociate but before this completes a data packet 
 > comes in on the wired interface for it; the packet will be forwarded by the 
 > upper layer and sent to the wireless interface where the driver will try to 
 > transmit it; in doing the xmit the ath driver asks the rate control module 
 > for a tx rate to use in sending the frame but because the station is not yet 
 > associated the rate control module hasn't been asked to setup the rate set 
 > (this depends on the negotiated rate set established on associate)
 >
 > This explains the assertion failure in the rate control module.  The station 
 > was in an authorized state but not yet associated.  Consequently the rate 
 > control data was zero'd and the assertion fired.
 >
 > Attached is a change to make ieee80211_find_txnode return NULL for this case 
 > (operating in hostap mode, sending a directed packet, and station not 
 > associated).  I believe this will fix your problem.  The change would be 
 > better placed in each driver's start routine but this would require changing 
 > all drivers.
 
    I think it's okay to do this in net80211 since it also reduces code 
 duplication in all drivers.
 
 >
 > Tai-hwa if you can review I'd appreciate it.
 
    Thank you for the elaboration. This patch looks reasonable to me.
 
 -- 
 Thanks,
 
 Tai-hwa Liang

From: Eric van Gyzen <eric@vangyzen.net>
To: Sam Leffler <sam@errno.com>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>, 
 Tai-hwa Liang <avatar@mmlab.cse.yzu.edu.tw>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Fri, 28 Apr 2006 21:15:41 -0500

 Sam Leffler wrote:
 > Attached is a change to make ieee80211_find_txnode return NULL for this 
 > case (operating in hostap mode, sending a directed packet, and station 
 > not associated).  I believe this will fix your problem.
 
 Excellent!  I booted the patched kernel last night.  Average 
 time-to-panic was about 5 days; maximum was 12.  If I get no panic in 
 two weeks, I'll consider it fixed and be very grateful.  Oh wait...I'm 
 already very grateful.  ;)
 
 Eric

From: Eric van Gyzen <eric@vangyzen.net>
To: Eric van Gyzen <eric@vangyzen.net>
Cc: Sam Leffler <sam@errno.com>, bug-followup@FreeBSD.org,
	Robert Watson <rwatson@FreeBSD.org>,
	Tai-hwa Liang <avatar@mmlab.cse.yzu.edu.tw>
Subject: Re: kern/94433: [panic] panic: sbdrop
Date: Fri, 12 May 2006 09:04:27 -0500

 Eric van Gyzen wrote:
 > Sam Leffler wrote:
 > >Attached is a change to make ieee80211_find_txnode return NULL for this 
 > >case (operating in hostap mode, sending a directed packet, and station 
 > >not associated).  I believe this will fix your problem.
 > 
 > Excellent!  I booted the patched kernel last night.  Average 
 > time-to-panic was about 5 days; maximum was 12.  If I get no panic in 
 > two weeks, I'll consider it fixed and be very grateful.  Oh wait...I'm 
 > already very grateful.  ;)
 
 The box has been up for 14 days.  I'm convinced...as I expected to be.
 
 Thank you all for your help.  I hope you're enjoying BSDCan.
 
 Eric
Responsible-Changed-From-To: rwatson->sam 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Mon Jun 12 11:50:37 UTC 2006 
Responsible-Changed-Why:  
Assign to sam, since he's provided a patch.  I'm not sure what the commit 
and merge status is, so will let him pick up from here. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=94433 
State-Changed-From-To: open->closed 
State-Changed-By: sam 
State-Changed-When: Mon Jun 12 21:06:43 UTC 2006 
State-Changed-Why:  
net80211 fix committed 

http://www.freebsd.org/cgi/query-pr.cgi?pr=94433 
>Unformatted:
