From nobody@FreeBSD.org  Wed Apr 19 09:42:33 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id F2B1616A400
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 19 Apr 2006 09:42:32 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id AD7C243D45
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 19 Apr 2006 09:42:32 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k3J9gW4e027378
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 19 Apr 2006 09:42:32 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k3J9gW5C027365;
	Wed, 19 Apr 2006 09:42:32 GMT
	(envelope-from nobody)
Message-Id: <200604190942.k3J9gW5C027365@www.freebsd.org>
Date: Wed, 19 Apr 2006 09:42:32 GMT
From: "Vladislav V. Prodan" <vlad11@onua.edu.ua>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Kernel panics with sbdrop
X-Send-Pr-Version: www-2.3

>Number:         96042
>Category:       kern
>Synopsis:       Kernel panics with sbdrop
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Apr 19 09:50:13 GMT 2006
>Closed-Date:    Sun Mar 02 02:46:01 UTC 2008
>Last-Modified:  Sun Mar 02 02:46:01 UTC 2008
>Originator:     Vladislav V. Prodan
>Release:        FreeBSD 6.1-PRERELEASE
>Organization:
Odessa National Academy of Law
>Environment:
FreeBSD bahus.nla.od.ua 6.1-PRERELEASE FreeBSD 6.1-PRERELEASE #0: Tue Apr 18 16:21:06 EEST 2006     vlad11@bahus.nla.od.ua:/usr/obj/usr/src/sys/bahus.10  i386
>Description:
I've encountered a strange problem while using FreeBSD 6.1-PRERELEASE for our file-server providing access via ftp and NFS.
The system crashes constantly with a kernel panic (panic: sbdrop).
Status netstat -m before crash:

337/473/810 mbufs in use (current/cache/total)
80/144/224/33792 mbuf clusters in use (current/cache/total/max)
244K/406K/650K bytes allocated to network (current/cache/total)
4009804/688682/688637 requests for mbufs denied
(mbufs/clusters/mbuf+clusters)
0/3/8704 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
131 calls to protocol drain routines

Also in the console appeared before panic messages: 
"
in_cksum_skip: out of data by 13260 
in_cksum_skip: out of data by 13260 
" 
and 
"
gig0: discard oversize frame (ether type 23c2 flags 3 len 1518> max 1514) 
gig0: discard oversize frame (ether type 23c2 flags 3 len 1518> max 1514)
sk0: watchdog timeout 
"


The kernel is a GENERIC kernel with changes for SMP and semaphores support.

<skip>

maxusers        512

# System V semaphores and tunable parameters
options         SYSVSEM         # include support for semaphores
options         SEMMAP=31       # amount of entries in semaphore map
options         SEMMNI=11       # number of semaphore identifiers in the system
options         SEMMNS=61       # number of semaphores in the system
options         SEMMNU=31       # number of undo structures in the system
options         SEMMSL=61       # max number of semaphores per id
options         SEMOPM=101      # max number of operations per semop call
options         SEMUME=11       # max number of undo entries per process

# System V message queues and tunable parameters
options         SYSVMSG         # include support for message queues
options         MSGMNB=2049     # max characters per message queue
options         MSGMNI=41       # max number of message queue identifiers
options         MSGSEG=2049     # max number of message segments in the system
options         MSGSSZ=16       # size of a message segment MUST be power of 2
options         MSGTQL=41       # max amount of messages in the system


# System V shared memory and tunable parameters
options         SYSVSHM         # include support for shared memory
options         SHMMAXPGS=2048  # max amount of shared memory pages (4k on i386)
options         SHMALL=2048     # max number of shared memory pages system wide
options         SHMMAX="(SHMMAXPGS*PAGE_SIZE+1)"         # max shared memory segment size (b
options         SHMMIN=2        # min shared memory segment size (bytes)
options         SHMMNI=33       # max number of shared memory identifiers
options         SHMSEG=9        # max shared memory segments per process



options         SMP
#options         MUTEX_NOINLINE
options         MUTEX_WAKE_ALL
options         PREEMPTION
options         FULL_PREEMPTION
options         MUTEX_DEBUG
#options         WITNESS
#options         WITNESS_KDB
#options         WITNESS_SKIPSPIN

options         MUTEX_PROFILING

# Set the number of buffers and the hash size.  The hash size MUST be
larger
# than the number of buffers.  Hash size should be prime.

options         MPROF_BUFFERS="1536"
options         MPROF_HASH_SIZE="1543"

# Profiling for internal hash tables.
options         SLEEPQUEUE_PROFILING
options         TURNSTILE_PROFILING

# Mandatory:
device          apic                    # I/O apic

# Optional:
options         MPTABLE_FORCE_HTT       # Enable HTT CPUs with the MP Table
options         IPI_PREEMPTION
<skip>


>How-To-Repeat:
It is necessary to try three times downloading a file with ftp nearby 1 GB for the speed 3-10 MB/c. System it is guaranteed receives panic.

P.S. I can give shell access to this server.
>Fix:

>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: glebius 
State-Changed-When: Wed Apr 19 10:26:02 UTC 2006 
State-Changed-Why:  
Please obtain kernel backtrace for this panic. Some instructions 
can be found here: 
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96042 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/96042: Kernel panics with sbdrop
Date: Wed, 19 Apr 2006 17:28:57 +0400

 ----- Forwarded message from "Vladislav V. Prodan" <vlad11@nla.od.ua> -----
 
 (kgdb) bt
 #0  doadump () at pcpu.h:165
 #1  0xc05627d8 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
 #2  0xc0562a6a in panic (fmt=0xc06f09a5 "%s") at
 /usr/src/sys/kern/kern_shutdown.c:555
 #3  0xc06cc844 in trap_fatal (frame=0xd5cdd92c, eva=2501437698) at
 /usr/src/sys/i386/i386/trap.c:836
 #4  0xc06cc583 in trap_pfault (frame=0xd5cdd92c, usermode=0, eva=2501437698)
     at /usr/src/sys/i386/i386/trap.c:744
 #5  0xc06cc22d in trap (frame=
       {tf_fs = -992542712, tf_es = -992542680, tf_ds = 40, tf_edi =
 1448, tf_esi = -989467904, tf_ebp = -707929716, tf_isp = -707929768,
 tf_ebx = 13083, tf_edx = 13082, tf_ecx = -1793529598, tf_eax = 1,
 tf_trapno = 12, tf_err = 0, tf_eip = -1066672420, tf_cs = 32, tf_eflags
 = 66050, tf_esp = -707985408, tf_ss = 70})
     at /usr/src/sys/i386/i386/trap.c:434
 #6  0xc06bab0a in calltrap () at /usr/src/sys/i386/i386/exception.s:139
 #7  0xc06bdedc in in_cksum_skip (m=0xc505eb00, len=0, skip=13082) at
 /usr/src/sys/i386/i386/in_cksum.c:335
 #8  0xc06038fb in in_delayed_cksum (m=0xc4d87a00) at
 /usr/src/sys/netinet/ip_output.c:1027
 #9  0xc0457b45 in fr_check (ip=0xc4d87a40, hlen=20, ifp=0x1, out=1,
 mp=0xd5cddaac)
     at /usr/src/sys/contrib/ipfilter/netinet/fil.c:2248
 #10 0xc045bcaa in fr_check_wrapper (arg=0x0, mp=0x331a, ifp=0xc4c88400,
 dir=2)
     at /usr/src/sys/contrib/ipfilter/netinet/ip_fil_freebsd.c:156
 #11 0xc05d1ba3 in pfil_run_hooks (ph=0xc0792d40, mp=0xd5cddb24,
 ifp=0xc4c88400, dir=2, inp=0xc4de8cc0)
     at /usr/src/sys/net/pfil.c:139
 #12 0xc060320a in ip_output (m=0xc4d87a00, opt=0xc4d87a40,
 ro=0xd5cddaf0, flags=0, imo=0x0, inp=0xc4de8cc0)
     at /usr/src/sys/netinet/ip_output.c:667
 #13 0xc060a9e5 in tcp_output (tp=0xc4dc68fc) at
 /usr/src/sys/netinet/tcp_output.c:1080
 #14 0xc0607c07 in tcp_input (m=0xc50ace00, off0=20) at
 /usr/src/sys/netinet/tcp_input.c:1245
 #15 0xc05fd39d in ip_input (m=0xc50ace00) at
 /usr/src/sys/netinet/ip_input.c:786
 #16 0xc05d07ee in netisr_processqueue (ni=0xc0791a78) at
 /usr/src/sys/net/netisr.c:236
 #17 0xc05d09b6 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
 #18 0xc05511c8 in ithread_loop (arg=0xc4a18e00) at
 /usr/src/sys/kern/kern_intr.c:547
 #19 0xc05506c1 in fork_exit (callout=0xc05510b4 <ithread_loop>,
 arg=0xc4a18e00, frame=0xd5cddd38)
     at /usr/src/sys/kern/kern_fork.c:789
 #20 0xc06bab6c in fork_trampoline () at
 /usr/src/sys/i386/i386/exception.s:208
 
 ----- End forwarded message -----
 
 -- 
 Totus tuus, Glebius.
 GLEBIUS-RIPN GLEB-RIPE

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: "Vladislav V. Prodan" <vlad11@nla.od.ua>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/96042: Kernel panics with sbdrop
Date: Wed, 19 Apr 2006 17:30:49 +0400

 The panic backtrace is attached to PR. However, the backtrace
 doesn't end with a panic about "sbdrop". Do you have two
 different panics.
 
 P.S. Also panic looks like ipfilter related. Can you try to disable
 
 -- 
 Totus tuus, Glebius.
 GLEBIUS-RIPN GLEB-RIPE

From: "Vladislav V. Prodan" <vlad11@nla.od.ua>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/96042: Kernel panics with sbdrop
Date: Wed, 19 Apr 2006 16:58:35 +0300

 Gleb Smirnoff :
 > The panic backtrace is attached to PR. However, the backtrace
 > doesn't end with a panic about "sbdrop". Do you have two
 > different panics.
 
 Well. Has provoked one more panic.
 
 netstat -m before kernel panic:
 
 [16:35]bahus:vlad11->sys/bahus.10# netstat -m
 323/547/870 mbufs in use (current/cache/total)
 66/384/450/33792 mbuf clusters in use (current/cache/total/max)
 212K/904K/1117K bytes allocated to network (current/cache/total)
 477203/256010/261501 requests for mbufs denied
 (mbufs/clusters/mbuf+clusters)
 0/3/8704 sfbufs in use (current/peak/max)
 0 requests for sfbufs denied
 0 requests for sfbufs delayed
 0 requests for I/O initiated by sendfile
 661 calls to protocol drain routines
 
 
 [16:43]bahus:vlad11->sys/bahus.10# kgdb kernel.debug /var/crash/vmcore.7
 [GDB will not be able to debug user-mode threads:
 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain
 conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-marcel-freebsd".
 
 Unread portion of the kernel message buffer:
 panic: sbdrop
 cpuid = 0
 Uptime: 4h27m52s
 Dumping 511 MB (2 chunks)
   chunk 0: 1MB (159 pages) ... ok
   chunk 1: 511MB (130800 pages) 495 479 463 447 431 415 399 383 367 351
 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47
 31 15
 
 #0  doadump () at pcpu.h:165
 165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
 (kgdb) bt
 #0  doadump () at pcpu.h:165
 #1  0xc05627d8 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
 #2  0xc0562a6a in panic (fmt=0xc0709efe "sbdrop")
     at /usr/src/sys/kern/kern_shutdown.c:555
 #3  0xc0598e7c in sbdrop_locked (sb=0xc4d59d80, len=1660846797)
     at /usr/src/sys/kern/uipc_socket2.c:1157
 #4  0xc059900e in sbdrop (sb=0xc4d59d80, len=1660879005)
     at /usr/src/sys/kern/uipc_socket2.c:1208
 #5  0xc0607b02 in tcp_input (m=0xc4adfd00, off0=20)
     at /usr/src/sys/netinet/tcp_input.c:1206
 #6  0xc05fd39d in ip_input (m=0xc4adfd00)
     at /usr/src/sys/netinet/ip_input.c:786
 #7  0xc05d07ee in netisr_processqueue (ni=0xc0791a78)
     at /usr/src/sys/net/netisr.c:236
 #8  0xc05d09b6 in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
 #9  0xc05511c8 in ithread_loop (arg=0xc4a18e00)
     at /usr/src/sys/kern/kern_intr.c:547
 #10 0xc05506c1 in fork_exit (callout=0xc05510b4 <ithread_loop>,
     arg=0xc4a18e00, frame=0xd5cddd38) at /usr/src/sys/kern/kern_fork.c:789
 #11 0xc06bab6c in fork_trampoline () at
 /usr/src/sys/i386/i386/exception.s:208
 
 
 Copyright (c) 1992-2006 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
 FreeBSD 6.1-PRERELEASE #0: Tue Apr 18 16:21:06 EEST 2006
     vlad11@bahus.nla.od.ua:/usr/obj/usr/src/sys/bahus.10
 ACPI APIC Table: <AMIINT >
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: Intel Pentium III (996.55-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
   Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
 real memory  = 536805376 (511 MB)
 avail memory = 515223552 (491 MB)
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
 ioapic0 <Version 1.1> irqs 0-23 on motherboard
 npx0: [FAST]
 npx0: <math processor> on motherboard
 npx0: INT 16 interface
 acpi0: <AMIINT > on motherboard
 acpi0: Power Button (fixed)
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 acpi_button0: <Power Button> on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 agp0: <VIA 82C691 (Apollo Pro) host to PCI bridge> mem 0xe0000000-0xe0ffffff at device 0.0 on pci0
 pcib1: <PCI-PCI bridge> at device 1.0 on pci0
 pci2: <PCI bus> on pcib1
 pci2: <display, VGA> at device 0.0 (no driver attached)
 isab0: <PCI-ISA bridge> at device 7.0 on pci0
 isa0: <ISA bus> on isab0
 atapci0: <VIA 82C686A UDMA66 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0
 ata0: <ATA channel 0> on atapci0
 ata1: <ATA channel 1> on atapci0
 pci0: <serial bus, SMBus> at device 7.4 (no driver attached)
 pcib2: <PCI-PCI bridge> at device 10.0 on pci0
 pci1: <PCI bus> on pcib2
 asr0: <Adaptec Caching SCSI RAID> mem 0xd4000000-0xd5ffffff irq 18 at device 10.1 on pci0
 asr0: [GIANT-LOCKED]
 asr0: ADAPTEC 2100S FW Rev. 370F, 1 channel, 256 CCBs, Protocol I2O
 skc0: <SysKonnect Gigabit Ethernet (V2.0)> port 0xd800-0xd8ff mem 0xdfff8000-0xdfffbfff irq 16 at device 12.0 on pci0
 skc0: Marvell Yukon Lite Gigabit Ethernet rev. A3(0x7)
 sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
 sk0: Ethernet address: 00:0c:46:46:7f:41
 miibus0: <MII bus> on sk0
 e1000phy0: <Marvell 88E1000 Gigabit PHY> on miibus0
 e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
 fxp0: <Intel 82550 Pro/100 Ethernet> port 0xdc00-0xdc3f mem 0xdffff000-0xdfffffff,0xdffc0000-0xdffdffff irq 17 at device 13.0 on pci0
 miibus1: <MII bus> on fxp0
 inphy0: <i82555 10/100 media interface> on miibus1
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 fxp0: Ethernet address: 00:02:b3:87:a2:5c
 acpi_button1: <Sleep Button> on acpi0
 acpi_tz0: <Thermal Zone> on acpi0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 psm0: <PS/2 Mouse> irq 12 on atkbdc0
 psm0: [GIANT-LOCKED]
 psm0: model IntelliMouse, device ID 3
 fdc0: <floppy drive controller> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
 fdc0: [FAST]
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcafff,0xcb000-0xd0fff on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 ppc0: parallel port not found.
 sio0: configured irq 4 not in bitmap of probed irqs 0
 sio0: port may not be enabled
 sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
 sio0: type 8250 or not responding
 sio1: configured irq 3 not in bitmap of probed irqs 0
 sio1: port may not be enabled
 Timecounters tick every 1.000 msec
 IP Filter: v4.1.8 initialized.  Default = pass all, Logging = enabled
 ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding enabled, default to accept, logging limited to 100 packets/entry by default
 ad2: 305245MB <WDC WD3200JB-00KFA0 08.05J08> at ata1-master UDMA66
 ad3: 305245MB <WDC WD3200JB-00KFA0 08.05J08> at ata1-slave UDMA66
 da0 at asr0 bus 0 target 1 lun 0
 da0: <ADAPTEC RAID-5 370F> Fixed Direct Access SCSI-2 device
 da0: Tagged Queueing Enabled
 da0: 35002MB (71684096 512 byte sectors: 255H 63S/T 4462C)
 SMP: AP CPU #1 Launched!
 
 I tried to change network card Allied Telesyn AT-2916T (sk0) on Intel
 Pro/100 S (fxp0), but the problem has not disappeared...
 
State-Changed-From-To: feedback->open 
State-Changed-By: linimon 
State-Changed-When: Thu Jun 15 07:47:22 UTC 2006 
State-Changed-Why:  
Feedback received some time ago. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96042 
State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Fri Nov 16 02:08:52 UTC 2007 
State-Changed-Why:  

Need feedback. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96042 
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Sun Mar 2 02:45:03 UTC 2008 
State-Changed-Why:  
Feedback timeout (> 3 months). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=96042 
>Unformatted:
Does this still happen with recent RELENG_6? This can be a result of incorrectly locked
mbuf manipulation.
