From nobody@FreeBSD.org  Wed Mar 16 12:24:02 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B70F3106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 16 Mar 2011 12:24:02 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 8B4FF8FC16
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 16 Mar 2011 12:24:02 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p2GCO1jG020447
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 16 Mar 2011 12:24:01 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p2GCO1SW020444;
	Wed, 16 Mar 2011 12:24:01 GMT
	(envelope-from nobody)
Message-Id: <201103161224.p2GCO1SW020444@red.freebsd.org>
Date: Wed, 16 Mar 2011 12:24:01 GMT
From: Vladimir Kutakov <vova@ashmanov.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Kernel panics with "sbdrop" message
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         155597
>Category:       kern
>Synopsis:       [panic] Kernel panics with "sbdrop" message
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-net
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Mar 16 12:30:12 UTC 2011
>Closed-Date:    
>Last-Modified:  Wed Jan 11 14:20:07 UTC 2012
>Originator:     Vladimir Kutakov
>Release:        7.4 RELEASE
>Organization:
>Environment:
FreeBSD nickel.novoteka.ru 7.4-RELEASE FreeBSD 7.4-RELEASE #0: Wed Mar  9 17:28:54 MSK 2011     prince@nickel.novoteka.ru:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
The machine works as webserver and has panic "sbdrop" after some hours of work. The network is rather loaded, but not extremly (about 20-30kpps input).

Please let me know if some additional information needed. I want to resolve the problem very much.

Here is the backtrace from the dumped core:
(kgdb) bt
#0  doadump () at pcpu.h:196
#1  0xffffff000706d3a0 in ?? ()
#2  0xffffffff8054d4da in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:421
#3  0xffffffff8054d8f2 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:576
#4  0xffffffff805a9976 in sbdrop_internal (sb=Variable "sb" is not available.
) at /usr/src/sys/kern/uipc_sockbuf.c:843
#5  0xffffffff8069d53c in tcp_do_segment (m=0xffffff01b5618400, th=0xffffff01573e4022, so=0xffffff013eecd2d0, tp=0xffffff016f9f8000, drop_hdrlen=40, tlen=0) at /usr/src/sys/netinet/tcp_input.c:2043
#6  0xffffffff8069ed48 in tcp_input (m=0xffffff01b5618400, off0=Variable "off0" is not available.
) at /usr/src/sys/netinet/tcp_input.c:847
#7  0xffffffff8063a2eb in ip_input (m=0xffffff01b5618400) at /usr/src/sys/netinet/ip_input.c:663
#8  0xffffffff805f2ee1 in ether_demux (ifp=0xffffff00070f5000, m=0xffffff01b5618400) at /usr/src/sys/net/if_ethersubr.c:834
#9  0xffffffff805f315e in ether_input (ifp=0xffffff00070f5000, m=0xffffff01b5618400) at /usr/src/sys/net/if_ethersubr.c:692
#10 0xffffffff8031ddb9 in igb_rxeof (que=Variable "que" is not available.
) at /usr/src/sys/dev/e1000/if_igb.c:4097
#11 0xffffffff8031e1a8 in igb_msix_que (arg=Variable "arg" is not available.
) at /usr/src/sys/dev/e1000/if_igb.c:1309
#12 0xffffffff8052b0e5 in ithread_loop (arg=0xffffff00070fe100) at /usr/src/sys/kern/kern_intr.c:1181
#13 0xffffffff80527b43 in fork_exit (callout=0xffffffff8052af70 <ithread_loop>, arg=0xffffff00070fe100, frame=0xffffff800013fc80) at /usr/src/sys/kern/kern_fork.c:811
#14 0xffffffff80800f3e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:554
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()

>How-To-Repeat:
Direct network traffic to the machine and wait some hours.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Mar 16 12:33:10 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=155597 

From: Vladimir Kutakov <vova@ashmanov.com>
To: bug-followup@FreeBSD.org, vova@ashmanov.com
Cc:  
Subject: Re: kern/155597: [panic] Kernel panics with "sbdrop" message
Date: Thu, 24 Mar 2011 12:35:34 +0300

 We are looking into this problem a bit. The panic is reproducible easily by means of big http trafic. We ran apache benchmark on 3 next servers: ab -c40 -n20000000 'http://thehost/somesmallfile'. After just some minutes panic occurs.
 
 However, after decreasing hw.igb.num_queues to 4 (default value is 8), it occurs only after 2 hours.
 
 Finally, the server with hw.igb.num_queues=1 works good already 7 days.
 
 It seems that some problem happens during parallel tcp processing.
 
 
 -- 
 WBR,
 Vladimir
 mailto:vova@ashmanov.com
 

From: Arnaud Lacombe <lacombar@gmail.com>
To: bug-followup@FreeBSD.org, vova@ashmanov.com
Cc:  
Subject: Re: kern/155597: [panic] Kernel panics with &quot;sbdrop&quot; message
Date: Tue, 16 Aug 2011 16:43:59 -0400

 Hi,
 
 Does this still happen with 9.0-BETA ?
 
 If so, could this be a use-after-free, where an mbuf is freed (during
 an m_pullup() or alike), but the old reference is still kept around,
 gets added to the sockbuf, then the mbuf is re-allocated and corrupt
 the sockbuf ?

From: Vladimir Kutakov <vova@ashmanov.com>
To: Arnaud Lacombe <lacombar@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/155597: [panic] Kernel panics with &quot;sbdrop&quot; message
Date: Wed, 11 Jan 2012 17:55:41 +0400

 We have tried RELENG_8_2 and the panic doesn't happen anymore. Many =
 thanks to the FreeBSD team.
 
 On Aug 17, 2011, at 1:43 AM, Arnaud Lacombe wrote:
 
 > Hi,
 >=20
 > Does this still happen with 9.0-BETA ?
 >=20
 > If so, could this be a use-after-free, where an mbuf is freed (during
 > an m_pullup() or alike), but the old reference is still kept around,
 > gets added to the sockbuf, then the mbuf is re-allocated and corrupt
 > the sockbuf ?
 
 
 --
 =D0=92=D0=BB=D0=B0=D0=B4=D0=B8=D0=BC=D0=B8=D1=80 =D0=9A=D1=83=D1=82=D0=B0=D0=
 =BA=D0=BE=D0=B2
 =D0=A2=D0=B5=D1=85=D0=BD=D0=B8=D1=87=D0=B5=D1=81=D0=BA=D0=B8=D0=B9 =
 =D0=B4=D0=B8=D1=80=D0=B5=D0=BA=D1=82=D0=BE=D1=80
 =D0=97=D0=90=D0=9E "=D0=9F=D0=BE=D0=B8=D1=81=D0=BA=D0=BE=D0=B2=D1=8B=D0=B5=
  =D1=82=D0=B5=D1=85=D0=BD=D0=BE=D0=BB=D0=BE=D0=B3=D0=B8=D0=B8"
 vova@ashmanov.com
 
>Unformatted:
