From nobody@FreeBSD.org  Wed Jul 21 06:16:10 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id C0AC41065781
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Jul 2010 06:16:10 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id AF0148FC16
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Jul 2010 06:16:10 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o6L6GAb7006773
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 21 Jul 2010 06:16:10 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o6L6GAjS006772;
	Wed, 21 Jul 2010 06:16:10 GMT
	(envelope-from nobody)
Message-Id: <201007210616.o6L6GAjS006772@www.freebsd.org>
Date: Wed, 21 Jul 2010 06:16:10 GMT
From: Mike Andrews <mandrews@bit0.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: 8.1-RELEASE "panic: sbdrop" and "panic: sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         148807
>Category:       kern
>Synopsis:       [panic] 8.1-RELEASE "panic: sbdrop" and "panic: sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    andre
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jul 21 06:20:01 UTC 2010
>Closed-Date:    
>Last-Modified:  Tue Sep 28 08:30:05 UTC 2010
>Originator:     Mike Andrews
>Release:        8.1-RELEASE amd64
>Organization:
>Environment:
FreeBSD bourbon.fark.com 8.1-RELEASE FreeBSD 8.1-RELEASE #6: Sat Jul 17 18:30:12 EDT 2010     root@beer.int.fark.com:/usr/obj/usr/src/sys/FARK64  amd64
>Description:
Under heavy load (i.e. enough to make an 8 GB machine start to swap) I'm
seeing multiple (identical) machines panic repeatedly with the above panic
 messages.  The panics go away once load goes down.

Also very occasionally seeing "em0: discard frame w/o packet header" before
the panics, though not very often.

Hardware is five identical Supermicro PDSMI+ systems, Q6600, 8 GB ECC memory.

The only references I'm finding to these panics on Google seem to point at
either IPv6 or em as potential issues, and we're using both.  :)  Specifically
http://groups.google.com/group/mailing.freebsd.net/browse_thread/thread/28db45413a889411
looks VERY similar.  I have not yet tried shutting off IPv6 or at least
switching some services back to IPv4.  (Our v6 usage is all internal-only.)

core.txt.* files are at http://www.bit0.com/tmp/core.txt.20100721.tar.gz

I have minidumps as well but as they may contain some proprietary data
I'd rather not post 'em online, however I can run whatever kgdb commands
are needed to help troubleshoot.  :)
>How-To-Repeat:
See above
>Fix:
Unknown

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: brucec 
Responsible-Changed-When: Wed Jul 21 16:30:32 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=148807 
Responsible-Changed-From-To: freebsd-net->andre 
Responsible-Changed-By: andre 
Responsible-Changed-When: Thu Aug 12 08:24:22 UTC 2010 
Responsible-Changed-Why:  
Take over. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=148807 

From: Andre Oppermann <andre@freebsd.org>
To: mandrews@bit0.com
Cc: bug-followup@freebsd.org
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Thu, 12 Aug 2010 10:35:24 +0200

 Mike,
 
 I see that you use ZERO_COPY_SOCKETS in your kernel file base on the
 information in the crash dumps.  ZERO_COPY_SOCKETS may have some bugs
 regarding the mbuf and vm page lifecycle.  Their use is not really
 supported at the moment and we have highly optimized the normal send
 path.  So further optimizations are not really necessary.
 
 Please recompile your kernel without ZERO_COPY_SOCKETS and report whether
 you still see sbdrop and sockbuf panics.
 
 Debugging ZERO_COPY_SOCKETS is very difficult because of the complex
 interactions between the VM, mbuf and sockbuf systems.
 
 -- 
 Andre

From: Mike Andrews <mandrews@bit0.com>
To: Andre Oppermann <andre@freebsd.org>
Cc: bug-followup@freebsd.org, pluknet@gmail.com
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Thu, 12 Aug 2010 12:21:16 -0400

 I removed ZERO_COPY_SOCKETS and am still seeing panics.  A new set of 
 core.txt files is at http://www.bit0.com/tmp/core.txt.20100812.tar.gz

From: Andre Oppermann <andre@freebsd.org>
To: Mike Andrews <mandrews@bit0.com>
Cc: bug-followup@freebsd.org, pluknet@gmail.com
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Fri, 13 Aug 2010 12:02:20 +0200

 This is a multi-part message in MIME format.
 --------------000602000600040105030805
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 On 12.08.2010 18:21, Mike Andrews wrote:
 > I removed ZERO_COPY_SOCKETS and am still seeing panics.  A new set of
 > core.txt files is at http://www.bit0.com/tmp/core.txt.20100812.tar.gz
 
 Please try the attached patch and compile your kernel with INVARIANTS.
 It contains some debugging code to catch any corruption to the sockbuf
 when it happens and may also a few potential fixes.
 
 We can narrow it down now.
 
 -- 
 Andre
 
 --------------000602000600040105030805
 Content-Type: text/plain;
  name="sockbuf_debug-20100813.diff"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment;
  filename="sockbuf_debug-20100813.diff"
 
 Index: kern/uipc_sockbuf.c
 ===================================================================
 --- kern/uipc_sockbuf.c	(revision 211227)
 +++ kern/uipc_sockbuf.c	(working copy)
 @@ -68,6 +68,22 @@
  static void	sbdrop_internal(struct sockbuf *sb, int len);
  static void	sbflush_internal(struct sockbuf *sb);
  
 +#ifdef INVARIANTS
 +static int
 +sb_cc_check(struct sockbuf *sb)
 +{
 +	struct mbuf *n = sb->sb_mb;
 +	int slen = 0;
 +
 +	while (n) {
 +		slen = m_length(n, &n);
 +		n = n->m_nextpkt;
 +	}
 +
 +	return (slen == sb->sb_cc ? 1 : 0);
 +}
 +#endif
 +
  /*
   * Socantsendmore indicates that no more data will be sent on the socket; it
   * would normally be applied to a socket when the user informs the system
 @@ -528,6 +544,9 @@
  
  	SBLASTMBUFCHK(sb);
  
 +	KASSERT(sb_cc_check(sb),
 +	    ("%s: sb_cc != total mbuf length - before", __func__));
 +
  	/* Remove all packet headers and mbuf tags to get a pure data chain. */
  	m_demote(m, 1);
  	
 @@ -535,6 +554,9 @@
  
  	sb->sb_lastrecord = sb->sb_mb;
  	SBLASTRECORDCHK(sb);
 +
 +	KASSERT(sb_cc_check(sb),
 +	    ("%s: sb_cc != total mbuf length - after", __func__));
  }
  
  /*
 @@ -811,6 +833,9 @@
  sbflush_internal(struct sockbuf *sb)
  {
  
 +	KASSERT(sb_cc_check(sb),
 +	    ("%s: sb_cc != total mbuf length", __func__));
 +
  	while (sb->sb_mbcnt) {
  		/*
  		 * Don't call sbdrop(sb, 0) if the leading mbuf is non-empty:
 @@ -851,10 +876,22 @@
  	struct mbuf *m;
  	struct mbuf *next;
  
 -	next = (m = sb->sb_mb) ? m->m_nextpkt : 0;
 +	KASSERT(len <= sb->sb_cc,
 +	    ("%s: len > sb->sb_cc", __func__));
 +	KASSERT(sb->sb_cc >= sb->sb_sndptroff,
 +	    ("%s: sb_sndptroff > sb_cc", __func__));
 +	KASSERT(sb_cc_check(sb),
 +	    ("%s: sb_cc != total mbuf length - enter", __func__));
 +
 +	if (len > sb->sb_sndptroff) {
 +		sb->sb_sndptroff = 0;
 +		sb->sb_sndptr = NULL;
 +	}
 +
 +	next = (m = sb->sb_mb) ? m->m_nextpkt : NULL;
  	while (len > 0) {
 -		if (m == 0) {
 -			if (next == 0)
 +		if (m == NULL) {
 +			if (next == NULL)
  				panic("sbdrop");
  			m = next;
  			next = m->m_nextpkt;
 @@ -874,7 +911,7 @@
  		sbfree(sb, m);
  		m = m_free(m);
  	}
 -	while (m && m->m_len == 0) {
 +	while (m != NULL && m->m_len == 0) {
  		sbfree(sb, m);
  		m = m_free(m);
  	}
 @@ -891,9 +928,13 @@
  	if (m == NULL) {
  		sb->sb_mbtail = NULL;
  		sb->sb_lastrecord = NULL;
 +		sb->sb_sndptr = NULL;
  	} else if (m->m_nextpkt == NULL) {
  		sb->sb_lastrecord = m;
  	}
 +
 +	KASSERT(sb_cc_check(sb),
 +	    ("%s: sb_cc != total mbuf length - exit", __func__));
  }
  
  /*
 @@ -922,13 +963,15 @@
   * avoid traversal of the entire socket buffer for larger offsets.
   */
  struct mbuf *
 -sbsndptr(struct sockbuf *sb, u_int off, u_int len, u_int *moff)
 +sbsndptr(struct sockbuf *sb, int off, int len, int *moff)
  {
  	struct mbuf *m, *ret;
  
  	KASSERT(sb->sb_mb != NULL, ("%s: sb_mb is NULL", __func__));
  	KASSERT(off + len <= sb->sb_cc, ("%s: beyond sb", __func__));
  	KASSERT(sb->sb_sndptroff <= sb->sb_cc, ("%s: sndptroff broken", __func__));
 +	KASSERT(sb_cc_check(sb),
 +	    ("%s: sb_cc != total mbuf length", __func__));
  
  	/*
  	 * Is off below stored offset? Happens on retransmits.
 Index: sys/sockbuf.h
 ===================================================================
 --- sys/sockbuf.h	(revision 211227)
 +++ sys/sockbuf.h	(working copy)
 @@ -152,7 +152,7 @@
  int	sbreserve_locked(struct sockbuf *sb, u_long cc, struct socket *so,
  	    struct thread *td);
  struct mbuf *
 -	sbsndptr(struct sockbuf *sb, u_int off, u_int len, u_int *moff);
 +	sbsndptr(struct sockbuf *sb, int off, int len, int *moff);
  void	sbtoxsockbuf(struct sockbuf *sb, struct xsockbuf *xsb);
  int	sbwait(struct sockbuf *sb);
  int	sblock(struct sockbuf *sb, int flags);
 Index: netinet/tcp_output.c
 ===================================================================
 --- netinet/tcp_output.c	(revision 211227)
 +++ netinet/tcp_output.c	(working copy)
 @@ -153,7 +153,7 @@
  	int idle, sendalot;
  	int sack_rxmit, sack_bytes_rxmt;
  	struct sackhole *p;
 -	int tso = 0;
 +	int tso;
  	struct tcpopt to;
  #if 0
  	int maxburst = TCP_MAXBURST;
 @@ -211,6 +211,7 @@
  	    SEQ_LT(tp->snd_nxt, tp->snd_max))
  		tcp_sack_adjust(tp);
  	sendalot = 0;
 +	tso = 0;
  	off = tp->snd_nxt - tp->snd_una;
  	sendwin = min(tp->snd_wnd, tp->snd_cwnd);
  	sendwin = min(sendwin, tp->snd_bwnd);
 @@ -490,9 +491,9 @@
  		} else {
  			len = tp->t_maxseg;
  			sendalot = 1;
 -			tso = 0;
  		}
  	}
 +
  	if (sack_rxmit) {
  		if (SEQ_LT(p->rxmit + len, tp->snd_una + so->so_snd.sb_cc))
  			flags &= ~TH_FIN;
 @@ -766,7 +794,7 @@
  	 */
  	if (len) {
  		struct mbuf *mb;
 -		u_int moff;
 +		int moff;
  
  		if ((tp->t_flags & TF_FORCEDATA) && len == 1)
  			TCPSTAT_INC(tcps_sndprobe);
 
 --------------000602000600040105030805--

From: Mike Andrews <mandrews@bit0.com>
To: Andre Oppermann <andre@freebsd.org>
Cc: bug-followup@freebsd.org, pluknet@gmail.com
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Fri, 13 Aug 2010 14:25:01 -0400

 I'll try this this evening.  Would options SOCKBUF_DEBUG help any?
 

From: pluknet <pluknet@gmail.com>
To: Andre Oppermann <andre@freebsd.org>
Cc: Mike Andrews <mandrews@bit0.com>, bug-followup@freebsd.org
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Sat, 14 Aug 2010 10:07:00 +0400

 On 13 August 2010 14:02, Andre Oppermann <andre@freebsd.org> wrote:
 > On 12.08.2010 18:21, Mike Andrews wrote:
 >>
 >> I removed ZERO_COPY_SOCKETS and am still seeing panics. =A0A new set of
 >> core.txt files is at http://www.bit0.com/tmp/core.txt.20100812.tar.gz
 >
 > Please try the attached patch and compile your kernel with INVARIANTS.
 > It contains some debugging code to catch any corruption to the sockbuf
 > when it happens and may also a few potential fixes.
 >
 > We can narrow it down now.
 
 [as I (occasionally) was added to cc: list]
 so, I tried this patch and box got panic near starting multiuser:
 
 My testbox has no swapspace, no debug symbols this time :(
 
 panic: sbflush_internal: sb_cc !=3D total mbuf length
 
 db> bt
 Tracing pid 982 tid 100111 td 0xffffff0008032440
 kdb_enter() at kdb_enter+0x3d
 panic() at panic+0x17b
 sbflush_internal() at sbflush_internal+0x98
 sbrelease_internal() at sbrelease_internal+0x1c
 sofree() at sofree+0x1bb
 soclose() at soclose+0x32b
 _fdrop() at _fdrop+0x23
 closef() at closef+0x5b
 kern_close() at kern_close+0x110
 syscallenter() at syscallenter+0x1aa
 syscall() at syscall+0x4c
 Xfast_syscall() at Xfast_syscall+0xe2
 --- syscall (6, FreeBSD ELF64, close), rip =3D 0x800742fbc, rsp =3D
 0x7fffffffd2f8, rbp =3D 0x800c07060 ---
 db> show proc 982
 Process 982 (ypbind) at 0xffffff0008034000:
  state: NORMAL
  uid: 0  gids: 0
  parent: pid 980 at 0xffffff00080358c0
  ABI: FreeBSD ELF64
  arguments: /usr/sbin/ypbind
  threads: 1
 100111                   Run     CPU 1                       ypbind
 
 --=20
 wbr,
 pluknet

From: =?UTF-8?Q?Marcin_Wi=C5=9Bnicki?= <mwisnicki@gmail.com>
To: bug-followup <bug-followup@FreeBSD.org>, mandrews <mandrews@bit0.com>
Cc:  
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Tue, 24 Aug 2010 22:24:34 +0200

 Same thing happened to me today. I don't have ZERO_COPY_SOCKETS,
 however I use IPv6.
 Until today my system was completely solid (no panics for weeks/months
 of uptime) and only change I did recently was enabling ALTQ in pf.conf
 week ago but it was working for a couple of days with no ill effects.
 
 Panic happened when I resumed Win7 client and opened web browser. I'm
 running Squid-3.1 with pf transparent redirection.
 
 System: FreeBSD 8.1-PRERELEASE #4: Wed Jul 14 21:47:49 CEST 2010
 Config: GENERIC - (everything that can go into kld) + SW_WATCHDOG +
 DEVICE_POLLING (not used) + ALTQ + DDB
 KLDs: kernel vesa.ko geom_journal.ko geom_label.ko if_rl.ko miibus.ko
 if_vr.ko snd_via8233.ko sound.ko usb.ko ukbd.ko ums.ko umass.ko cam.ko
 agp.ko uhci.ko ehci.ko kbdmux.ko geom_part_gpt.ko atapicam.ko if_br
 idge.ko bridgestp.ko wlan_wep.ko wlan.ko wlan_tkip.ko wlan_ccmp.ko
 wlan_xauth.ko wlan_acl.ko cpufreq.ko netgraph.ko aio.ko sem.ko acpi.ko
 geom_eli.ko crypto.ko zlib.ko procfs.ko pseudofs.ko linprocfs.ko linu
 x.ko nullfs.ko pf.ko if_tun.ko ng_ether.ko ng_pppoe.ko ng_socket.ko
 if_stf.ko nfsclient.ko nfs_common.ko krpc.ko nfsserver.ko nfssvc.ko
 nfslockd.ko ng_mppc.ko rc4.ko fuse.ko accf_http.ko accf_data.ko
 
 Dmesg:
 
 Unread portion of the kernel message buffer:
 panic: sbsndptr: sockbuf 0xc6ea65bc and mbuf 0xc6248100 clashing
 cpuid = 0
 KDB: stack backtrace:
 db_trace_self_wrapper(c0764c90,e4208910,c04fa0b9,c077f5e6,0,...) at
 db_trace_self_wrapper+0x26
 kdb_backtrace(c077f5e6,0,c0767e81,e420891c,0,...) at kdb_backtrace+0x29
 panic(c0767e81,c0749504,c6ea65bc,c6248100,c4eb24f0,...) at panic+0x119
 sbsndptr(c6ea65bc,0,4fd,e42089cc,c4f56ef8,...) at sbsndptr+0xa6
 tcp_output(c4eb24f0,0,0,c3e45800,c47b0700,...) at tcp_output+0xc84
 tcp_do_segment(c4eb24f0,28,0,0,2,...) at tcp_do_segment+0x1f45
 tcp_input(c6079d00,14,c3e45800,1,0,...) at tcp_input+0x11d0
 ip_input(c6079d00,e4208bb0,80246,24,e4208bd8,...) at ip_input+0x6e5
 netisr_dispatch_src(1,0,c6079d00,e4208c00,c05abb61,...) at
 netisr_dispatch_src+0x89
 netisr_dispatch(1,c6079d00,0,c3e45800,c62d8808,...) at netisr_dispatch+0x20
 ether_demux(c3e45800,c6079d00,3,0,3,...) at ether_demux+0x161
 ether_input(c3e45800,c6079d00,c0940756,587,c3e8eaec,...) at ether_input+0x323
 vr_rxeof(c3e8eaec,0,c0940756,68b,c3e8eaec,...) at vr_rxeof+0x219
 vr_intr(c3e8e000,0,109,801bafb1,c8ac,...) at vr_intr+0x114
 intr_event_execute_handlers(c3d407f8,c3d3d300,c075f556,52d,c3d3d370,...)
 at intr_event_execute_handlers+0x14b
 ithread_loop(c3e1a0e0,e4208d38,7afffffb,ddfffbff,84ff77ff,...) at
 ithread_loop+0x6b
 fork_exit(c04d0fd0,c3e1a0e0,e4208d38) at fork_exit+0x90
 fork_trampoline() at fork_trampoline+0x8
 --- trap 0, eip = 0, esp = 0xe4208d70, ebp = 0 ---
 Uptime: 1d17h48m27s
 Physical memory: 1007 MB
 
 Kgdb:
 
 (kgdb) bt
 #0  doadump () at pcpu.h:230
 #1  0xc04f9e57 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
 #2  0xc04fa0f5 in panic (fmt=Variable "fmt" is not available.
 ) at /usr/src/sys/kern/kern_shutdown.c:590
 #3  0xc0558126 in sbsndptr (sb=0xc6ea65bc, off=0,
 len=dwarf2_read_address: Corrupted DWARF expression.
 ) at /usr/src/sys/kern/uipc_sockbuf.c:954
 #4  0xc0636c04 in tcp_output (tp=0xc4eb24f0) at
 /usr/src/sys/netinet/tcp_output.c:817
 #5  0xc0633a85 in tcp_do_segment (m=0xc6079d00, th=0xc62d882a,
 so=0xc6ea64d4, tp=0xc4eb24f0, drop_hdrlen=40, tlen=0, iptos=0 '\0',
 ti_locked=2)
     at /usr/src/sys/netinet/tcp_input.c:2693
 #6  0xc0635080 in tcp_input (m=0xc6079d00, off0=20) at
 /usr/src/sys/netinet/tcp_input.c:1029
 #7  0xc05cc6f5 in ip_input (m=0xc6079d00) at /usr/src/sys/netinet/ip_input.c:793
 #8  0xc05aed89 in netisr_dispatch_src (proto=1, source=0,
 m=0xc6079d00) at /usr/src/sys/net/netisr.c:917
 #9  0xc05af050 in netisr_dispatch (proto=1, m=0xc6079d00) at
 /usr/src/sys/net/netisr.c:1004
 #10 0xc05abb61 in ether_demux (ifp=0xc3e45800, m=0xc6079d00) at
 /usr/src/sys/net/if_ethersubr.c:901
 #11 0xc05ac0c3 in ether_input (ifp=0xc3e45800, m=0xc6079d00) at
 /usr/src/sys/net/if_ethersubr.c:760
 #12 0xc093bea9 in vr_rxeof (sc=0xc3e8e000) at
 /usr/src/sys/modules/vr/../../dev/vr/if_vr.c:1416
 #13 0xc093f374 in vr_intr (arg=0xc3e8e000) at
 /usr/src/sys/modules/vr/../../dev/vr/if_vr.c:1710
 #14 0xc04cf94b in intr_event_execute_handlers (p=0xc3d407f8,
 ie=0xc3d3d300) at /usr/src/sys/kern/kern_intr.c:1220
 #15 0xc04d103b in ithread_loop (arg=0xc3e1a0e0) at
 /usr/src/sys/kern/kern_intr.c:1233
 #16 0xc04cd1d0 in fork_exit (callout=0xc04d0fd0 <ithread_loop>,
 arg=0xc3e1a0e0, frame=0xe4208d38) at /usr/src/sys/kern/kern_fork.c:844
 #17 0xc070eb94 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:273
 
 (kgdb) frame 3
 #3  0xc0558126 in sbsndptr (sb=0xc6ea65bc, off=0,
 len=dwarf2_read_address: Corrupted DWARF expression.
 ) at /usr/src/sys/kern/uipc_sockbuf.c:954
 954                     panic("%s: sockbuf %p and mbuf %p clashing",
 __func__, sb, ret);
 
 (kgdb) p *sb
 $1 = {sb_sel = {si_tdlist = {tqh_first = 0x0, tqh_last = 0xc6ea65bc},
 si_note = {kl_list = {slh_first = 0x0}, kl_lock = 0xc04c6910
 <knlist_mtx_lock>,
       kl_unlock = 0xc04c68c0 <knlist_mtx_unlock>, kl_assert_locked =
 0xc04c35e0 <knlist_mtx_assert_locked>,
       kl_assert_unlocked = 0xc04c35f0 <knlist_mtx_assert_unlocked>,
 kl_lockarg = 0xc6ea65e0}, si_mtx = 0xc4490448}, sb_mtx = {lock_object
 = {
       lo_name = 0xc0767ab5 "so_snd", lo_flags = 16973824, lo_data = 0,
 lo_witness = 0x0}, mtx_lock = 3285714816}, sb_sx = {lock_object = {
       lo_name = 0xc0767f89 "so_snd_sx", lo_flags = 36896768, lo_data =
 0, lo_witness = 0x0}, sx_lock = 1}, sb_state = 0, sb_mb = 0xc6248100,
   sb_mbtail = 0xc6292a00, sb_lastrecord = 0xc6248100, sb_sndptr = 0x0,
 sb_sndptroff = 601, sb_cc = 1277, sb_hiwat = 33580, sb_mbcnt = 2048,
 sb_mcnt = 8,
   sb_ccnt = 0, sb_mbmax = 262144, sb_ctl = 0, sb_lowat = 2048,
 sb_timeo = 0, sb_flags = 2048, sb_upcall = 0, sb_upcallarg = 0x0}
 
 (kgdb) p *buf
 $2 = {b_bufobj = 0xc6835b20, b_bcount = 16384, b_caller1 = 0x0,
   b_data = 0xd7f90000 "<unreadable garbage>"..., b_error = 0, b_iocmd
 = 1 '\001', b_ioflags = 2 '\002',
   b_iooffset = 469651013632, b_resid = 0, b_iodone = 0, b_blkno =
 917287136, b_offset = 72007680, b_bobufs = {tqe_next = 0xd7e1de00,
     tqe_prev = 0xd7d52888}, b_left = 0x0, b_right = 0x0, b_vflags = 0,
 b_freelist = {tqe_next = 0xd7d90f20, tqe_prev = 0xd7df1adc}, b_qindex
 = 1,
   b_flags = 805306400, b_xflags = 2 '\002', b_lock = {lock_object =
 {lo_name = 0xc0768b4a "bufwait", lo_flags = 91422720, lo_data = 0,
 lo_witness = 0x0},
     lk_lock = 1, lk_timo = 0, lk_pri = 80}, b_bufsize = 16384,
 b_runningbufspace = 0,
   b_kvabase = 0xd7f90000 "<unreadable garbage>"..., b_kvasize = 16384,
 b_lblkno = 4395, b_vp = 0xc6835a78,
   b_dirtyoff = 0, b_dirtyend = 0, b_rcred = 0x0, b_wcred = 0x0,
 b_saveaddr = 0xd7f90000, b_pager = {pg_reqpage = 0}, b_cluster =
 {cluster_head = {
       tqh_first = 0xd7ef93a0, tqh_last = 0xd7dfd1b0}, cluster_entry =
 {tqe_next = 0xd7ef93a0, tqe_prev = 0xd7dfd1b0}}, b_pages =
 {0xc1b6e200, 0xc150bde8,
     0xc150be30, 0xc1ba1c08, 0x0 <repeats 28 times>}, b_npages = 4,
 b_dep = {lh_first = 0x0}, b_fsprivate1 = 0x0, b_fsprivate2 = 0x0,
 b_fsprivate3 = 0x0,
   b_pin_count = 0}
 
 
 "<unreadable garbage>" were inserted by me in place of contents.
 I wonder why gdb is never fully working with anything more than -O0
 (variables not available, corrupted dwarf, etc.). I didn't have in
 distant past (FreeBSD 5.x I think).

From: Mike Andrews <mandrews@bit0.com>
To: bug-followup@FreeBSD.org, mandrews@bit0.com
Cc:  
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Thu, 09 Sep 2010 13:20:23 -0400

 With the patch and with INVARIANTS it's harder to reproduce but still 
 possible.  I had a few crashes that hung while attempting to write dumps.
 
 Today I had this one on a different machine (same kernel though) that 
 was NOT under significant load but shares other characteristics -- same 
 hardware, same use of IPv6, etc.
 
 http://www.bit0.com/tmp/core.beer.2010-09-09.txt.gz
 

From: Mike Andrews <mandrews@bit0.com>
To: bug-followup@FreeBSD.org, mandrews@bit0.com
Cc:  
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Wed, 15 Sep 2010 21:53:19 -0400

 Got a panic when attempting to manually reboot just now.
 During the shutdown:
 
 panic: sbflush_internal: sb_cc != total mbuf length
 cpuid = 1
 Uptime: 1d6h22m22s
 Physical memory: 8179 MB
 Dumping 3250 MB: 3235 3219 3203
 
 ...and it never finished the dump, it just hung at 3203.
 
 I'm going to try running today's 8.1-STABLE instead of -RELEASE.
 We'll see if that helps.

From: pluknet <pluknet@gmail.com>
To: Andre Oppermann <andre@freebsd.org>
Cc: Mike Andrews <mandrews@bit0.com>, bug-followup@freebsd.org
Subject: Re: kern/148807: [panic] 8.1-RELEASE "panic: sbdrop" and "panic:
 sbsndptr: sockbuf _ and mbuf _ clashing" under heavy load
Date: Tue, 28 Sep 2010 12:20:23 +0400

 hi there,
 
 If my eyes do not deceive me, the source of my panic
 posted above was in the wrong KASSERT() check.
 Should be '+=' in slen = m_length(n, &n) like this:
 
 #ifdef INVARIANTS
 static int
 sb_cc_check(struct sockbuf *sb)
 {
         struct mbuf *n = sb->sb_mb;
         int slen = 0;
 
         while (n) {
                 slen += m_length(n, &n);
                 n = n->m_nextpkt;
         }
 
         return (slen == sb->sb_cc ? 1 : 0);
 }
 #endif
>Unformatted:
