From nobody@FreeBSD.org  Wed Apr 24 19:20:42 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id A1A3B4B5
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 24 Apr 2013 19:20:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 91B5B1850
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 24 Apr 2013 19:20:42 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r3OJKgUd038025
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 24 Apr 2013 19:20:42 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r3OJKgOD038024;
	Wed, 24 Apr 2013 19:20:42 GMT
	(envelope-from nobody)
Message-Id: <201304241920.r3OJKgOD038024@red.freebsd.org>
Date: Wed, 24 Apr 2013 19:20:42 GMT
From: Nate Denning <nate.denning@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Kernel panic: general protection fault in tcp_do_segment
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         178116
>Category:       kern
>Synopsis:       [ipfilter] [panic] Kernel panic: general protection fault in tcp_do_segment
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    cy
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Apr 24 19:30:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Wed Jul 03 05:25:29 UTC 2013
>Originator:     Nate Denning
>Release:        9.1-STABLE
>Organization:
>Environment:
FreeBSD xxx 9.1-STABLE FreeBSD 9.1-STABLE #0 r249765: Mon Apr 22 19:22:46 MDT 2013     nate@xxx:/usr/obj/usr/src-9-stable/sys/GENERIC  amd64
>Description:
I'm getting the below panic at random intervals ranging from about 1 hour to 2 days.

To note, this is a repost of PR 178017 (http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/178017) which was closed due to my email address bouncing (fixed; sorry about that). In that PR it was suggested I upgrade to 9.1-STABLE which I have done to no avail. Stack trace from 9.1-STABLE and 9.1-RELEASE is below.

--- core.txt from 9.1-STABLE (r249765) ---

Unread portion of the kernel message buffer:
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (em1 que)
trap number             = 9
panic: general protection fault
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff80952f46 at kdb_backtrace+0x66
#1 0xffffffff8091a2de at panic+0x1ce
#2 0xffffffff80ca8b80 at trap_fatal+0x290
#3 0xffffffff80ca9391 at trap+0x241
#4 0xffffffff80c92813 at calltrap+0x8
#5 0xffffffff809840a8 at sbappendstream_locked+0x18
#6 0xffffffff80aae74e at tcp_do_segment+0x29ae
#7 0xffffffff80aafa36 at tcp_input+0xbb6
#8 0xffffffff80a403ca at ip_input+0xaa
#9 0xffffffff809e2578 at netisr_dispatch_src+0x218
#10 0xffffffff809d920d at ether_demux+0x14d
#11 0xffffffff809d94e4 at ether_nh_input+0x1f4
#12 0xffffffff809e2578 at netisr_dispatch_src+0x218
#13 0xffffffff804dd248 at em_rxeof+0x1c8
#14 0xffffffff804dd6f8 at em_handle_que+0x48
#15 0xffffffff8095f9b4 at taskqueue_run_locked+0x74
#16 0xffffffff80960966 at taskqueue_thread_loop+0x46
#17 0xffffffff808e83af at fork_exit+0x11f
Uptime: 1d4h4m46s
Dumping 4007 out of 8158 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipl.ko...Reading symbols from /boot/kernel/ipl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipl.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:229
229     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:229
#1  0xffffffff80919db4 in kern_reboot (howto=260)
    at /usr/src-9-stable/sys/kern/kern_shutdown.c:449
#2  0xffffffff8091a2b7 in panic (fmt=0x1 <Address 0x1 out of bounds>)
    at /usr/src-9-stable/sys/kern/kern_shutdown.c:637
#3  0xffffffff80ca8b80 in trap_fatal (frame=0x9, eva=<value optimized out>)
    at /usr/src-9-stable/sys/amd64/amd64/trap.c:878
#4  0xffffffff80ca9391 in trap (frame=0xffffff83428df610)
    at /usr/src-9-stable/sys/amd64/amd64/trap.c:605
#5  0xffffffff80c92813 in calltrap ()
    at /usr/src-9-stable/sys/amd64/amd64/exception.S:228
#6  0xffffffff80983cd6 in sbcompress (sb=0xfffffe002f6fa338, 
    m=0x2a63737717c07a, n=0xfffffe006529b800)
    at /usr/src-9-stable/sys/kern/uipc_sockbuf.c:759
#7  0xffffffff809840a8 in sbappendstream_locked (sb=0xfffffe002f6fa338, 
    m=<value optimized out>) at /usr/src-9-stable/sys/kern/uipc_sockbuf.c:531
#8  0xffffffff80aae74e in tcp_do_segment (m=0xfffffe006529b800, 
    th=<value optimized out>, so=0xfffffe002f6fa2a8, tp=0xfffffe003eb683d0, 
    drop_hdrlen=52, tlen=48, iptos=8 '\b', ti_locked=1)
    at /usr/src-9-stable/sys/netinet/tcp_input.c:1835
#9  0xffffffff80aafa36 in tcp_input (m=0xfffffe006529b800, 
    off0=<value optimized out>)
    at /usr/src-9-stable/sys/netinet/tcp_input.c:1393
#10 0xffffffff80a403ca in ip_input (m=0xfffffe006529b800)
    at /usr/src-9-stable/sys/netinet/ip_input.c:760
#11 0xffffffff809e2578 in netisr_dispatch_src (proto=1, 
    source=<value optimized out>, m=<value optimized out>)
    at /usr/src-9-stable/sys/net/netisr.c:1013
#12 0xffffffff809d920d in ether_demux (ifp=0xfffffe0006050800, 
    m=0xfffffe006529b800) at /usr/src-9-stable/sys/net/if_ethersubr.c:940
#13 0xffffffff809d94e4 in ether_nh_input (m=<value optimized out>)
    at /usr/src-9-stable/sys/net/if_ethersubr.c:759
#14 0xffffffff809e2578 in netisr_dispatch_src (proto=9, 
    source=<value optimized out>, m=<value optimized out>)
    at /usr/src-9-stable/sys/net/netisr.c:1013
#15 0xffffffff804dd248 in em_rxeof (rxr=0xfffffe0006100a00, count=99, 
    done=0x0) at /usr/src-9-stable/sys/dev/e1000/if_em.c:4515
#16 0xffffffff804dd6f8 in em_handle_que (context=<value optimized out>, 
    pending=<value optimized out>)
    at /usr/src-9-stable/sys/dev/e1000/if_em.c:1518
#17 0xffffffff8095f9b4 in taskqueue_run_locked (queue=0xfffffe0006118480)
    at /usr/src-9-stable/sys/kern/subr_taskqueue.c:312
#18 0xffffffff80960966 in taskqueue_thread_loop (arg=<value optimized out>)
    at /usr/src-9-stable/sys/kern/subr_taskqueue.c:501
#19 0xffffffff808e83af in fork_exit (
    callout=0xffffffff80960920 <taskqueue_thread_loop>, 
    arg=0xffffff8000ac6730, frame=0xffffff83428dfc40)
    at /usr/src-9-stable/sys/kern/kern_fork.c:988
#20 0xffffffff80c92d3e in fork_trampoline ()
    at /usr/src-9-stable/sys/amd64/amd64/exception.S:602
#21 0x0000000000000000 in ?? ()
(kgdb) 


--- core.txt from 9.1-RELEASE (r249710) ---

Fatal trap 9: general protection fault while in kernel mode
cpuid = 4; apic id = 32
instruction pointer     = 0x20:0xffffffff8094b595
stack pointer           = 0x28:0xffffff834291c6d0
frame pointer           = 0x28:0xffffff834291c6f0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (em0 que)
trap number             = 9
panic: general protection fault
cpuid = 4
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd88d5 at trap+0x105
#4 0xffffffff80bc315f at calltrap+0x8
#5 0xffffffff80a72141 at tcp_do_segment+0x2991
#6 0xffffffff80a73313 at tcp_input+0xa63
#7 0xffffffff80a043dc at ip_input+0xac
#8 0xffffffff809adafb at netisr_dispatch_src+0x20b
#9 0xffffffff809a35cd at ether_demux+0x14d
#10 0xffffffff809a38a4 at ether_nh_input+0x1f4
#11 0xffffffff809adafb at netisr_dispatch_src+0x20b
#12 0xffffffff804ad8ca at em_rxeof+0x1ca
#13 0xffffffff804add48 at em_handle_que+0x48
#14 0xffffffff8092cf55 at taskqueue_run_locked+0x85
#15 0xffffffff8092ded6 at taskqueue_thread_loop+0x46
#16 0xffffffff808bb9ef at fork_exit+0x11f
#17 0xffffffff80bc368e at fork_trampoline+0xe
Uptime: 6h19m5s
Dumping 2649 out of 8160 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipl.ko...Reading symbols from /boot/kernel/ipl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipl.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff808ea3a1 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>)
    at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff80bd8240 in trap_fatal (frame=0x9, eva=Variable "eva" is not available.
)
    at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff80bd88d5 in trap (frame=0xffffff834291c620)
    at /usr/src/sys/amd64/amd64/trap.c:599
#5  0xffffffff80bc315f in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:228
#6  0xffffffff8094b595 in m_freem (mb=0xcfa96cc23cf42256)
    at /usr/src/sys/kern/uipc_mbuf.c:160
#7  0xffffffff80a72141 in tcp_do_segment (m=0xfffffe001fb67900, 
    th=0xfffffe001fb55022, so=0xfffffe01a66c92a8, tp=0xfffffe01437993d0, 
    drop_hdrlen=52, tlen=0, iptos=0 '\0', ti_locked=1)
    at /usr/src/sys/netinet/tcp_input.c:1697
#8  0xffffffff80a73313 in tcp_input (m=0xfffffe001fb67900, off0=Variable "off0" is not available.
)
    at /usr/src/sys/netinet/tcp_input.c:1387
#9  0xffffffff80a043dc in ip_input (m=0xfffffe001fb67900)
    at /usr/src/sys/netinet/ip_input.c:760
#10 0xffffffff809adafb in netisr_dispatch_src (proto=1, source=Variable "source" is not available.
)
    at /usr/src/sys/net/netisr.c:1013
#11 0xffffffff809a35cd in ether_demux (ifp=0xfffffe0006109000, 
    m=0xfffffe001fb67900) at /usr/src/sys/net/if_ethersubr.c:940
#12 0xffffffff809a38a4 in ether_nh_input (m=Variable "m" is not available.
)
    at /usr/src/sys/net/if_ethersubr.c:759
#13 0xffffffff809adafb in netisr_dispatch_src (proto=9, source=Variable "source" is not available.
)
    at /usr/src/sys/net/netisr.c:1013
#14 0xffffffff804ad8ca in em_rxeof (rxr=0xfffffe0006111400, count=99, 
    done=0x0) at /usr/src/sys/dev/e1000/if_em.c:4502
#15 0xffffffff804add48 in em_handle_que (context=Variable "context" is not available.
)
    at /usr/src/sys/dev/e1000/if_em.c:1509
#16 0xffffffff8092cf55 in taskqueue_run_locked (queue=0xfffffe0006115880)
    at /usr/src/sys/kern/subr_taskqueue.c:308
#17 0xffffffff8092ded6 in taskqueue_thread_loop (arg=Variable "arg" is not available.
)
    at /usr/src/sys/kern/subr_taskqueue.c:497
#18 0xffffffff808bb9ef in fork_exit (
    callout=0xffffffff8092de90 <taskqueue_thread_loop>, 
    arg=0xffffff8000ac9740, frame=0xffffff834291cc40)
    at /usr/src/sys/kern/kern_fork.c:992
#19 0xffffffff80bc368e in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:602
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000004 in ?? ()
#45 0xffffffff81242c00 in tdq_cpu ()
#46 0xfffffe0003c4f000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0xffffff834291caf0 in ?? ()
#49 0xffffff834291ca98 in ?? ()
#50 0xfffffe0003a7b000 in ?? ()
#51 0xffffffff8091352e in sched_switch (td=0x0, newtd=0xffffff8000ac9740, 
    flags=Variable "flags" is not available.
) at /usr/src/sys/kern/sched_ule.c:1921
Previous frame inner to this frame (corrupt stack?)
(kgdb) 

>How-To-Repeat:
Unknown. This system is typically under a moderate load of web, mail, and DNS traffic.
>Fix:


>Release-Note:
>Audit-Trail:

From: Nate Denning <nate.denning@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/178116: Kernel panic: general protection fault in tcp_do_segment
Date: Thu, 25 Apr 2013 08:42:36 -0600

 --047d7b414c2ef2ac2004db306d84
 Content-Type: text/plain; charset=ISO-8859-1
 
 Please reassign this to freebsd-net (I couldn't start there from the web
 based send-pr form).
 
 --047d7b414c2ef2ac2004db306d84--
Responsible-Changed-From-To: freebsd-amd64->freebsd-net 
Responsible-Changed-By: pluknet 
Responsible-Changed-When: Thu Apr 25 15:34:45 UTC 2013 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=178116 

From: Nate Denning <nate.denning@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/178116: [tcp] [panic] Kernel panic: general protection fault
 in tcp_do_segment
Date: Mon, 29 Apr 2013 23:46:58 -0600

 --089e013a16ec95f6f204db8d8750
 Content-Type: text/plain; charset=ISO-8859-1
 
 The em panics continue. I've received them from both em0 and em1. Below is
 a new variation not related to tcp_do_segment. Let me know if I can provide
 more info.
 
 Thanks,
 
 Nate
 
 --- core.txt ---
 
 Fatal trap 9: general protection fault while in kernel mode
 cpuid = 4; apic id = 32
 instruction pointer     = 0x20:0xffffffff8097dc20
 stack pointer           = 0x28:0xffffff83428cd9b0
 frame pointer           = 0x28:0xffffff83428cd9c0
 code segment            = base 0x0, limit 0xfffff, type 0x1b
                         = DPL 0, pres 1, long 1, def32 0, gran 1
 processor eflags        = interrupt enabled, resume, IOPL = 0
 current process         = 0 (em0 que)
 trap number             = 9
 panic: general protection fault
 cpuid = 4
 KDB: stack backtrace:
 #0 0xffffffff80952f46 at kdb_backtrace+0x66
 #1 0xffffffff8091a2de at panic+0x1ce
 #2 0xffffffff80ca8b80 at trap_fatal+0x290
 #3 0xffffffff80ca9391 at trap+0x241
 #4 0xffffffff80c92813 at calltrap+0x8
 #5 0xffffffff809cf257 at bpf_mtap+0x37
 #6 0xffffffff809d9549 at ether_nh_input+0x259
 #7 0xffffffff809e2578 at netisr_dispatch_src+0x218
 #8 0xffffffff804dd248 at em_rxeof+0x1c8
 #9 0xffffffff804dd6f8 at em_handle_que+0x48
 #10 0xffffffff8095f9b4 at taskqueue_run_locked+0x74
 #11 0xffffffff80960966 at taskqueue_thread_loop+0x46
 #12 0xffffffff808e83af at fork_exit+0x11f
 #13 0xffffffff80c92d3e at fork_trampoline+0xe
 Uptime: 17h44m42s
 Dumping 2486 out of 8158
 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
 
 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
 /boot/kernel/zfs.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/zfs.ko
 Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
 /boot/kernel/opensolaris.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/opensolaris.ko
 Reading symbols from /boot/kernel/ipl.ko...Reading symbols from
 /boot/kernel/ipl.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipl.ko
 Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from
 /boot/kernel/accf_http.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/accf_http.ko
 Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from
 /boot/kernel/accf_data.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/accf_data.ko
 #0  doadump (textdump=<value optimized out>) at pcpu.h:229
 229     pcpu.h: No such file or directory.
         in pcpu.h
 (kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:229
 #1  0xffffffff80919db4 in kern_reboot (howto=260)
     at /usr/src-9-stable/sys/kern/kern_shutdown.c:449
 #2  0xffffffff8091a2b7 in panic (fmt=0x1 <Address 0x1 out of bounds>)
     at /usr/src-9-stable/sys/kern/kern_shutdown.c:637
 #3  0xffffffff80ca8b80 in trap_fatal (frame=0x9, eva=<value optimized out>)
     at /usr/src-9-stable/sys/amd64/amd64/trap.c:878
 #4  0xffffffff80ca9391 in trap (frame=0xffffff83428cd900)
     at /usr/src-9-stable/sys/amd64/amd64/trap.c:605
 #5  0xffffffff80c92813 in calltrap ()
     at /usr/src-9-stable/sys/amd64/amd64/exception.S:228
 #6  0xffffffff8097dc20 in m_length (m0=0x3b4d5ae18672812f, last=0x0)
     at /usr/src-9-stable/sys/kern/uipc_mbuf.c:1459
 #7  0xffffffff809cf257 in bpf_mtap (bp=0xfffffe000605ec00,
     m=0xfffffe0036056600) at /usr/src-9-stable/sys/net/bpf.c:2110
 #8  0xffffffff809d9549 in ether_nh_input (m=<value optimized out>)
     at /usr/src-9-stable/sys/net/if_ethersubr.c:636
 #9  0xffffffff809e2578 in netisr_dispatch_src (proto=9,
     source=<value optimized out>, m=<value optimized out>)
     at /usr/src-9-stable/sys/net/netisr.c:1013
 #10 0xffffffff804dd248 in em_rxeof (rxr=0xfffffe0006056a00, count=98,
     done=0x0) at /usr/src-9-stable/sys/dev/e1000/if_em.c:4515
 #11 0xffffffff804dd6f8 in em_handle_que (context=<value optimized out>,
     pending=<value optimized out>)
     at /usr/src-9-stable/sys/dev/e1000/if_em.c:1518
 #12 0xffffffff8095f9b4 in taskqueue_run_locked (queue=0xfffffe000605fb00)
     at /usr/src-9-stable/sys/kern/subr_taskqueue.c:312
 #13 0xffffffff80960966 in taskqueue_thread_loop (arg=<value optimized out>)
     at /usr/src-9-stable/sys/kern/subr_taskqueue.c:501
 #14 0xffffffff808e83af in fork_exit (
     callout=0xffffffff80960920 <taskqueue_thread_loop>,
     arg=0xffffff8000aae730, frame=0xffffff83428cdc40)
     at /usr/src-9-stable/sys/kern/kern_fork.c:988
 #15 0xffffffff80c92d3e in fork_trampoline ()
     at /usr/src-9-stable/sys/amd64/amd64/exception.S:602
 #16 0x0000000000000000 in ?? ()
 (kgdb)
 
 --089e013a16ec95f6f204db8d8750
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 <div dir=3D"ltr">The em panics continue. I&#39;ve received them from both e=
 m0 and em1. Below is a new variation not related to tcp_do_segment. Let me =
 know if I can provide more info.<div><br></div><div>Thanks,</div><div><br>
 </div><div>Nate<br><div><br></div><div>--- core.txt ---<br><div><br></div><=
 div><div>Fatal trap 9: general protection fault while in kernel mode</div><=
 div>cpuid =3D 4; apic id =3D 32</div><div>instruction pointer =A0 =A0 =3D 0=
 x20:0xffffffff8097dc20</div>
 <div>stack pointer =A0 =A0 =A0 =A0 =A0 =3D 0x28:0xffffff83428cd9b0</div><di=
 v>frame pointer =A0 =A0 =A0 =A0 =A0 =3D 0x28:0xffffff83428cd9c0</div><div>c=
 ode segment =A0 =A0 =A0 =A0 =A0 =A0=3D base 0x0, limit 0xfffff, type 0x1b</=
 div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =3D DPL 0, pres 1,=
  long 1, def32 0, gran 1</div>
 <div>processor eflags =A0 =A0 =A0 =A0=3D interrupt enabled, resume, IOPL =
 =3D 0</div><div>current process =A0 =A0 =A0 =A0 =3D 0 (em0 que)</div><div>t=
 rap number =A0 =A0 =A0 =A0 =A0 =A0 =3D 9</div><div>panic: general protectio=
 n fault</div><div>cpuid =3D 4</div><div>
 KDB: stack backtrace:</div><div>#0 0xffffffff80952f46 at kdb_backtrace+0x66=
 </div><div>#1 0xffffffff8091a2de at panic+0x1ce</div><div>#2 0xffffffff80ca=
 8b80 at trap_fatal+0x290</div><div>#3 0xffffffff80ca9391 at trap+0x241</div=
 >
 <div>#4 0xffffffff80c92813 at calltrap+0x8</div><div>#5 0xffffffff809cf257 =
 at bpf_mtap+0x37</div><div>#6 0xffffffff809d9549 at ether_nh_input+0x259</d=
 iv><div>#7 0xffffffff809e2578 at netisr_dispatch_src+0x218</div><div>#8 0xf=
 fffffff804dd248 at em_rxeof+0x1c8</div>
 <div>#9 0xffffffff804dd6f8 at em_handle_que+0x48</div><div>#10 0xffffffff80=
 95f9b4 at taskqueue_run_locked+0x74</div><div>#11 0xffffffff80960966 at tas=
 kqueue_thread_loop+0x46</div><div>#12 0xffffffff808e83af at fork_exit+0x11f=
 </div>
 <div>#13 0xffffffff80c92d3e at fork_trampoline+0xe</div><div>Uptime: 17h44m=
 42s</div><div>Dumping 2486 out of 8158 MB:..1%..11%..21%..31%..41%..51%..61=
 %..71%..81%..91%</div><div><br></div><div>Reading symbols from /boot/kernel=
 /zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.</div>
 <div>done.</div><div>Loaded symbols for /boot/kernel/zfs.ko</div><div>Readi=
 ng symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/ke=
 rnel/opensolaris.ko.symbols...done.</div><div>done.</div><div>Loaded symbol=
 s for /boot/kernel/opensolaris.ko</div>
 <div>Reading symbols from /boot/kernel/ipl.ko...Reading symbols from /boot/=
 kernel/ipl.ko.symbols...done.</div><div>done.</div><div>Loaded symbols for =
 /boot/kernel/ipl.ko</div><div>Reading symbols from /boot/kernel/accf_http.k=
 o...Reading symbols from /boot/kernel/accf_http.ko.symbols...done.</div>
 <div>done.</div><div>Loaded symbols for /boot/kernel/accf_http.ko</div><div=
 >Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boo=
 t/kernel/accf_data.ko.symbols...done.</div><div>done.</div><div>Loaded symb=
 ols for /boot/kernel/accf_data.ko</div>
 <div>#0 =A0doadump (textdump=3D&lt;value optimized out&gt;) at pcpu.h:229</=
 div><div>229 =A0 =A0 pcpu.h: No such file or directory.</div><div>=A0 =A0 =
 =A0 =A0 in pcpu.h</div><div><div>(kgdb) #0 =A0doadump (textdump=3D&lt;value=
  optimized out&gt;) at pcpu.h:229</div>
 <div>#1 =A00xffffffff80919db4 in kern_reboot (howto=3D260)</div><div>=A0 =
 =A0 at /usr/src-9-stable/sys/kern/kern_shutdown.c:449</div><div>#2 =A00xfff=
 fffff8091a2b7 in panic (fmt=3D0x1 &lt;Address 0x1 out of bounds&gt;)</div><=
 div>=A0 =A0 at /usr/src-9-stable/sys/kern/kern_shutdown.c:637</div>
 <div>#3 =A00xffffffff80ca8b80 in trap_fatal (frame=3D0x9, eva=3D&lt;value o=
 ptimized out&gt;)</div><div>=A0 =A0 at /usr/src-9-stable/sys/amd64/amd64/tr=
 ap.c:878</div><div>#4 =A00xffffffff80ca9391 in trap (frame=3D0xffffff83428c=
 d900)</div>
 <div>=A0 =A0 at /usr/src-9-stable/sys/amd64/amd64/trap.c:605</div><div>#5 =
 =A00xffffffff80c92813 in calltrap ()</div><div>=A0 =A0 at /usr/src-9-stable=
 /sys/amd64/amd64/exception.S:228</div><div>#6 =A00xffffffff8097dc20 in m_le=
 ngth (m0=3D0x3b4d5ae18672812f, last=3D0x0)</div>
 <div>=A0 =A0 at /usr/src-9-stable/sys/kern/uipc_mbuf.c:1459</div><div>#7 =
 =A00xffffffff809cf257 in bpf_mtap (bp=3D0xfffffe000605ec00,=A0</div><div>=
 =A0 =A0 m=3D0xfffffe0036056600) at /usr/src-9-stable/sys/net/bpf.c:2110</di=
 v><div>#8 =A00xffffffff809d9549 in ether_nh_input (m=3D&lt;value optimized =
 out&gt;)</div>
 <div>=A0 =A0 at /usr/src-9-stable/sys/net/if_ethersubr.c:636</div><div>#9 =
 =A00xffffffff809e2578 in netisr_dispatch_src (proto=3D9,=A0</div><div>=A0 =
 =A0 source=3D&lt;value optimized out&gt;, m=3D&lt;value optimized out&gt;)<=
 /div><div>=A0 =A0 at /usr/src-9-stable/sys/net/netisr.c:1013</div>
 <div>#10 0xffffffff804dd248 in em_rxeof (rxr=3D0xfffffe0006056a00, count=3D=
 98,=A0</div><div>=A0 =A0 done=3D0x0) at /usr/src-9-stable/sys/dev/e1000/if_=
 em.c:4515</div><div>#11 0xffffffff804dd6f8 in em_handle_que (context=3D&lt;=
 value optimized out&gt;,=A0</div>
 <div>=A0 =A0 pending=3D&lt;value optimized out&gt;)</div><div>=A0 =A0 at /u=
 sr/src-9-stable/sys/dev/e1000/if_em.c:1518</div><div>#12 0xffffffff8095f9b4=
  in taskqueue_run_locked (queue=3D0xfffffe000605fb00)</div><div>=A0 =A0 at =
 /usr/src-9-stable/sys/kern/subr_taskqueue.c:312</div>
 <div>#13 0xffffffff80960966 in taskqueue_thread_loop (arg=3D&lt;value optim=
 ized out&gt;)</div><div>=A0 =A0 at /usr/src-9-stable/sys/kern/subr_taskqueu=
 e.c:501</div><div>#14 0xffffffff808e83af in fork_exit (</div><div>=A0 =A0 c=
 allout=3D0xffffffff80960920 &lt;taskqueue_thread_loop&gt;,=A0</div>
 <div>=A0 =A0 arg=3D0xffffff8000aae730, frame=3D0xffffff83428cdc40)</div><di=
 v>=A0 =A0 at /usr/src-9-stable/sys/kern/kern_fork.c:988</div><div>#15 0xfff=
 fffff80c92d3e in fork_trampoline ()</div><div>=A0 =A0 at /usr/src-9-stable/=
 sys/amd64/amd64/exception.S:602</div>
 <div>#16 0x0000000000000000 in ?? ()</div><div>(kgdb)=A0</div></div></div><=
 div><br></div></div></div></div>
 
 --089e013a16ec95f6f204db8d8750--

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Nate Denning <nate.denning@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/178116: [tcp] [panic] Kernel panic: general protection
 fault in tcp_do_segment
Date: Wed, 1 May 2013 18:02:06 +0400

   Nate,
 
   do you run any additional network modules: ipfw, pf, netgraph,
 accept filters, etc? How your system differes from a default
 installation?
 
   Is it possible for you to run with INVARIANTS option in the kernel?
 The option adds additional debugging, thus hurts system performance,
 but with it we can obtain a more informative crashdump.
 
 -- 
 Totus tuus, Glebius.

From: Nate Denning <nate.denning@gmail.com>
To: Gleb Smirnoff <glebius@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/178116: [tcp] [panic] Kernel panic: general protection fault in tcp_do_segment
Date: Wed, 1 May 2013 09:26:04 -0600

 On May 1, 2013, at 8:02 AM, Gleb Smirnoff <glebius@FreeBSD.org> wrote:
 
 >  Nate,
 >=20
 >  do you run any additional network modules: ipfw, pf, netgraph,
 > accept filters, etc? How your system differes from a default
 > installation?
 >=20
 
 Yes, ipfilter, accf_http and accf_data (accf is for Apache). No ipfw, =
 pf, or netgraph. Output of kldstat:
 
 Id Refs Address            Size     Name
  1   15 0xffffffff80200000 1558e18  kernel
  2    1 0xffffffff81759000 2324e0   zfs.ko
  3    2 0xffffffff8198c000 84e8     opensolaris.ko
  4    1 0xffffffff81a12000 330db    ipl.ko
  5    1 0xffffffff81a46000 163a     accf_http.ko
  6    1 0xffffffff81a48000 cda      accf_data.ko
 
 IPv4 is configured natively and IPv6 over a gif tunnel, with ipfilter =
 rules setup for both. Other than all that I'm not seeing anything =
 related to networking that is not default.
 
 >  Is it possible for you to run with INVARIANTS option in the kernel?
 > The option adds additional debugging, thus hurts system performance,
 > but with it we can obtain a more informative crashdump.
 >=20
 
 Yes, I can try that.
 
 Thanks,
 
 Nate
 

From: Gleb Smirnoff <glebius@FreeBSD.org>
To: Nate Denning <nate.denning@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/178116: [tcp] [panic] Kernel panic: general protection
 fault in tcp_do_segment
Date: Wed, 1 May 2013 19:27:22 +0400

   Nate,
 
 On Wed, May 01, 2013 at 09:26:04AM -0600, Nate Denning wrote:
 N> >  do you run any additional network modules: ipfw, pf, netgraph,
 N> > accept filters, etc? How your system differes from a default
 N> > installation?
 N> 
 N> Yes, ipfilter, accf_http and accf_data (accf is for Apache). No ipfw, pf, or netgraph. Output of kldstat:
 
 I would suspect ipfilter. :(
 
 Is it possible for you to rewrite your rules to ipfw or pf and try
 running with that?
 
 -- 
 Totus tuus, Glebius.

From: Nate Denning <nate.denning@gmail.com>
To: Gleb Smirnoff <glebius@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/178116: [tcp] [panic] Kernel panic: general protection fault in tcp_do_segment
Date: Wed, 1 May 2013 09:32:08 -0600

 On May 1, 2013, at 9:27 AM, Gleb Smirnoff <glebius@FreeBSD.org> wrote:
 
 >  Nate,
 >=20
 > On Wed, May 01, 2013 at 09:26:04AM -0600, Nate Denning wrote:
 > N> >  do you run any additional network modules: ipfw, pf, netgraph,
 > N> > accept filters, etc? How your system differes from a default
 > N> > installation?
 > N>=20
 > N> Yes, ipfilter, accf_http and accf_data (accf is for Apache). No =
 ipfw, pf, or netgraph. Output of kldstat:
 >=20
 > I would suspect ipfilter. :(
 >=20
 > Is it possible for you to rewrite your rules to ipfw or pf and try
 > running with that?
 >=20
 
 Certainly, I'll switch to pf and see how that goes.
 
 Thanks,
 
 Nate
 

From: Nate Denning <nate.denning@gmail.com>
To: Gleb Smirnoff <glebius@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/178116: [tcp] [panic] Kernel panic: general protection fault in tcp_do_segment
Date: Wed, 8 May 2013 08:30:01 -0600

 On May 1, 2013, at 9:32 AM, Nate Denning <nate.denning@gmail.com> wrote:
 
 >=20
 > On May 1, 2013, at 9:27 AM, Gleb Smirnoff <glebius@FreeBSD.org> wrote:
 >=20
 >> Nate,
 >>=20
 >> On Wed, May 01, 2013 at 09:26:04AM -0600, Nate Denning wrote:
 >> N> >  do you run any additional network modules: ipfw, pf, netgraph,
 >> N> > accept filters, etc? How your system differes from a default
 >> N> > installation?
 >> N>=20
 >> N> Yes, ipfilter, accf_http and accf_data (accf is for Apache). No =
 ipfw, pf, or netgraph. Output of kldstat:
 >>=20
 >> I would suspect ipfilter. :(
 >>=20
 >> Is it possible for you to rewrite your rules to ipfw or pf and try
 >> running with that?
 >>=20
 >=20
 > Certainly, I'll switch to pf and see how that goes.
 
 I switched to pf and I'm at about a week now with no panics where there =
 were typically several per day with ipfilter. I need this host to be =
 stable so I would like to stick to pf, but is there any more info, =
 configs, etc. I can provide to help debug the ipfilter issue?
 
 Thanks,
 
 Nate=
Responsible-Changed-From-To: freebsd-net->cy 
Responsible-Changed-By: cy 
Responsible-Changed-When: Wed Jul 3 05:24:54 UTC 2013 
Responsible-Changed-Why:  
Assign to myself. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=178116 
>Unformatted:
