From scf@farley.org  Sun Mar  2 01:02:46 2008
Return-Path: <scf@farley.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 95D021065672
	for <FreeBSD-gnats-submit@freebsd.org>; Sun,  2 Mar 2008 01:02:46 +0000 (UTC)
	(envelope-from scf@farley.org)
Received: from mail.farley.org (farley.org [67.64.95.201])
	by mx1.freebsd.org (Postfix) with ESMTP id 4555E8FC15
	for <FreeBSD-gnats-submit@freebsd.org>; Sun,  2 Mar 2008 01:02:46 +0000 (UTC)
	(envelope-from scf@farley.org)
Received: from thor.farley.org (thor.farley.org [192.168.1.5])
	by mail.farley.org (8.14.2/8.14.2) with ESMTP id m220fEBl004961
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 1 Mar 2008 18:41:14 -0600 (CST)
	(envelope-from scf@mail.farley.org)
Received: from thor.farley.org (localhost [127.0.0.1])
	by thor.farley.org (8.14.2/8.14.2) with ESMTP id m220fEwi026036
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 1 Mar 2008 18:41:14 -0600 (CST)
	(envelope-from scf@thor.farley.org)
Received: (from sean@localhost)
	by thor.farley.org (8.14.2/8.14.2/Submit) id m220fEx7026035;
	Sat, 1 Mar 2008 18:41:14 -0600 (CST)
	(envelope-from scf)
Message-Id: <200803020041.m220fEx7026035@thor.farley.org>
Date: Sat, 1 Mar 2008 18:41:14 -0600 (CST)
From: Sean Farley <scf@freebsd.org>
Reply-To: Sean Farley <scf@freebsd.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Panic in ether_input() with different NIC's.
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         121274
>Category:       kern
>Synopsis:       [ipfilter] [gif] Panic in ether_input() with different NIC's.
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    cy
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 02 01:10:01 UTC 2008
>Closed-Date:    
>Last-Modified:  Thu Jul  4 17:50:01 UTC 2013
>Originator:     Sean Farley
>Release:        FreeBSD 7.0-STABLE i386
>Organization:
>Environment:
System: FreeBSD thor.farley.org 7.0-STABLE FreeBSD 7.0-STABLE #0: Thu Feb 28 19:10:50 CST 2008 sean@thor.farley.org:/usr/FreeBSD/RELENG_7/obj/usr/FreeBSD/RELENG_7/src/sys/THOR i386

        Athlon XP 2100, Asus A7V880, Netgear FA311-TX (sis driver)

>Description:
I have received two panics at ether_input():545 (frame #7).  One panic was
concerning the sis(4), and the other with sk(4).  This hardware has worked
well in the past as a desktop.  I recently configured it to be my gateway
server replacing an older system.

The setup is basically the same as the old system with the jails, ipfilter
and ipnat as they were before.

Panic message:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x3000c
fault code              = supervisor read, page not present
instruction pointer     = 0x20:0xc05f4825
stack pointer           = 0x28:0xdda4fc1c
frame pointer           = 0x28:0xdda4fc3c
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, def32 1, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 22 (irq18: skc0)
trap number             = 12
panic: page fault
cpuid = 0
Uptime: 6h58m0s
Physical memory: 758 MB
Dumping 115 MB: 100 84 68 52 36 20 4

The stack trace:
(kgdb) where
#0  doadump () at pcpu.h:195
#1  0xc055d3b7 in boot (howto=260) at /usr/FreeBSD/RELENG_7/src/sys/kern/kern_shutdown.c:409
#2  0xc055d679 in panic (fmt=Variable "fmt" is not available.) at /usr/FreeBSD/RELENG_7/src/sys/kern/kern_shutdown.c:563
#3  0xc0782bec in trap_fatal (frame=0xdda4fbdc, eva=196620) at /usr/FreeBSD/RELENG_7/src/sys/i386/i386/trap.c:899
#4  0xc0782e50 in trap_pfault (frame=0xdda4fbdc, usermode=0, eva=196620) at /usr/FreeBSD/RELENG_7/src/sys/i386/i386/trap.c:812
#5  0xc07837d2 in trap (frame=0xdda4fbdc) at /usr/FreeBSD/RELENG_7/src/sys/i386/i386/trap.c:490
#6  0xc076a14b in calltrap () at /usr/FreeBSD/RELENG_7/src/sys/i386/i386/exception.s:139
#7  0xc05f4825 in ether_input (ifp=0xc371a800, m=0xc42e0200) at /usr/FreeBSD/RELENG_7/src/sys/net/if_ethersubr.c:545
#8  0xc04c2c56 in sk_rxeof (sc_if=0xc371c000) at /usr/FreeBSD/RELENG_7/src/sys/dev/sk/if_sk.c:2917
#9  0xc04c2e18 in sk_intr (xsc=0xc370eb00) at /usr/FreeBSD/RELENG_7/src/sys/dev/sk/if_sk.c:3246
#10 0xc05405eb in ithread_loop (arg=0xc3729b30) at /usr/FreeBSD/RELENG_7/src/sys/kern/kern_intr.c:1036
#11 0xc053d3e9 in fork_exit (callout=0xc0540440 <ithread_loop>, arg=0xc3729b30, frame=0xdda4fd38) at /usr/FreeBSD/RELENG_7/src/sys/kern/kern_fork.c:781
#12 0xc076a1c0 in fork_trampoline () at /usr/FreeBSD/RELENG_7/src/sys/i386/i386/exception.s:205

A few more details can be found here:
http://www.farley.org/freebsd/tmp/panic/

I still have the second core and kernel if more information is needed.

>How-To-Repeat:
In both panics, it looks like it was while the regex milter was processing
an E-mail.

>Fix:
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Mar 2 02:56:55 UTC 2008 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121274 
State-Changed-From-To: open->feedback 
State-Changed-By: rwatson 
State-Changed-When: Sun Mar 2 15:03:57 UTC 2008 
State-Changed-Why:  
Could you run memtest86 or some other memory testing tool on the box? 
While this could well be a software bug, it would be nice to give it a 
whirl and make sure you're not running into, say, a 1-bit memory error 
that might easily explain the panic across various drivers.  Thanks! 


http://www.freebsd.org/cgi/query-pr.cgi?pr=121274 

From: Sean Farley <sean@farley.org>
To: bug-followup@FreeBSD.org, scf@FreeBSD.org
Cc:  
Subject: Re: kern/121274: [panic] Panic in ether_input() with different
 NIC's.
Date: Sun, 2 Mar 2008 15:37:24 -0600 (CST)

 It passed memtest86+ v2.01 scrutiny.

From: Sean Farley <scf@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/121274: [panic] Panic in ether_input() with different
 NIC's.
Date: Sun, 2 Mar 2008 17:51:35 -0600 (CST)

 I am currently running a build with INVARIANTS and WITNESS.  No panics
 so far, but I did get a LOR:
 Mar  2 15:35:22 gw kernel: 1st 0xc39ee8a0 ipf filter load/unload mutex (ipf filter load/unload mutex) @ /usr/FreeBSD/RELENG_7/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/fil.c:2431
 Mar  2 15:35:22 gw kernel: 2nd 0xc3a4a404 gif softc (gif softc) @ /usr/FreeBSD/RELENG_7/src/sys/net/if_gif.c:411

From: "Sean C. Farley" <scf@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/121274: [panic] Panic in ether_input() with different
 NIC's.
Date: Tue, 4 Mar 2008 13:15:49 -0600 (CST)

 Two additional items of interest, but I do not know for certain if this
 is related to the panic I am seeing:
 1. I have recently found that running "ipnat -s" will cause a panic
     regardless of how long the system has been running.
 2. Here is the LOR along with a backtrace from running "ipnat -s".  More
     information can be found here:
     http://www.farley.org/freebsd/tmp/panic/dmesg.boot
 
 
 IP Filter: v4.1.28 initialized.  Default = pass all, Logging = enabled
 Kernel page fault with the following non-sleepable locks held:
 shared rw ipf filter load/unload mutex r = 0 (0xc52088a0) locked @ /usr/FreeBSD/RELENG_7/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_fil_freebsd.c:350
 KDB: enter: witness_warn
 
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address	= 0x28202000
 fault code		= supervisor write, page not present
 instruction pointer	= 0x20:0xc0777556
 stack pointer	        = 0x28:0xdf59cde4
 frame pointer	        = 0x28:0xdf59dbb4
 code segment		= base 0x0, limit 0xfffff, type 0x1b
  			= DPL 0, pres 1, def32 1, gran 1
 processor eflags	= interrupt enabled, resume, IOPL = 0
 current process		= 62 (ipnat)
 lock order reversal: (sleepable after non-sleepable)
   1st 0xc52088a0 ipf filter load/unload mutex (ipf filter load/unload mutex) @ /usr/FreeBSD/RELENG_7/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_fil_freebsd.c:350
   2nd 0xc4f0b12c user map (user map) @ /usr/FreeBSD/RELENG_7/src/sys/vm/vm_map.c:3111
 KDB: stack backtrace:
 db_trace_self_wrapper(c07ba661,df59cb00,c059895e,c07bcc04,c4f0b12c,...) at db_trace_self_wrapper+0x26
 kdb_backtrace(c07bcc04,c4f0b12c,c07d5a0c,c07d5a0c,c07d5990,...) at kdb_backtrace+0x29
 witness_checkorder(c4f0b12c,9,c07d5990,c27,c0461b23,...) at witness_checkorder+0x6de
 _sx_xlock(c4f0b12c,0,c07d5990,c27,df59cb68,...) at _sx_xlock+0x7d
 _vm_map_lock_read(c4f0b0e8,c07d5990,c27,0,0,...) at _vm_map_lock_read+0x50
 vm_map_lookup(df59cc60,28202000,2,df59cc64,df59cc54,...) at vm_map_lookup+0x38
 vm_fault(c4f0b0e8,28202000,2,8,28202000,...) at vm_fault+0x83
 trap_pfault(5,0,c07df7db,0,c,...) at trap_pfault+0xf9
 trap(df59cda4) at trap+0x3f2
 calltrap() at calltrap+0x6
 --- trap 0xc, eip = 0xc0777556, esp = 0xdf59cde4, ebp = 0xdf59dbb4 ---
 generic_copyout(c51a1480,c034725d,1,0,c5020880,...) at generic_copyout+0x36
 iplioctl(c5191a00,c034725d,c51a1480,1,c5020880,...) at iplioctl+0xca
 devfs_ioctl_f(c51b0120,c034725d,c51a1480,c524f000,c5020880,...) at devfs_ioctl_f+0xc9
 kern_ioctl(c5020880,3,c034725d,c51a1480,1000000,...) at kern_ioctl+0x243
 ioctl(c5020880,df59dcfc,c,c07b487b,c07f8bb0,...) at ioctl+0x134
 syscall(df59dd38) at syscall+0x2b3
 Xint0x80_syscall() at Xint0x80_syscall+0x20
 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x28166363, esp = 0xbfbfeccc, ebp = 0xbfbfed38 ---
 KDB: enter: witness_checkorder
 Kernel page fault with the following non-sleepable locks held:
 shared rw ipf filter load/unload mutex r = 0 (0xc52088a0) locked @ /usr/FreeBSD/RELENG_7/src/sys/modules/ipfilter/../../contrib/ipfilter/netinet/ip_fil_freebsd.c:350
 KDB: enter: witness_warn
 
 
 Fatal trap 12: page fault while in kernel mode
 cpuid = 0; apic id = 00
 fault virtual address	= 0x28203000
 fault code		= supervisor write, page not present
 instruction pointer	= 0x20:0xc0777556
 stack pointer	        = 0x28:0xdf59cde4
 frame pointer	        = 0x28:0xdf59dbb4
 code segment		= base 0x0, limit 0xfffff, type 0x1b
  			= DPL 0, pres 1, def32 1, gran 1
 processor eflags	= interrupt enabled, resume, IOPL = 0
 current process		= 62 (ipnat)
State-Changed-From-To: feedback->open 
State-Changed-By: scf 
State-Changed-When: Sun Mar 16 15:42:57 CDT 2008 
State-Changed-Why:  
As this looks like an issue between ipnat and gif, assign PR to darrenr 
for analysis. 


Responsible-Changed-From-To: freebsd-net->darrenr 
Responsible-Changed-By: scf 
Responsible-Changed-When: Sun Mar 16 15:42:57 CDT 2008 
Responsible-Changed-Why:  
As this looks like an issue between ipnat and gif, assign PR to darrenr 
for analysis. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121274 
State-Changed-From-To: open->open 
State-Changed-By: linimon 
State-Changed-When: Wed Jul 3 00:50:32 UTC 2013 
State-Changed-Why:  
commit bit has been taken in for safekeeping. 


Responsible-Changed-From-To: darrenr->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Jul 3 00:50:32 UTC 2013 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=121274 
Responsible-Changed-From-To: freebsd-net->cy 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Jul 3 01:43:20 UTC 2013 
Responsible-Changed-Why:  
by request. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=121274 

From: Cy Schubert <Cy.Schubert@komquats.com>
To: Sean Farley <scf@freebsd.org>
Cc: bug-followup <bug-followup@freebsd.org>
Subject: Re:PR/121274 Panic in ether_input() with different NIC's.
Date: Tue, 02 Jul 2013 20:19:07 -0700

 Hi Sean,
 
 Just picked up this PR. I notice that this occurred a while ago under 
 FreeBSD 7. Has this improved since then? Can you reproduce this with 
 STABLE/8, STABLE/9, or HEAD? If you can a new backtrace would be muchly 
 appreciated and and a kernel dump would be better (assuming there's no PI 
 in the dump).
 
 
 -- 
 Cheers,
 Cy Schubert <Cy.Schubert@komquats.com>
 FreeBSD UNIX:  <cy@FreeBSD.org>   Web:  http://www.FreeBSD.org
 
 	The need of the many outweighs the greed of the few.
 
 

From: "Sean C. Farley" <scf@FreeBSD.org>
To: Cy Schubert <Cy.Schubert@komquats.com>
Cc: bug-followup <bug-followup@freebsd.org>
Subject: Re:PR/121274 Panic in ether_input() with different NIC's.
Date: Thu, 4 Jul 2013 13:47:53 -0400 (EDT)

 On Tue, 2 Jul 2013, Cy Schubert wrote:
 
 > Hi Sean,
 >
 > Just picked up this PR. I notice that this occurred a while ago under 
 > FreeBSD 7. Has this improved since then? Can you reproduce this with 
 > STABLE/8, STABLE/9, or HEAD? If you can a new backtrace would be 
 > muchly appreciated and and a kernel dump would be better (assuming 
 > there's no PI in the dump).
 
 Hi Cy,
 
 The system that had these issues was retired from service a few years 
 ago.
 
 I still have the system, however, I no longer have the IPF/IPNAT 
 configuration which is what I think was tickling the panic. 
 Additionally, I now see that one cap is bulging and leaking a bit, so it 
 may produce panics that are unrelated to any software bugs.
 
 I think this PR can be closed yet reopened if anyone can reproduce it on 
 non-suspect hardware.  The only part of the PR that would probably not 
 be hardware-related is being able to induce a panic with "ipnat -s". 
 Unfortunately without that configuration, I doubt it can be reproduced.
 
 Sean
 -- 
 scf@FreeBSD.org
>Unformatted:
