From nobody@FreeBSD.org  Mon Mar 11 06:40:52 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	by hub.freebsd.org (Postfix) with ESMTP id D486A323
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 11 Mar 2013 06:40:52 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id BC8298F2
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 11 Mar 2013 06:40:52 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r2B6eqcl034853
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 11 Mar 2013 06:40:52 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r2B6eqNA034852;
	Mon, 11 Mar 2013 06:40:52 GMT
	(envelope-from nobody)
Message-Id: <201303110640.r2B6eqNA034852@red.freebsd.org>
Date: Mon, 11 Mar 2013 06:40:52 GMT
From: Martin Bishop <martin.bishop@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Fatal trap 12: page fault while in kernel mode
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         176835
>Category:       amd64
>Synopsis:       Fatal trap 12: page fault while in kernel mode
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Mar 11 06:50:00 UTC 2013
>Closed-Date:    
>Last-Modified:  Mon Mar 18 05:40:00 UTC 2013
>Originator:     Martin Bishop
>Release:        9.1
>Organization:
>Environment:
FreeBSD host.domain 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec  4 09:23:10 UTC 2012     root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Periodic panic, though it looks like it's happening in the same place. Stealing some advice from another bug report, there's some gdb output at the bottom..

panic #1
Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0xffffffffd398481f
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff808eb0b3
stack pointer           = 0x28:0xffffff80913d4970
frame pointer           = 0x28:0xffffff80913d4980
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 91049 (maildrop)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff808efb14 at tdsendsignal+0x454
#7 0xffffffff808f0b6d at killpg1+0x3bd
#8 0xffffffff808f0de4 at sys_kill+0x1e4
#9 0xffffffff80bd7ae6 at amd64_syscall+0x546

panic #2

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address   = 0xffffffffd398481f
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff808eb0b3
stack pointer           = 0x28:0xffffff8091109970
frame pointer           = 0x28:0xffffff8091109980
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 57769 (maildrop)
trap number             = 12
panic: page fault
cpuid = 2
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80bd8240 at trap_fatal+0x290
#3 0xffffffff80bd857d at trap_pfault+0x1ed
#4 0xffffffff80bd8b9e at trap+0x3ce
#5 0xffffffff80bc315f at calltrap+0x8
#6 0xffffffff808efb14 at tdsendsignal+0x454
#7 0xffffffff808f0b6d at killpg1+0x3bd
#8 0xffffffff808f0de4 at sys_kill+0x1e4
#9 0xffffffff80bd7ae6 at amd64_syscall+0x546

% gdb /boot/kernel/kernel
<snip>
(gdb) l *tdsendsignal+0x454
0xffffffff808efb14 is in tdsendsignal (/usr/src/sys/kern/kern_sig.c:2046).
warning: Source file is more recent than executable.

2041             * IEEE Std 1003.1-2001: return success when killing a zombie.
2042             */
2043            if (p->p_state == PRS_ZOMBIE) {
2044                    if (ksi && (ksi->ksi_flags & KSI_INS))
2045                            ksiginfo_tryfree(ksi);
2046                    return (ret);
2047            }
2048
2049            ps = p->p_sigacts;
2050            KNOTE_LOCKED(&p->p_klist, NOTE_SIGNAL | sig);
(gdb) 

I can grab a core if need be, but am hoping that given the consistency of the backtrace the problem will be obvious to the maintainers of the relevant portions of code :)
>How-To-Repeat:
I haven't been triggering it, but given the current process is consistently the same:

- run 9.1 release
- use maildrop to filter mail
- wait for crash

>Fix:


>Release-Note:
>Audit-Trail:

From: Konstantin Belousov <kostikbel@gmail.com>
To: Martin Bishop <martin.bishop@gmail.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: amd64/176835: Fatal trap 12: page fault while in kernel mode
Date: Mon, 11 Mar 2013 14:01:52 +0200

 --YyN8ymQDUlI+Fj6t
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On Mon, Mar 11, 2013 at 06:40:52AM +0000, Martin Bishop wrote:
 > I can grab a core if need be, but am hoping that given the consistency of the backtrace the problem will be obvious to the maintainers of the relevant portions of code :)
 
 You do need to grab core, and use the sources corresponding to your kernel.
 The data you demonstrated do not tell much.
 
 If the faulting address was an attempt to access any non-trivial structure,
 then it is very suspicious, both because the faulting addresses are same
 for both panics, and because the address is not aligned.
 
 --YyN8ymQDUlI+Fj6t
 Content-Type: application/pgp-signature
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iQIcBAEBAgAGBQJRPcewAAoJEJDCuSvBvK1ByoUP+QGmNBZDeMwa+pEzb5NxbJsH
 N3A78kAODXhilw0+RqxVCNrdoBVB7rZi10YAkuFSZY7sIY1sqNdXBEs+m6CUbng8
 e4U3ON9XxupKlh0u1Tifln5SJD2Bc0IkZwuED1m6QqlNDP8P3C+wB74QlkZOYlbC
 hIO7bmDLTFS/fSDI5ZpG2ReQhkvxO5sITQ8eXO5PVeJcxXLRePu5Li/pdHSumzHf
 dOLNZAbsv4DE49K8iUtQW22iqC1/gXXlgvoq6aKkzXRF/efqqHYzkx8GkDFWgP8g
 UECcyO/xQUxZnLIMbWIgpccHLLHc+zJ0vodvdnj/DbC0/kVWX5I3VhKJaWnPKfSA
 X08j64nEjlmdzklVPkAl8OqejICkW8ni9eIanbuO64B+eC/+nanNdV5Drm890Lit
 txpQW2d+RYsAr/Yhc+qczvc9Zp09pHL1D6r7l4BTwS72LQRpfdd4KP7Tr70peq4T
 6ZvPAD4VhM3SqJkhfSslwZAq+xfMnPIF8FZym9aPX1M+PmQVtxY5GV69CmGpmdYI
 jfpiIIsKrLolu+jkeXGaVDKG8Puhr2K2p7sZ2SL9rcNeVNGl/W6IlKkx9pglY5k+
 ITHY76s8q999r3nE/vwSYQX+Bmn4e0mMOFsyb91+IQjUJl+8e7DlW+pXA7iwLgw8
 LLwmyBnsOYMh/gLtftvE
 =JfPQ
 -----END PGP SIGNATURE-----
 
 --YyN8ymQDUlI+Fj6t--

From: Martin Bishop <martin.bishop@gmail.com>
To: bug-followup@FreeBSD.org, martin.bishop@gmail.com
Cc:  
Subject: Re: amd64/176835: Fatal trap 12: page fault while in kernel mode
Date: Sun, 17 Mar 2013 22:33:50 -0700

 After one more crash to enable dumping, and one to produce a core.
 
 Backtrace from kgdb
 
 (kgdb) bt
 #0  doadump (textdump=Variable "textdump" is not available.
 ) at pcpu.h:224
 #1  0xffffffff808ea3a1 in kern_reboot (howto=260) at
 /usr/src/sys/kern/kern_shutdown.c:448
 #2  0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>)
 at /usr/src/sys/kern/kern_shutdown.c:636
 #3  0xffffffff80bd8240 in trap_fatal (frame=0xc, eva=Variable "eva" is
 not available.
 ) at /usr/src/sys/amd64/amd64/trap.c:857
 #4  0xffffffff80bd857d in trap_pfault (frame=0xffffff80913118c0,
 usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773
 #5  0xffffffff80bd8b9e in trap (frame=0xffffff80913118c0) at
 /usr/src/sys/amd64/amd64/trap.c:456
 #6  0xffffffff80bc315f in calltrap () at
 /usr/src/sys/amd64/amd64/exception.S:228
 #7  0xffffffff808eb0b3 in sigtd (p=0xfffffe00212d3000, sig=1,
 prop=Variable "prop" is not available.
 ) at /usr/src/sys/kern/kern_sig.c:1927
 #8  0xffffffff808efb14 in tdsendsignal (p=0xfffffe00212d3000, td=0x0,
 sig=1, ksi=0xffffff8091311a70) at /usr/src/sys/kern/kern_sig.c:2045
 #9  0xffffffff808f0b6d in killpg1 (td=0xfffffe00341278e0, sig=1,
 pgid=Variable "pgid" is not available.
 ) at /usr/src/sys/kern/kern_sig.c:1650
 #10 0xffffffff808f0de4 in sys_kill (td=0xfffffe00341278e0,
 uap=0xffffff8091311bb0) at /usr/src/sys/kern/kern_sig.c:1703
 #11 0xffffffff80bd7ae6 in amd64_syscall (td=0xfffffe00341278e0,
 traced=0) at subr_syscall.c:135
 #12 0xffffffff80bc3447 in Xfast_syscall () at
 /usr/src/sys/amd64/amd64/exception.S:387
 #13 0x00000008015dbc2c in ?? ()
 
 Any particular information to grab to make this easier to track down ?
>Unformatted:
