From nobody@FreeBSD.org  Tue Oct  1 19:49:06 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id BB3A7160
	for <freebsd-gnats-submit@FreeBSD.org>; Tue,  1 Oct 2013 19:49:06 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from oldred.freebsd.org (oldred.freebsd.org [8.8.178.121])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id A65C8248D
	for <freebsd-gnats-submit@FreeBSD.org>; Tue,  1 Oct 2013 19:49:06 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id r91Jn6Zq007829
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 1 Oct 2013 19:49:06 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id r91Jn6H4007828;
	Tue, 1 Oct 2013 19:49:06 GMT
	(envelope-from nobody)
Message-Id: <201310011949.r91Jn6H4007828@oldred.freebsd.org>
Date: Tue, 1 Oct 2013 19:49:06 GMT
From: Berend de Boer <berend@pobox.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Frequent reboots due to kernel trap happening in pf_test_rule
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         182557
>Category:       kern
>Synopsis:       Frequent reboots due to kernel trap happening in pf_test_rule
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Oct 01 19:50:00 UTC 2013
>Closed-Date:    Wed Jan 01 21:52:08 UTC 2014
>Last-Modified:  Wed Jan 22 10:50:00 UTC 2014
>Originator:     Berend de Boer
>Release:        9.2-RELEASE
>Organization:
Xplain Hosting
>Environment:
FreeBSD bmach.nederware.nl 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255962: Wed Oct  2 07:24:02 NZDT 2013     root@bmach.nederware.nl:/usr/obj/usr/src/9.2/sys/BMACH  amd64

>Description:
Oct  2 08:34:09 bmach syslogd: kernel boot file is /boot/kernel/kernel
Oct  2 08:34:09 bmach kernel: 
Oct  2 08:34:09 bmach kernel: 
Oct  2 08:34:09 bmach kernel: Fatal trap 12: page fault while in kernel mode
Oct  2 08:34:09 bmach kernel: cpuid = 1; apic id = 01
Oct  2 08:34:09 bmach kernel: fault virtual address	= 0x10
Oct  2 08:34:09 bmach kernel: fault code		= supervisor read data, page not present
Oct  2 08:34:09 bmach kernel: instruction pointer	= 0x20:0xffffffff81a1b8b8
Oct  2 08:34:09 bmach kernel: stack pointer	        = 0x28:0xffffff81281f2380
Oct  2 08:34:09 bmach kernel: frame pointer	        = 0x28:0xffffff81281f2390
Oct  2 08:34:09 bmach kernel: code segment		= base 0x0, limit 0xfffff, type 0x1b
Oct  2 08:34:09 bmach kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Oct  2 08:34:09 bmach kernel: processor eflags	= interrupt enabled, resume, IOPL = 0
Oct  2 08:34:09 bmach kernel: current process		= 1843 (named)
Oct  2 08:34:09 bmach kernel: trap number		= 12
Oct  2 08:34:09 bmach kernel: panic: page fault
Oct  2 08:34:09 bmach kernel: cpuid = 1
Oct  2 08:34:09 bmach kernel: KDB: stack backtrace:
Oct  2 08:34:09 bmach kernel: #0 0xffffffff80950086 at kdb_backtrace+0x66
Oct  2 08:34:09 bmach kernel: #1 0xffffffff809161fd at panic+0x1cd
Oct  2 08:34:09 bmach kernel: #2 0xffffffff80d00fb0 at trap_fatal+0x290
Oct  2 08:34:09 bmach kernel: #3 0xffffffff80d01311 at trap_pfault+0x211
Oct  2 08:34:09 bmach kernel: #4 0xffffffff80d018c4 at trap+0x344
Oct  2 08:34:09 bmach kernel: #5 0xffffffff80ceac53 at calltrap+0x8
Oct  2 08:34:09 bmach kernel: #6 0xffffffff81a25e7c at pf_test_rule+0xf3c
Oct  2 08:34:09 bmach kernel: #7 0xffffffff81a29634 at pf_test+0xe34
Oct  2 08:34:09 bmach kernel: #8 0xffffffff81a311c1 at pf_check_out+0x41
Oct  2 08:34:09 bmach kernel: #9 0xffffffff809e249e at pfil_run_hooks+0x9e
Oct  2 08:34:09 bmach kernel: #10 0xffffffff80a46373 at ip_output+0x3a3
Oct  2 08:34:09 bmach kernel: #11 0xffffffff80ac3588 at udp_send+0x508
Oct  2 08:34:09 bmach kernel: #12 0xffffffff80985c4f at sosend_dgram+0x2cf
Oct  2 08:34:09 bmach kernel: #13 0xffffffff80989e93 at kern_sendit+0x1a3
Oct  2 08:34:09 bmach kernel: #14 0xffffffff8098a14c at sendit+0xdc
Oct  2 08:34:09 bmach kernel: #15 0xffffffff8098a1d7 at sys_sendmsg+0x87
Oct  2 08:34:09 bmach kernel: #16 0xffffffff80d0075a at amd64_syscall+0x5ea
Oct  2 08:34:09 bmach kernel: #17 0xffffffff80ceaf37 at Xfast_syscall+0xf7
Oct  2 08:34:09 bmach kernel: Uptime: 46m10s

>How-To-Repeat:
Just wait...

Custom compiled kernel, GENERIC + the following options:


options         MROUTING                # Multicast routing

# ALTQ
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_CDNR
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)

Options don't matter btw, reboot happens also with generic.


Have simplified my pf.conf to this:


set limit { frags 30000, states 30000 }
set optimization conservative

scrub in all

nat on egress proto udp from { lan:network, $vpn_if:network, 192.168.233.21, 192.168.233.25 } port $voip_ports to any -> (egress) static-port
nat on egress from any to any -> (egress) round-robin sticky-address

block log
pass on lan
pass on public_wifi
pass on $vpn_if
set skip on lo0
pass out on egress



But the kind of firewall doesn't seem to be matter.

I also described the problem here with experiments on FreeBSD-9.1: http://www.freebsd.org/cgi/query-pr.cgi?pr=182141

Trap happens on i386 or amd64, so doesn't look like a hardware problem (and hardware has been fine with FreeBSD 8-STABLE).
>Fix:
Have tried everything. Never had an issue like this in the 18 years I've been using FreeBSD.

>Release-Note:
>Audit-Trail:

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: kern/182557: Frequent reboots due to kernel trap happening in
 pf_test_rule
Date: Wed, 09 Oct 2013 11:43:40 +1300

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --m8PnxkPi4fANfrXgfiwC2bHbC45iND1mD
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Happens on 10-ALPHA5 as well:
 
 Oct  9 08:29:08 bmach kernel: Kernel page fault with the following
 non-sleepable locks held:
 Oct  9 08:29:08 bmach kernel: shared rw pf rulesets (pf rulesets) r =3D 0=
 
 (0xffffffff81d03ae8) locked @
 /usr/src/head/sys/modules/pf/../../netpfil/pf/pf.c:5660
 Oct  9 08:29:08 bmach kernel: shared rm PFil shared rmlock (PFil shared
 rmlock) r =3D 0 (0xffffffff8170cd00) locked @ /usr/src/head/sys/net/pfil.=
 c:77
 Oct  9 08:29:08 bmach kernel: shared rw udpinp (udpinp) r =3D 0
 (0xfffff80017affda8) locked @ /usr/src/head/sys/netinet/udp_usrreq.c:981
 Oct  9 08:29:08 bmach kernel: KDB: stack backtrace:
 Oct  9 08:29:08 bmach kernel: db_trace_self_wrapper() at
 db_trace_self_wrapper+0x2b/frame 0xfffffe011e9acb30
 Oct  9 08:29:08 bmach kernel: kdb_backtrace() at
 kdb_backtrace+0x39/frame 0xfffffe011e9acbe0
 Oct  9 08:29:08 bmach kernel: witness_warn() at witness_warn+0x4a8/frame
 0xfffffe011e9acca0
 Oct  9 08:29:08 bmach kernel: trap_pfault() at trap_pfault+0x5a/frame
 0xfffffe011e9acd40
 Oct  9 08:29:08 bmach kernel: trap() at trap+0x670/frame 0xfffffe011e9acf=
 60
 Oct  9 08:29:08 bmach kernel: calltrap() at calltrap+0x8/frame
 0xfffffe011e9acf60
 Oct  9 08:29:08 bmach kernel: --- trap 0xc, rip =3D 0xffffffff81ce1f6e,
 rsp =3D 0xfffffe011e9ad020, rbp =3D 0xfffffe011e9ad4f0 ---
 Oct  9 08:29:08 bmach kernel: pf_test_rule() at
 pf_test_rule+0x248e/frame 0xfffffe011e9ad4f0
 Oct  9 08:29:08 bmach kernel: pf_test() at pf_test+0x9c8/frame
 0xfffffe011e9ad690
 Oct  9 08:29:08 bmach kernel: pf_check_out() at pf_check_out+0x3c/frame
 0xfffffe011e9ad6c0
 Oct  9 08:29:08 bmach kernel: pfil_run_hooks() at
 pfil_run_hooks+0x93/frame 0xfffffe011e9ad750
 Oct  9 08:29:08 bmach kernel: ip_output() at ip_output+0x8c8/frame
 0xfffffe011e9ad840
 Oct  9 08:29:08 bmach kernel: udp_send() at udp_send+0x866/frame
 0xfffffe011e9ad900
 Oct  9 08:29:08 bmach kernel: sosend_dgram() at sosend_dgram+0x36b/frame
 0xfffffe011e9ad980
 Oct  9 08:29:08 bmach kernel: kern_sendit() at kern_sendit+0x207/frame
 0xfffffe011e9ada30
 Oct  9 08:29:08 bmach kernel: sendit() at sendit+0x116/frame
 0xfffffe011e9ada80
 Oct  9 08:29:08 bmach kernel: sys_sendmsg() at sys_sendmsg+0x61/frame
 0xfffffe011e9adae0
 Oct  9 08:29:08 bmach kernel: amd64_syscall() at
 amd64_syscall+0x265/frame 0xfffffe011e9adbf0
 Oct  9 08:29:08 bmach kernel: Xfast_syscall() at
 Xfast_syscall+0xfb/frame 0xfffffe011e9adbf0
 Oct  9 08:29:08 bmach kernel: --- syscall (28, FreeBSD ELF64,
 sys_sendmsg), rip =3D 0x80131769a, rsp =3D 0x7fffffbfd178, rbp =3D
 0x7fffffbfd1a0 ---
 Oct  9 08:29:08 bmach kernel:
 Oct  9 08:29:08 bmach kernel:
 Oct  9 08:29:08 bmach kernel: Fatal trap 12: page fault while in kernel m=
 ode
 Oct  9 08:29:08 bmach kernel: cpuid =3D 1; apic id =3D 01
 Oct  9 08:29:08 bmach kernel: fault virtual address	=3D 0x10
 Oct  9 08:29:08 bmach kernel: fault code		=3D supervisor read data, page
 not present
 Oct  9 08:29:08 bmach kernel: instruction pointer	=3D 0x20:0xffffffff81ce=
 1f6e
 Oct  9 08:29:08 bmach kernel: stack pointer	        =3D
 0x28:0xfffffe011e9ad020
 Oct  9 08:29:08 bmach kernel: frame pointer	        =3D
 0x28:0xfffffe011e9ad4f0
 Oct  9 08:29:11 bmach kernel: code segment		=3D base 0x0, limit 0xfffff,
 type 0x1b
 Oct  9 08:29:11 bmach kernel: =3D DPL 0, pres 1, long 1, def32 0, gran 1
 Oct  9 08:29:11 bmach kernel: processor eflags	=3D interrupt enabled,
 resume, IOPL =3D 0
 Oct  9 08:29:11 bmach kernel: current process		=3D 1082 (named)
 
 
 --m8PnxkPi4fANfrXgfiwC2bHbC45iND1mD
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSVIqcAAoJEKOfeD48G3g5Sc4QALf3p0PkP0MW9Ffmk903TqEB
 V1H/9mffN2jvvg/4Gs7svQnawbXJsG7sMvTKG4+rxYMiWZC4L1cp0AIWwR0A6BHc
 DfSWmLA1OzvmeMN6r1jcUacDpZW59rfRjkuMiwHIf8SmzDS4Pk7RTU1X6h4TWjOY
 7TTx3agIVZk1quHxw5nXUJE/5bIfcPs9YgxSaG53pCM3R593m37iV4828RurQiZm
 XBgiX2ktwABpyp1ja2YkI5XuOIZLM3xUAsdOMVV+RmIn2P1Q2Ctify99xAGWG83a
 3L44uvgkRLDTstKpKReJiPxln4isphbjYJiXVUjovpORlGRnWYi7Taafv+ZXlVAQ
 Q7EAerCboTyfuxyMeDLc7ehpkH8kWerxTJ49DtFRAyJjy3pIdvFT8OCuMdlmO9kJ
 PQpfhXm0TcH4C+18UqSIgYL/9T3RkvT/yPy1BCpD8gIvPQX7lcmg+RdvFNAdm4Wn
 fe7BCnK/IpEqlO/kqY4hFuZ9V24ugpX0I2keFZBTwybn7MP5tNxa9FZkTPF6aKHV
 Z2gev74wAfqylcQPedZUwg0hTM6f3DXZvbmeI+hbGzovC1nhFMRB01GXQ8WTdIcn
 N0kO/WVNSKUcEGFvxVSfGzgDImiLmd0ErUs5yStQevGAFYoj7Pjqh/rPTQmbcEA4
 62SXoQkqYX8xY8BWZxds
 =9vqT
 -----END PGP SIGNATURE-----
 
 --m8PnxkPi4fANfrXgfiwC2bHbC45iND1mD--
State-Changed-From-To: open->closed 
State-Changed-By: glebius 
State-Changed-When: Wed Jan 1 21:51:17 UTC 2014 
State-Changed-Why:  
This is duplicate of 182141, and the latter has much more information in it, 
so preferred to be left open. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=182557 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/182557: commit references a PR
Date: Mon,  6 Jan 2014 19:05:12 +0000 (UTC)

 Author: glebius
 Date: Mon Jan  6 19:05:04 2014
 New Revision: 260377
 URL: http://svnweb.freebsd.org/changeset/base/260377
 
 Log:
   When pf_get_translation() fails, it should leave *sn pointer pristine,
   otherwise we will panic in pf_test_rule().
   
   PR:		182557
 
 Modified:
   head/sys/netpfil/pf/pf_lb.c
 
 Modified: head/sys/netpfil/pf/pf_lb.c
 ==============================================================================
 --- head/sys/netpfil/pf/pf_lb.c	Mon Jan  6 17:23:22 2014	(r260376)
 +++ head/sys/netpfil/pf/pf_lb.c	Mon Jan  6 19:05:04 2014	(r260377)
 @@ -686,6 +686,7 @@ notrans:
  	uma_zfree(V_pf_state_key_z, *nkp);
  	uma_zfree(V_pf_state_key_z, *skp);
  	*skp = *nkp = NULL;
 +	*sn = NULL;
  
  	return (NULL);
  }
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/182557: commit references a PR
Date: Wed, 22 Jan 2014 10:45:24 +0000 (UTC)

 Author: glebius
 Date: Wed Jan 22 10:45:16 2014
 New Revision: 261023
 URL: http://svnweb.freebsd.org/changeset/base/261023
 
 Log:
   Merge r260377: fix panic on pf_get_translation() failure.
   
   PR:		182557
 
 Modified:
   stable/10/sys/netpfil/pf/pf_lb.c
 Directory Properties:
   stable/10/   (props changed)
 
 Modified: stable/10/sys/netpfil/pf/pf_lb.c
 ==============================================================================
 --- stable/10/sys/netpfil/pf/pf_lb.c	Wed Jan 22 10:38:01 2014	(r261022)
 +++ stable/10/sys/netpfil/pf/pf_lb.c	Wed Jan 22 10:45:16 2014	(r261023)
 @@ -663,6 +663,7 @@ notrans:
  	uma_zfree(V_pf_state_key_z, *nkp);
  	uma_zfree(V_pf_state_key_z, *skp);
  	*skp = *nkp = NULL;
 +	*sn = NULL;
  
  	return (NULL);
  }
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
