From Joerg.Pulz@frm2.tum.de  Mon May 21 07:29:47 2012
Return-Path: <Joerg.Pulz@frm2.tum.de>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 8BA0A1065673;
	Mon, 21 May 2012 07:29:47 +0000 (UTC)
	(envelope-from Joerg.Pulz@frm2.tum.de)
Received: from mailhost.frm2.tum.de (mailhost.frm2.tum.de [129.187.179.12])
	by mx1.freebsd.org (Postfix) with ESMTP id BF6078FC14;
	Mon, 21 May 2012 07:29:46 +0000 (UTC)
Received: from mailhost.frm2.tum.de (localhost [127.0.0.1])
	by mailhost.frm2.tum.de (8.14.4/8.14.4) with ESMTP id q4L7Oa6q084013;
	Mon, 21 May 2012 09:26:08 +0200 (CEST)
	(envelope-from jpulz@frm2.tum.de)
Received: from hades.admin.frm2 (hades.admin.frm2 [172.25.1.10])
	by mailhost.frm2.tum.de (8.14.4/8.14.4) with ESMTP id q4L7Q6D9084047
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT);
	Mon, 21 May 2012 09:26:06 +0200 (CEST)
	(envelope-from jpulz@frm2.tum.de)
Received: from hades.admin.frm2 (localhost [127.0.0.1])
	by hades.admin.frm2 (8.14.4/8.14.3) with ESMTP id q4L7Q6ku064259;
	Mon, 21 May 2012 09:26:06 +0200 (CEST)
	(envelope-from jpulz@frm2.tum.de)
Received: (from jpulz@localhost)
	by hades.admin.frm2 (8.14.4/8.14.3/Submit) id q4L7Q6m9064258;
	Mon, 21 May 2012 09:26:06 +0200 (CEST)
	(envelope-from jpulz)
Message-Id: <201205210726.q4L7Q6m9064258@hades.admin.frm2>
Date: Mon, 21 May 2012 09:26:06 +0200 (CEST)
From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
Reply-To: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: FreeBSD-gnats-submit@freebsd.org
Cc: freebsd-pf@freebsd.org
Subject: panic when using pf and route-to (maybe: bad fragment handling?)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         168190
>Category:       kern
>Synopsis:       [pf] panic when using pf and route-to (maybe: bad fragment handling?)
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-pf
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon May 21 07:30:05 UTC 2012
>Closed-Date:    
>Last-Modified:  Tue Jun  5 12:50:02 UTC 2012
>Originator:     Joerg Pulz
>Release:        FreeBSD 9.0-RELEASE-p1
>Organization:
TU Muenchen / FRM II
>Environment:
System: FreeBSD charon 9.0-RELEASE-p1 FreeBSD 9.0-RELEASE-p1 #3: Sun May 20 08:42:19 CEST 2012     root@charon:/usr/obj/usr/src/sys/IPSEC  amd64


	
>Description:
	We have a dual home VPN (IPSec) gateway running ipsec-tools.
	All packets coming from VPN clients have to hit the main router to
	generate and evaluate states. Therefor we use pf(4) and the route-to
	feature.
	Unfortunately the system panics (and it paniced with FreeBSD-8.x too)
	at irregular intervals with:
		"panic: m_copym, offset > size of mbuf chain"
	I'm unable to reproduce the network traffic to force the problem to
	appear.
	I recompiled the Kernel and configured the system to keep dumps to find
	the find the relevant place in the code for this issue.
	As i'm not that deep in network processing and pf code i can only guess
	that handling fragmented packets with pf(4)  and route-to is buggy
	somewhere.
	Below is the kgdb(1) output for the last dump with some more
	information.
	If someone can give me some advise i can produce more and detailed
	output.
	If necessary i can upload all requiered stuff somewhere.
>How-To-Repeat:
	
>Fix:

	

--- kgdb.out begins here ---
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: m_copym, offset > size of mbuf chain
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x182
m_copym() at m_copym+0x280
ip_fragment() at ip_fragment+0x1e5
pf_route() at pf_route+0x75c
pf_test() at pf_test+0xc29
pf_route() at pf_route+0x30a
pf_test() at pf_test+0xc29
pf_check_out() at pf_check_out+0x3a
pfil_run_hooks() at pfil_run_hooks+0xd2
ip_output() at ip_output+0x655
ip_forward() at ip_forward+0x175
ip_input() at ip_input+0x5fd
swi_net() at swi_net+0x15a
intr_event_execute_handlers() at intr_event_execute_handlers+0x66
ithread_loop() at ithread_loop+0xaf
fork_exit() at fork_exit+0x12a
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
KDB: enter: panic
Dumping 649 out of 4077 MB:..3%..13%..23%..33%..42%..52%..62%..72%..82%..92%

Reading symbols from /boot/kernel.old/geom_mirror.ko...Reading symbols from /boot/kernel.old/geom_mirror.ko.symbols...done.
done.
Loaded symbols for /boot/kernel.old/geom_mirror.ko
Reading symbols from /boot/kernel.old/ipmi.ko...Reading symbols from /boot/kernel.old/ipmi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel.old/ipmi.ko
#0  doadump (textdump=0) at pcpu.h:224
224		__asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) up 10
#10 0xffffffff806e9079 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
    at /usr/src/sys/kern/uipc_mbuf.c:541
541			KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
(kgdb) list
536		KASSERT(len >= 0, ("m_copym, negative len %d", len));
537		MBUF_CHECKSLEEP(wait);
538		if (off == 0 && m->m_flags & M_PKTHDR)
539			copyhdr = 1;
540		while (off > 0) {
541			KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
542			if (off < m->m_len)
543				break;
544			off -= m->m_len;
545			m = m->m_next;
(kgdb) p off
$1 = 1166
(kgdb) up 
#11 0xffffffff8077fe1f in ip_fragment (ip=0xfffffe0005b7a974, 
    m_frag=0xffffff8000241428, mtu=) at /usr/src/sys/netinet/ip_output.c:816
816			m->m_next = m_copym(m0, off, len, M_DONTWAIT);
(kgdb) list
811				len = ip->ip_len - off;
812				m->m_flags |= M_LASTFRAG;
813			} else
814				mhip->ip_off |= IP_MF;
815			mhip->ip_len = htons((u_short)(len + mhlen));
816			m->m_next = m_copym(m0, off, len, M_DONTWAIT);
817			if (m->m_next == NULL) {	/* copy failed */
818				m_free(m);
819				error = ENOBUFS;	/* ??? */
820				IPSTAT_INC(ips_odropped);
(kgdb) p *m0
$2 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0, 
    mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1, 
    pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800, 
        header = 0x0, len = 334, flowid = 0, csum_flags = 1, 
        csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, 
        tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = {
          ext_buf = 0x493a017c0045 <Address 0x493a017c0045 out of bounds>, 
          ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437, 
          ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586, 
          ref_cnt = 0xc6be758202079b0a, ext_type = 89195267}, 
        MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OGI\201WE\000\001N\002\016\000\000?\001$\n\233\a\002\202u\003\003Q\005\000\000\000\000E\000\0012S\000\0000\021;\217\202u\n\233\a\002\a\001\036QKS\000\000\001B\000\000\000\001\v\001"}}, 
    M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000\220\n\005\000E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OGI\201WE\000\001N\002\016\000\000?\001$\n\233\a\002\202u\003\003Q\005\000\000\000\000E\000\0012S\000\0000\021;\217\202u\n\233\a\002\a\001\036QKS\000\000\001B\000\000\000\001\v\001"...}}
(kgdb) p off
$3 = 1500
(kgdb) p len
$4 = 1480
(kgdb) p hlen
$5 = 20
(kgdb) up  
#12 0xffffffff8032842a in pf_route (m=0xffffff8000241658, 
    r=0xfffffe0005dc8af8, dir=) at /usr/src/sys/contrib/pf/net/pf.c:6138
6138		error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
(kgdb) list
6133		/*
6134		 * XXX: is cheaper + less error prone than own function
6135		 */
6136		NTOHS(ip->ip_len);
6137		NTOHS(ip->ip_off);
6138		error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
6139	#else
6140		error = ip_fragment(m0, ifp, ifp->if_mtu);
6141	#endif
6142		if (error) {
(kgdb) p *ip
$6 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 19969, 
  ip_id = 3586, ip_off = 0, ip_ttl = 63 '?', ip_p = 1 '\001', ip_sum = 51492, 
  ip_src = {s_addr = 34052874}, ip_dst = {s_addr = 3334370690}}
(kgdb) p *m0
$7 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0, 
    mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1, 
    pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800, 
        header = 0x0, len = 334, flowid = 0, csum_flags = 1, 
        csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, 
        tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = {
          ext_buf = 0x493a017c0045 <Address 0x493a017c0045 out of bounds>, 
          ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437, 
          ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586, 
          ref_cnt = 0xc6be758202079b0a, ext_type = 89195267}, 
        MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OGI\201WE\000\001N\002\016\000\000?\001$\n\233\a\002\202u\003\003Q\005\000\000\000\000E\000\0012S\000\0000\021;\217\202u\n\233\a\002\a\001\036QKS\000\000\001B\000\000\000\001\v\001"}}, 
    M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000\220\n\005\000E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OGI\201WE\000\001N\002\016\000\000?\001$\n\233\a\002\202u\003\003Q\005\000\000\000\000E\000\0012S\000\0000\021;\217\202u\n\233\a\002\a\001\036QKS\000\000\001B\000\000\000\001\v\001"...}}
(kgdb) p *m1
$8 = {m_hdr = {mh_next = 0xfffffe0052a2e200, mh_nextpkt = 0x0, 
    mh_data = 0xfffffe0005b7a974 "E", mh_len = 60, mh_flags = 66, mh_type = 1, 
    pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800, 
        header = 0x0, len = 334, flowid = 0, csum_flags = 1, 
        csum_data = 27136, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0}, 
        tags = {slh_first = 0xfffffe00050a90c0}}, MH_dat = {MH_ext = {
          ext_buf = 0x493a017c0045 <Address 0x493a017c0045 out of bounds>, 
          ext_free = 0x493a014e0045, ext_arg1 = 0x4947cb4f287c0437, 
          ext_arg2 = 0x4e01004557b3bb81, ext_size = 3586, 
          ref_cnt = 0xc6be758202079b0a, ext_type = 89195267}, 
        MH_databuf = "E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OGI\201WE\000\001N\002\016\000\000?\001$\n\233\a\002\202u\003\003Q\005\000\000\000\000E\000\0012S\000\0000\021;\217\202u\n\233\a\002\a\001\036QKS\000\000\001B\000\000\000\001\v\001"}}, 
    M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000N\001\000\000\000\000\000\000\001\000\000\000\000j\000\000\000\000\000\000\220\n\005\000E\000|\001:I\000\000E\000N\001:I\000\0007\004|(OGI\201WE\000\001N\002\016\000\000?\001$\n\233\a\002\202u\003\003Q\005\000\000\000\000E\000\0012S\000\0000\021;\217\202u\n\233\a\002\a\001\036QKS\000\000\001B\000\000\000\001\v\001"...}}
(kgdb) 
--- kgdb.out ends here ---


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-pf 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 21 08:28:29 UTC 2012 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=168190 

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Mon, 21 May 2012 16:14:30 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Mon, 21 May 2012, Daniel Hartmeier wrote:
 
 > Semantics of such route-to chains are not well defined. There's an mbuf
 > tag to prevent endless loops, but obviously even short chains are not
 > working properly. I'd try to avoid them.
 
 Daniel,
 
 thanks for looking at this and your explanation.
 The route-to rules are pretty specific:
 
 #### pf.conf
 ext_if="bge0"
 int_if="bge1"
 vpn_net="10.1.1.0/24"
 srv_net="172.16.1.0/24"
 gw_addr="172.16.1.254"
 
 scrub in all
 
 pass out on $ext_if route-to ($int_if $gw_addr) from $vpn_net to any keep state
 pass out on $int_if route-to ($int_if $gw_addr) from $vpn_net to $srv_net keep state
 #### pf.conf
 
 The default gateway for the host itself is reachable on the external 
 interface and there is a static route for our internal networks 
 (172.16.0.0/16) configured for the system itself.
 All client traffic has to hit the $gw_addr first regardless in which 
 direction it goes afterwards, that's where the route-to rules kick in.
 Do they have to be even more specific if possible at all?
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPuk3JSPOsGF+KA+MRAh9BAJ9TRkTeB12NtqYOOmdRcDaTpBjPOgCdE3S1
 6rcDkcoro92HI/db4pMLDn4=
 =w64E
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Mon, 21 May 2012 20:48:12 +0200 (CEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-1197118405-1337625847=:74709
 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 8BIT
 Content-ID: <alpine.BSF.2.00.1205212044251.74709@unqrf.nqzva.sez2>
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Content-ID: <alpine.BSF.2.00.1205212044250.74709@unqrf.nqzva.sez2>
 
 On Mon, 21 May 2012, Daniel Hartmeier wrote:
 
 > Or, instead of adding a third rule, add '! tagged from_vpn' to the
 > second rule, if tagged packets can still pass out on $int_if by another
 > rule.
 
 Daniel,
 
 thanks again.
 There is one main question left for me.
 How could a packet which is directed to a host in $srv_net (and $srv_net 
 is the same broadcast domain as $int_if) ever leave through $ext_if and 
 make the first route-to match and therefor lead to a double route-to?
 
 Regarding your other mail with the reproduce instructions
 
 > BTW, if the theory is correct, you should be able to reproduce the
 > problem by sending a packet in from $vpn_net to a host in $srv_net
 > of size 334 (=306+28), e.g. with
 >
 >  $ ping -s 306 172.16.1.1
 
 I tried this, but it did no harm to the system at all. Packets where 
 flowing as they should and answers came back.
 
 PF ruleset is still unchanged, no tagging.
 Meanwhile the system panic'ed again, and this time there is no double 
 route-to involved at all. kgdb(1) output below.
 If your assumption in your first response is right:
 
 > It looks like the byte order of ip_len is wrong, htons(334) = 19969,
 > triggering fragmentation (334 < if_mtu, but 19969 > if_mtu).
 
 Then it should read htons(175) = 44800
 (175 < if_mtu, but 44800 > if_mtu")
 
 But where is it coming from if double route-to can't be the culprit in 
 this case?
 
 Any thoughts on this?
 
 Kind regards
 Joerg
 
 ### kgdb.out2
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: m_copym, offset > size of mbuf chain
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic+0x182
 m_copym() at m_copym+0x280
 ip_fragment() at ip_fragment+0x1e5
 pf_route() at pf_route+0x75c
 pf_test() at pf_test+0xc29
 pf_check_out() at pf_check_out+0x3a
 pfil_run_hooks() at pfil_run_hooks+0xd2
 ip_output() at ip_output+0x655
 ip_forward() at ip_forward+0x175
 ip_input() at ip_input+0x5fd
 swi_net() at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 Dumping 628 out of 4077 MB:..3%..11%..21%..31%..41%..51%..62%..72%..82%..92%
 
 Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_mirror.ko
 Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipmi.ko
 #0  doadump (textdump=0) at pcpu.h:224
 224		__asm("movq %%gs:0,%0" : "=r" (td));
 (kgdb) up 10
 #10 0xffffffff806e9079 in m_copym (m=0x0, off0=1500, len=1480, wait=1)
      at /usr/src/sys/kern/uipc_mbuf.c:541
 541			KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
 (kgdb) list
 536		KASSERT(len >= 0, ("m_copym, negative len %d", len));
 537		MBUF_CHECKSLEEP(wait);
 538		if (off == 0 && m->m_flags & M_PKTHDR)
 539			copyhdr = 1;
 540		while (off > 0) {
 541			KASSERT(m != NULL, ("m_copym, offset > size of mbuf chain"));
 542			if (off < m->m_len)
 543				break;
 544			off -= m->m_len;
 545			m = m->m_next;
 (kgdb) p off
 $1 = 1325
 (kgdb) up
 #11 0xffffffff8077fe1f in ip_fragment (ip=0xfffffe0173b1fc74,
      m_frag=0xffffff8000241658, mtu=) at /usr/src/sys/netinet/ip_output.c:816
 816			m->m_next = m_copym(m0, off, len, M_DONTWAIT);
 (kgdb) list
 811				len = ip->ip_len - off;
 812				m->m_flags |= M_LASTFRAG;
 813			} else
 814				mhip->ip_off |= IP_MF;
 815			mhip->ip_len = htons((u_short)(len + mhlen));
 816			m->m_next = m_copym(m0, off, len, M_DONTWAIT);
 817			if (m->m_next == NULL) {	/* copy failed */
 818				m_free(m);
 819				error = ENOBUFS;	/* ??? */
 820				IPSTAT_INC(ips_odropped);
 (kgdb) p *m0
 $2 = {m_hdr = {mh_next = 0xfffffe0005b70000, mh_nextpkt = 0x0,
      mh_data = 0xfffffe0173b1fc74 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
      pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
          header = 0x0, len = 175, flowid = 0, csum_flags = 1,
          csum_data = 59883, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
          tags = {slh_first = 0xfffffe00058c3840}}, MH_dat = {MH_ext = {
            ext_buf = 0xd29300ec0045 <Address 0xd29300ec0045 out of bounds>,
            ext_free = 0xd29300af0045, ext_arg1 = 0xba41965002280437,
            ext_arg2 = 0xaf00004557b3bb81, ext_size = 11615,
            ref_cnt = 0x240119ac02079b0a, ext_type = 1740374787},
          MH_databuf = "E\000\000\223\000\000E\000\000\223\000\0007\004(\002P\226A\201WE\000\000_-\000\000\177\001\034G\n\233\a\002\031\001$\003\003g\000\000\000\000E\000\000\223\215:\000\000>\0210F\031\001$\n\233\a\002\0005v\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000"}},
      M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000@8\214\005\000E\000\000\223\000\000E\000\000\223\000\0007\004(\002P\226A\201WE\000\000_-\000\000\177\001\034G\n\233\a\002\031\001$\003\003g\000\000\000\000E\000\000\223\215:\000\000>\0210F\031\001$\n\233\a\002\0005v\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000"...}}
 (kgdb) p off
 $3 = 1500
 (kgdb) p len
 $4 = 1480
 (kgdb) p hlen
 $5 = 20
 (kgdb) up
 #12 0xffffffff8032842a in pf_route (m=0xffffff80002418e8,
      r=0xfffffe0005d87750, dir=) at /usr/src/sys/contrib/pf/net/pf.c:6138
 6138		error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
 (kgdb) list
 6133		/*
 6134		 * XXX: is cheaper + less error prone than own function
 6135		 */
 6136		NTOHS(ip->ip_len);
 6137		NTOHS(ip->ip_off);
 6138		error = ip_fragment(ip, &m0, ifp->if_mtu, ifp->if_hwassist, sw_csum);
 6139	#else
 6140		error = ip_fragment(m0, ifp, ifp->if_mtu);
 6141	#endif
 6142		if (error) {
 (kgdb) p *ip
 $6 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 44800,
    ip_id = 11615, ip_off = 0, ip_ttl = 127 '\177', ip_p = 1 '\001',
    ip_sum = 18204, ip_src = {s_addr = 34052874}, ip_dst = {s_addr = 604051884}}
 (kgdb) p *m0
 $7 = {m_hdr = {mh_next = 0xfffffe0005b70000, mh_nextpkt = 0x0,
      mh_data = 0xfffffe0173b1fc74 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
      pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
          header = 0x0, len = 175, flowid = 0, csum_flags = 1,
          csum_data = 59883, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
          tags = {slh_first = 0xfffffe00058c3840}}, MH_dat = {MH_ext = {
            ext_buf = 0xd29300ec0045 <Address 0xd29300ec0045 out of bounds>,
            ext_free = 0xd29300af0045, ext_arg1 = 0xba41965002280437,
            ext_arg2 = 0xaf00004557b3bb81, ext_size = 11615,
            ref_cnt = 0x240119ac02079b0a, ext_type = 1740374787},
          MH_databuf = "E\000\000\223\000\000E\000\000\223\000\0007\004(\002P\226A\201WE\000\000_-\000\000\177\001\034G\n\233\a\002\031\001$\003\003g\000\000\000\000E\000\000\223\215:\000\000>\0210F\031\001$\n\233\a\002\0005v\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000"}},
      M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000@8\214\005\000E\000\000\223\000\000E\000\000\223\000\0007\004(\002P\226A\201WE\000\000_-\000\000\177\001\034G\n\233\a\002\031\001$\003\003g\000\000\000\000E\000\000\223\215:\000\000>\0210F\031\001$\n\233\a\002\0005v\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000"...}}
 (kgdb) p *m1
 $8 = {m_hdr = {mh_next = 0xfffffe0005b70000, mh_nextpkt = 0x0,
      mh_data = 0xfffffe0173b1fc74 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
      pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
          header = 0x0, len = 175, flowid = 0, csum_flags = 1,
          csum_data = 59883, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
          tags = {slh_first = 0xfffffe00058c3840}}, MH_dat = {MH_ext = {
            ext_buf = 0xd29300ec0045 <Address 0xd29300ec0045 out of bounds>,
            ext_free = 0xd29300af0045, ext_arg1 = 0xba41965002280437,
            ext_arg2 = 0xaf00004557b3bb81, ext_size = 11615,
            ref_cnt = 0x240119ac02079b0a, ext_type = 1740374787},
          MH_databuf = "E\000\000\223\000\000E\000\000\223\000\0007\004(\002P\226A\201WE\000\000_-\000\000\177\001\034G\n\233\a\002\031\001$\003\003g\000\000\000\000E\000\000\223\215:\000\000>\0210F\031\001$\n\233\a\002\0005v\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000"}},
      M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\000@8\214\005\000E\000\000\223\000\000E\000\000\223\000\0007\004(\002P\226A\201WE\000\000_-\000\000\177\001\034G\n\233\a\002\031\001$\003\003g\000\000\000\000E\000\000\223\215:\000\000>\0210F\031\001$\n\233\a\002\0005v\000\177\vC\200\035\201\200\000\001\000\002\000\002\000\000"...}}
 (kgdb)
 
 ### kgdb.out2
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPuo3vSPOsGF+KA+MRAu+AAKDGT0lDqaIcYO4Q6Lx37oUX64GeCgCffGhf
 AtfXrgD94GTXHsX7roaKfAI=
 =wEqQ
 -----END PGP SIGNATURE-----
 --3469798045-1197118405-1337625847=:74709--

Date: Mon, 21 May 2012 11:27:50 +0200
From: Daniel Hartmeier <daniel@benzedrine.cx>
To: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
Subject: Re: panic when using pf and route-to (maybe: bad fragment handling?)

 It looks like the byte order of ip_len is wrong, htons(334) = 19969,
 triggering fragmentation (334 < if_mtu, but 19969 > if_mtu).
 
 The reason is most likely the recursive pf_route() call:
 
 > m_copym() at m_copym+0x280
 > ip_fragment() at ip_fragment+0x1e5
 > pf_route() at pf_route+0x75c
 > pf_test() at pf_test+0xc29
 > pf_route() at pf_route+0x30a
 > pf_test() at pf_test+0xc29
 > pf_check_out() at pf_check_out+0x3a
 > pfil_run_hooks() at pfil_run_hooks+0xd2
 > ip_output() at ip_output+0x655
 
 i.e. the packet is filtered when going out on some interface, matching a
 route-to rule.
 
 Now the packet is filtered again on the routed-to interface. So far ok,
 this is expected.
 
 But now it matches a route-to rule again, possibly the same one, because
 it doesn't restrict the interface.
 
 Usually, this is not intentional (double route-to), and can be fixed by
 checking the route-to rule(s) and making them more restrictive.
 
 Semantics of such route-to chains are not well defined. There's an mbuf
 tag to prevent endless loops, but obviously even short chains are not
 working properly. I'd try to avoid them.
 
 Daniel

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Tue, 22 May 2012 08:05:39 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Mon, 21 May 2012, Daniel Hartmeier wrote:
 
 > Or, instead of adding a third rule, add '! tagged from_vpn' to the
 > second rule, if tagged packets can still pass out on $int_if by another
 > rule.
 
 And i got another panic, this time without pf(4) involved at all.
 Unfortunately "dump" in kdb is doing nothing but hang. :-(
 
 Here is what was displayed on the screen:
 
 panic: m_copym, offset > size of mbuf chain
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic_0x182
 m_copym() at m_copym+0x280
 ip_fragment() at ip_fragment+0x1e5
 ip_output() at ip_output+0xeab
 ip_forward()  at ip_forward+0x175
 ip_input() at ip_input+0x5fd
 swi_net()  at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xfffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 [ thread pid 12 tid 100008 ]
 
 Any thoughts about this one?
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPuyy2SPOsGF+KA+MRAtcgAJwO4zh4/AZN2tHhySI+24y+kozM3gCgn+HS
 /ZIUDnpvQCGdRWXYvvBauZw=
 =xctG
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Tue, 22 May 2012 13:51:51 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Tue, 22 May 2012, Daniel Hartmeier wrote:
 
 > What checksumming support (ifconfig if)?
 
 Daniel,
 
 mails to your personal eMail address are bouncing.
 relay=insomnia.benzedrine.cx. [62.65.145.30], dsn=4.0.0, stat=Deferred: 
 insomnia.benzedrine.cx.: No route to host
 
 I've found another report and a patch which i already tried without 
 success, so i reverted back to stock 9.0-p1.
 
 http://lists.freebsd.org/pipermail/freebsd-pf/2005-March/000922.html
 
 I've the following relevant options in the kernel configuration:
 
 options         IPFIREWALL
 options         IPFIREWALL_VERBOSE
 options         IPFIREWALL_VERBOSE_LIMIT=100
 options         IPFIREWALL_DEFAULT_TO_ACCEPT
 options         IPDIVERT
 options         IPFILTER
 options         IPFILTER_LOG
 options         IPSTEALTH
 
 options         ALTQ
 options         ALTQ_CBQ        # Class Bases Queueing
 options         ALTQ_RED        # Random Early Drop
 options         ALTQ_RIO        # RED In/Out
 options         ALTQ_HFSC       # Hierarchical Packet Scheduler
 options         ALTQ_CDNR       # Traffic conditioner
 options         ALTQ_PRIQ       # Priority Queueing
 options         ALTQ_NOPCC      # Required for SMP build
 
 options         IPSEC
 options         IPSEC_NAT_T
 
 device          crypto
 device          cryptodev
 device          hifn
 
 device          enc
 
 device          pf              # PF OpenBSD packet-filter firewall
 device          pflog           # logging support interface for PF
 device          pfsync          # synchronization interface for PF
 device          carp            # common address redundancy protocol
 
 Only pf(4) is configured and used.
 
    net.inet.ip.forwarding: 1
    net.inet.ip.fastforwarding: 0
    net.inet6.ip6.forwarding: 0
 
 No netgraph, divert or dup-to.
 
 Interface list:
 
 bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
          options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
 bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
          options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
 pflog0: flags=0<> metric 0 mtu 33152
 pfsync0: flags=0<> metric 0 mtu 1500
 ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
          options=3<RXCSUM,TXCSUM>
 enc0: flags=0<> metric 0 mtu 1536
 
 Only bge0 and bge1 are configured and used. bge0 ist $ext_if and bge1 is 
 $int_if.
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPu33aSPOsGF+KA+MRAjkLAJ0Z6K0Smp5M2p9r/VcSAUy1nqnkAACgqMq7
 oHMudSKOjU3nQIGaq3M0fAo=
 =SuIg
 -----END PGP SIGNATURE-----

Date: Tue, 22 May 2012 12:19:30 +0200 (CEST)
From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Subject: Re: panic when using pf and route-to (maybe: bad fragment
 handling?)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-1358106948-1337681906=:89783
 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-15; FORMAT=flowed
 Content-Transfer-Encoding: 8BIT
 Content-ID: <alpine.BSF.2.00.1205221218541.89783@unqrf.nqzva.sez2>
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Content-ID: <alpine.BSF.2.00.1205221218540.89783@unqrf.nqzva.sez2>
 
 On Tue, 22 May 2012, Daniel Hartmeier wrote:
 
 > Can you print *ifp in this context, please?
 >
 > Just to make sure if_mtu is sane.
 
 Daniel,
 
 thanks for all your effort.
 Here comes *ifp
 
 Joerg
 
 #### kgdb.out
 #12 0xffffffff8032842a in pf_route (m=0xffffff80002418e8,
      r=0xfffffe0005e05750, dir=Variable "dir" is not available.
 ) at /usr/src/sys/contrib/pf/net/pf.c:6138
 6138            error = ip_fragment(ip, &m0, ifp->if_mtu, 
 ifp->if_hwassist, sw_csum);
 (kgdb) p *ifp
 $1 = {if_softc = 0xffffff80007b1000, if_l2com = 0xfffffe000300ba40,
    if_vnet = 0x0, if_link = {tqe_next = 0xfffffe0003001000,
      tqe_prev = 0xfffffe0003001818},
    if_xname = "bge1", '\0' <repeats 11 times>,
    if_dname = 0xfffffe00028f07d8 "bge", if_dunit = 1, if_refcount = 1,
    if_addrhead = {tqh_first = 0xfffffe0003009800,
      tqh_last = 0xfffffe0005abdcb8}, if_pcount = 0, if_carp = 0x0,
    if_bpf = 0xfffffe00050e7900, if_index = 6, if_index_reserved = 0,
    if_vlantrunk = 0x0, if_flags = 34819, if_capabilities = 524443,
    if_capenable = 524443, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
      ifi_type = 6 '\006', ifi_physical = 0 '\0', ifi_addrlen = 6 '\006',
      ifi_hdrlen = 18 '\022', ifi_link_state = 2 '\002',
      ifi_spare_char1 = 0 '\0', ifi_spare_char2 = 0 '\0',
      ifi_datalen = 152 '\230', ifi_mtu = 1500, ifi_metric = 0,
      ifi_baudrate = 1000000000, ifi_ipackets = 54812, ifi_ierrors = 0,
      ifi_opackets = 34745, ifi_oerrors = 0, ifi_collisions = 0,
      ifi_ibytes = 41868704, ifi_obytes = 5296902, ifi_imcasts = 10095,
      ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 3,
      ifi_epoch = 1, ifi_lastchange = {tv_sec = 1337441486, tv_usec = 788343}},
    if_multiaddrs = {tqh_first = 0xfffffe00059137c0,
      tqh_last = 0xfffffe0005914300}, if_amcount = 0,
    if_output = 0xffffffff8073d4b5 <ether_output>,
    if_input = 0xffffffff8073ca8b <ether_input>,
    if_start = 0xffffffff803c2da7 <bge_start>,
    if_ioctl = 0xffffffff803c8fda <bge_ioctl>,
    if_init = 0xffffffff803c8f94 <bge_init>,
    if_resolvemulti = 0xffffffff8073c44d <ether_resolvemulti>,
    if_qflush = 0xffffffff807352f2 <if_qflush>,
    if_transmit = 0xffffffff807351be <if_transmit>, if_reassign = 0,
    if_home_vnet = 0x0, if_addr = 0xfffffe0003009800, if_llsoftc = 0x0,
    if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0,
      ifq_maxlen = 511, ifq_drops = 0, ifq_mtx = {lock_object = {
          lo_name = 0xfffffe0003002028 "bge1", lo_flags = 16973824, lo_data = 0,
          lo_witness = 0xffffff80006cf480}, mtx_lock = 4}, ifq_drv_head = 0x0,
      ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 511, altq_type = 0,
      altq_flags = 1, altq_disc = 0x0, altq_ifp = 0xfffffe0003002000,
      altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0,
      altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0},
    if_broadcastaddr = 0xffffffff80ada7c0 "", if_bridge = 0x0,
    if_label = 0x0, if_prefixhead = {tqh_first = 0x0,
      tqh_last = 0xfffffe0003002278}, if_afdata = {0x0, 0x0, 0xfffffe000581fa00,
      0x0 <repeats 25 times>, 0xfffffe0005814800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, if_afdata_initialized = 2, if_afdata_lock = {
      lock_object = {lo_name = 0xffffffff80ad9a5a "if_afdata",
        lo_flags = 69402624, lo_data = 0, lo_witness = 0xffffff80006cf400},
      rw_lock = 1}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
      ta_priority = 0, ta_func = 0xffffffff80737799 <do_link_state_change>,
      ta_context = 0xfffffe0003002000}, if_addr_mtx = {lock_object = {
        lo_name = 0xffffffff80acbb20 "if_addr_mtx", lo_flags = 16973824,
        lo_data = 0, lo_witness = 0xffffff80006c8b80}, mtx_lock = 4},
    if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {
      tqh_first = 0xfffffe0005093ae0, tqh_last = 0xfffffe0005093ae8},
    if_pf_kif = 0xfffffe0005889300, if_lagg = 0x0, if_description = 0x0,
    if_fib = 0, if_alloctype = 6 '\006', if_cspare = "\000\000", if_ispare = {0,
      0, 0, 0}, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
 #### kgdb.out
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPu2g1SPOsGF+KA+MRAoTIAJ9zBBTdm9naccUy+S2u89hqcXl2kACfRApP
 bJ+OVmJETP0NtLujBxb7Kqg=
 =MqcS
 -----END PGP SIGNATURE-----
 --3469798045-1358106948-1337681906=:89783--

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: =?ISO-8859-15?Q?Ermal_Lu=E7i?= <eri@freebsd.org>,
        FreeBSD-gnats-submit@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Wed, 23 May 2012 21:48:03 +0200 (CEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-1716925155-1337802345=:21881
 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 8BIT
 Content-ID: <alpine.BSF.2.00.1205232145561.21881@unqrf.nqzva.sez2>
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Content-ID: <alpine.BSF.2.00.1205232145560.21881@unqrf.nqzva.sez2>
 
 On Tue, 22 May 2012, Daniel Hartmeier wrote:
 
 > If you have the chance, please try the patch below.
 >
 > It adds byte order checks all over the place, hoping for a panic closer
 > to the source of the problem.
 
 Daniel,
 
 system was running for about a day with your patch with many users using 
 it. It panic'ed some minutes ago.
 System configuration is still the same, no other patches, no changed 
 interface settings or removed/changed kernel options.
 
 Here is the kgdb(1) output with "m" and "ifp" listed.
 I hope this helps to get closer to the source of the problem.
 
 Let me know if you need more output.
 
 Kind regards
 Joerg
 
 
 #### kgdb.out_assert
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: ASSERT_HOST_BYTE_ORDER
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic+0x182
 pfil_run_hooks() at pfil_run_hooks+0x159
 ip_output() at ip_output+0x6de
 ip_forward() at ip_forward+0x19e
 ip_input() at ip_input+0x670
 swi_net() at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 Dumping 585 out of 4077 MB:..3%..11%..22%..31%..41%..52%..61%..72%..82%..91%
 
 Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_mirror.ko
 Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipmi.ko
 #0  doadump (textdump=0) at pcpu.h:224
 224		__asm("movq %%gs:0,%0" : "=r" (td));
 (kgdb) up 10
 #10 0xffffffff8074b325 in pfil_run_hooks (ph=0xfffffe000581f880,
      mp=0xffffff8000241978, ifp=0xfffffe0003002000, dir=2, inp=0x0)
      at /usr/src/sys/net/pfil.c:89
 89				ASSERT_HOST_BYTE_ORDER(m);
 (kgdb) list
 84				ASSERT_HOST_BYTE_ORDER(m);
 85				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 86				    inp);
 87				if (rv != 0 || m == NULL)
 88					break;
 89				ASSERT_HOST_BYTE_ORDER(m);
 90			}
 91		}
 92		PFIL_RUNLOCK(ph, &rmpt);
 93		*mp = m;
 (kgdb) p *m
 $1 = {m_hdr = {mh_next = 0xfffffe000586bb00, mh_nextpkt = 0x0,
      mh_data = 0xfffffe010045c974 "E", mh_len = 60, mh_flags = 66, mh_type = 1,
      pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
          header = 0x0, len = 450, flowid = 0, csum_flags = 768,
          csum_data = 26073, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
          tags = {slh_first = 0xfffffe000572c700}}, MH_dat = {MH_ext = {
            ext_buf = 0xc02c01fc0045 <Address 0xc02c01fc0045 out of bounds>,
            ext_free = 0xc02c01c20045, ext_arg1 = 0x4d46cb4f398a0437,
            ext_arg2 = 0xc201004557b3bb81, ext_size = 21286,
            ref_cnt = 0x240119ac02079b0a, ext_type = 2059207427},
          MH_databuf = "E\000\001,\000\000E\000\001,\000\0007\004\2129OFM\201WE\000\001&S\000\000?\001\224\016\n\233\a\002\031\001$\003\003z\000\000\000\000E\000\001\000\000>\021\177\031\001$\n\233\a\002\0005_\001\222)dh\201\200\000\001\000\003\000\b\000\b"}},
      M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\003\000\000e\000\000\000\000\000\000\000r\005\000E\000\001,\000\000E\000\001,\000\0007\004\2129OFM\201WE\000\001&S\000\000?\001\224\016\n\233\a\002\031\001$\003\003z\000\000\000\000E\000\001\000\000>\021\177\031\001$\n\233\a\002\0005_\001\222)dh\201\200\000\001\000\003\000\b\000\b"...}}
 (kgdb) p *ifp
 $2 = {if_softc = 0xffffff80007b1000, if_l2com = 0xfffffe000300ba40,
    if_vnet = 0x0, if_link = {tqe_next = 0xfffffe0003001000,
      tqe_prev = 0xfffffe0003001818},
    if_xname = "bge1", '\0' <repeats 11 times>,
    if_dname = 0xfffffe00028f07d8 "bge", if_dunit = 1, if_refcount = 1,
    if_addrhead = {tqh_first = 0xfffffe0003009800,
      tqh_last = 0xfffffe000591b4b8}, if_pcount = 0, if_carp = 0x0,
    if_bpf = 0xfffffe0005126900, if_index = 6, if_index_reserved = 0,
    if_vlantrunk = 0x0, if_flags = 34819, if_capabilities = 524443,
    if_capenable = 524443, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
      ifi_type = 6 '\006', ifi_physical = 0 '\0', ifi_addrlen = 6 '\006',
      ifi_hdrlen = 18 '\022', ifi_link_state = 2 '\002',
      ifi_spare_char1 = 0 '\0', ifi_spare_char2 = 0 '\0',
      ifi_datalen = 152 '\230', ifi_mtu = 1500, ifi_metric = 0,
      ifi_baudrate = 1000000000, ifi_ipackets = 1922972, ifi_ierrors = 0,
      ifi_opackets = 962786, ifi_oerrors = 0, ifi_collisions = 0,
      ifi_ibytes = 1150684321, ifi_obytes = 312161748, ifi_imcasts = 942443,
      ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 3,
      ifi_epoch = 1, ifi_lastchange = {tv_sec = 1337714565, tv_usec = 347019}},
    if_multiaddrs = {tqh_first = 0xfffffe0005915900,
      tqh_last = 0xfffffe0005a39100}, if_amcount = 0,
    if_output = 0xffffffff8073d805 <ether_output>,
    if_input = 0xffffffff8073cddb <ether_input>,
    if_start = 0xffffffff803c3087 <bge_start>,
    if_ioctl = 0xffffffff803c92ba <bge_ioctl>,
    if_init = 0xffffffff803c9274 <bge_init>,
    if_resolvemulti = 0xffffffff8073c79d <ether_resolvemulti>,
    if_qflush = 0xffffffff807355d2 <if_qflush>,
    if_transmit = 0xffffffff8073549e <if_transmit>, if_reassign = 0,
    if_home_vnet = 0x0, if_addr = 0xfffffe0003009800, if_llsoftc = 0x0,
    if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0,
      ifq_maxlen = 511, ifq_drops = 0, ifq_mtx = {lock_object = {
          lo_name = 0xfffffe0003002028 "bge1", lo_flags = 16973824, lo_data = 0,
          lo_witness = 0xffffff80006cf480}, mtx_lock = 4}, ifq_drv_head = 0x0,
      ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 511, altq_type = 0,
      altq_flags = 1, altq_disc = 0x0, altq_ifp = 0xfffffe0003002000,
      altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0,
      altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0},
    if_broadcastaddr = 0xffffffff80adb000 "", if_bridge = 0x0,
    if_label = 0x0, if_prefixhead = {tqh_first = 0x0,
      tqh_last = 0xfffffe0003002278}, if_afdata = {0x0, 0x0, 0xfffffe000581fa00,
      0x0 <repeats 25 times>, 0xfffffe0005814800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, if_afdata_initialized = 2, if_afdata_lock = {
      lock_object = {lo_name = 0xffffffff80ada29a "if_afdata",
        lo_flags = 69402624, lo_data = 0, lo_witness = 0xffffff80006cf400},
      rw_lock = 1}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
      ta_priority = 0, ta_func = 0xffffffff80737a79 <do_link_state_change>,
      ta_context = 0xfffffe0003002000}, if_addr_mtx = {lock_object = {
        lo_name = 0xffffffff80acc360 "if_addr_mtx", lo_flags = 16973824,
        lo_data = 0, lo_witness = 0xffffff80006c8b80}, mtx_lock = 4},
    if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {
      tqh_first = 0xfffffe00050d3ae0, tqh_last = 0xfffffe00050d3ae8},
    if_pf_kif = 0xfffffe0005889300, if_lagg = 0x0, if_description = 0x0,
    if_fib = 0, if_alloctype = 6 '\006', if_cspare = "\000\000", if_ispare = {0,
      0, 0, 0}, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
 (kgdb)
 
 #### kgdb.out_assert
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPvT72SPOsGF+KA+MRAvxgAJ91uOe4RymMtaUOoZ7IK61/qHpoSQCZAbd0
 /LVHK3BmvPKBUbd6e5rokUE=
 =9vPz
 -----END PGP SIGNATURE-----
 --3469798045-1716925155-1337802345=:21881--

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: FreeBSD-gnats-submit@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Thu, 24 May 2012 00:06:54 +0200 (CEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-489758976-1337810639=:24195
 Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-15; format=flowed
 Content-Transfer-Encoding: 8BIT
 Content-ID: <alpine.BSF.2.00.1205240006221.24271@unqrf.nqzva.sez2>
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Content-ID: <alpine.BSF.2.00.1205240006220.24271@unqrf.nqzva.sez2>
 
 On Wed, 23 May 2012, Daniel Hartmeier wrote:
 
 > Oh, we can identify the pfil hook by printing *pfh, pfh->pfil_func and
 > comparing the address to that of pf_check_out, fr_check_wrapper, and the
 > one for ipfw, right?
 
 Daniel,
 
 here is what i could get out.
 I was unable to print *pfh and pfh->pfil_func, but i printed the other 
 two and *ph, maybe this helps.
 
 Joerg
 
 #### kgdb.out_assert2
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: ASSERT_HOST_BYTE_ORDER
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic+0x182
 pfil_run_hooks() at pfil_run_hooks+0x159
 ip_output() at ip_output+0x6de
 ip_forward() at ip_forward+0x19e
 ip_input() at ip_input+0x670
 swi_net() at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 Dumping 585 out of 4077 MB:..3%..11%..22%..31%..41%..52%..61%..72%..82%..91%
 
 Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_mirror.ko
 Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipmi.ko
 #0  doadump (textdump=0) at pcpu.h:224
 224		__asm("movq %%gs:0,%0" : "=r" (td));
 (kgdb) up 10
 #10 0xffffffff8074b325 in pfil_run_hooks (ph=0xfffffe000581f880,
      mp=0xffffff8000241978, ifp=0xfffffe0003002000, dir=2, inp=0x0)
      at /usr/src/sys/net/pfil.c:89
 89				ASSERT_HOST_BYTE_ORDER(m);
 (kgdb) list
 84				ASSERT_HOST_BYTE_ORDER(m);
 85				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 86				    inp);
 87				if (rv != 0 || m == NULL)
 88					break;
 89				ASSERT_HOST_BYTE_ORDER(m);
 90			}
 91		}
 92		PFIL_RUNLOCK(ph, &rmpt);
 93		*mp = m;
 (kgdb) p *pfh
 (kgdb) p pfh->pfil_func
 (kgdb) p pf_check_out
 $1 = {int (void *, struct mbuf **, struct ifnet *, int, struct inpcb
       *)} 0xffffffff8032d17a <pf_check_out>
 (kgdb) p fr_check_wrapper
 $2 = {int (void *, struct mbuf **, struct ifnet *,
      int)} 0xffffffff802fae2d <fr_check_wrapper>
 (kgdb) p *ph
 $3 = {ph_in = {tqh_first = 0xfffffe0003007b40, tqh_last = 0xfffffe000581fb00},
    ph_out = {tqh_first = 0xffffffff80779733, tqh_last = 0x0},
    ph_type = 92404512, ph_nhooks = -512, ph_lock = {lock_object = {
        lo_name = 0xfffffe0003007ae0 "\201\005", lo_flags = 2155321139,
        lo_data = 4294967295, lo_witness = 0x0}, rm_writecpus = {__bits = {
          -2198926812672}}, rm_activeReaders = {lh_first = 0xfffffe0005bf9b00},
      _rm_lock = {_rm_lock_mtx = {lock_object = {
            lo_name = 0x1 <Address 0x1 out of bounds>, lo_flags = 0,
            lo_data = 0, lo_witness = 0xfffffe000589f800},
          mtx_lock = 18446741874779218176}, _rm_lock_sx = {lock_object = {
            lo_name = 0x1 <Address 0x1 out of bounds>, lo_flags = 0,
            lo_data = 0, lo_witness = 0xfffffe000589f800},
          sx_lock = 18446741874779218176}}}, ph_un = {phu_val = 0,
      phu_ptr = 0x0}, ph_list = {le_next = 0x0, le_prev = 0xfffffe000581f800}}
 (kgdb)
 
 #### kgdb.out_assert2
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPvV+BSPOsGF+KA+MRAudvAJ4kQQl4isOAkmVCvzcj1ipGEagbwACgkhhO
 Ib9Dfbm6bUJcUHS6yBcbrQU=
 =FnJL
 -----END PGP SIGNATURE-----
 --3469798045-489758976-1337810639=:24195--

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@FreeBSD.org, freebsd-pf@FreeBSD.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Thu, 24 May 2012 10:58:54 +0200 (CEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-664628730-1337849937=:89783
 Content-Type: TEXT/PLAIN; format=flowed; charset=ISO-8859-15
 Content-Transfer-Encoding: 8BIT
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Thu, 24 May 2012, Daniel Hartmeier wrote:
 
 > On Wed, May 23, 2012 at 10:10:04PM +0000, Joerg Pulz wrote:
 >
 >>  here is what i could get out.
 >>  I was unable to print *pfh and pfh->pfil_func, but i printed the other
 >>  two and *ph, maybe this helps.
 >
 > That looks corrupted: ph_type = 92404512, ph_nhooks = -512 makes no
 > sense to me.
 >
 > Can you go up one stack frame (to #11), which should be ip_output()
 >
 > 509     /* Run through list of hooks for output packets. */
 > 510     odst.s_addr = ip->ip_dst.s_addr;
 > 511     ASSERT_HOST_BYTE_ORDER(m);
 > 512     error = pfil_run_hooks(&V_inet_pfil_hook, &m, ifp, PFIL_OUT, inp);
 > 513     if (error != 0 || m == NULL)
 > 514             goto done;
 >
 > and there print V_inet_pfil_hook?
 
 Daniel,
 
 i can't print V_inet_pfil_hook: No symbol "V_inet_pfil_hook" in current 
 context.
 
 Meanwhile, the system was running over night with your second patch and 
 panic'ed in the morning, about 3 hours ago.
 
 I was able to print *ifp, *pfh, pfh->pfil_func, pf_check_out, 
 fr_check_wrapper and ipfw_check_hook.
 I couldn't print:
 *ph: Variable "ph" is not available.
 *m0: Cannot access memory at address 0xb000b0
 
 Below is the output.
 
 Kind regards
 Joerg
 
 #### kgdb.out_assert_new
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: ipfw_check_hook:281 ASSERT_HOST_BYTE_ORDER 45056 176
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic+0x182
 ipfw_check_hook() at ipfw_check_hook+0x511
 pfil_run_hooks() at pfil_run_hooks+0xf1
 ip_output() at ip_output+0x6de
 ip_forward() at ip_forward+0x19e
 ip_input() at ip_input+0x680
 swi_net() at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 Dumping 559 out of 4077 MB:..3%..12%..21%..32%..41%..52%..61%..72%..81%..92%
 
 Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_mirror.ko
 Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipmi.ko
 #0  doadump (textdump=0) at pcpu.h:224
 224		__asm("movq %%gs:0,%0" : "=r" (td));
 (kgdb) up 10
 #10 0xffffffff8077a144 in ipfw_check_hook (arg=)
      at /usr/src/sys/netinet/ipfw/ip_fw_pfil.c:281
 281			ASSERT_HOST_BYTE_ORDER(*m0);
 (kgdb) list
 276				FREE_PKT(*m0);
 277			*m0 = NULL;
 278		}
 279		if (*m0 && mtod(*m0, struct ip *)->ip_v == 4) {
 280			SET_HOST_IPLEN(mtod(*m0, struct ip *));
 281			ASSERT_HOST_BYTE_ORDER(*m0);
 282		}
 283		return ret;
 284	}
 285 
 (kgdb) p *ifp
 $1 = {if_softc = 0xffffff80007a9000, if_l2com = 0xfffffe000300b200,
    if_vnet = 0x0, if_link = {tqe_next = 0xfffffe0003002000,
      tqe_prev = 0xfffffe0003003818},
    if_xname = "bge0", '\0' <repeats 11 times>,
    if_dname = 0xfffffe00028f07d8 "bge", if_dunit = 0, if_refcount = 1,
    if_addrhead = {tqh_first = 0xfffffe000300a000,
      tqh_last = 0xfffffe000591a0b8}, if_pcount = 0, if_carp = 0x0,
    if_bpf = 0xfffffe00050d4680, if_index = 5, if_index_reserved = 0,
    if_vlantrunk = 0x0, if_flags = 34819, if_capabilities = 524443,
    if_capenable = 524443, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
      ifi_type = 6 '\006', ifi_physical = 0 '\0', ifi_addrlen = 6 '\006',
      ifi_hdrlen = 18 '\022', ifi_link_state = 2 '\002',
      ifi_spare_char1 = 0 '\0', ifi_spare_char2 = 0 '\0',
      ifi_datalen = 152 '\230', ifi_mtu = 1500, ifi_metric = 0,
      ifi_baudrate = 1000000000, ifi_ipackets = 221591, ifi_ierrors = 0,
      ifi_opackets = 3800, ifi_oerrors = 0, ifi_collisions = 0,
      ifi_ibytes = 18564820, ifi_obytes = 2351574, ifi_imcasts = 205582,
      ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 3,
      ifi_epoch = 1, ifi_lastchange = {tv_sec = 1337811753, tv_usec = 642476}},
    if_multiaddrs = {tqh_first = 0xfffffe0005915300,
      tqh_last = 0xfffffe00058d10c0}, if_amcount = 0,
    if_output = 0xffffffff8073da85 <ether_output>,
    if_input = 0xffffffff8073d05b <ether_input>,
    if_start = 0xffffffff803c32f7 <bge_start>,
    if_ioctl = 0xffffffff803c952a <bge_ioctl>,
    if_init = 0xffffffff803c94e4 <bge_init>,
    if_resolvemulti = 0xffffffff8073ca1d <ether_resolvemulti>,
    if_qflush = 0xffffffff80735842 <if_qflush>,
    if_transmit = 0xffffffff8073570e <if_transmit>, if_reassign = 0,
    if_home_vnet = 0x0, if_addr = 0xfffffe000300a000, if_llsoftc = 0x0,
    if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0,
      ifq_maxlen = 511, ifq_drops = 0, ifq_mtx = {lock_object = {
          lo_name = 0xfffffe0003001828 "bge0", lo_flags = 16973824, lo_data = 0,
          lo_witness = 0xffffff80006cf480}, mtx_lock = 4}, ifq_drv_head = 0x0,
      ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 511, altq_type = 0,
      altq_flags = 1, altq_disc = 0x0, altq_ifp = 0xfffffe0003001800,
      altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0,
      altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0},
    if_broadcastaddr = 0xffffffff80adb860 "", if_bridge = 0x0,
    if_label = 0x0, if_prefixhead = {tqh_first = 0x0,
      tqh_last = 0xfffffe0003001a78}, if_afdata = {0x0, 0x0, 0xfffffe0005821a20,
      0x0 <repeats 25 times>, 0xfffffe00058168c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, if_afdata_initialized = 2, if_afdata_lock = {
      lock_object = {lo_name = 0xffffffff80adaafa "if_afdata",
        lo_flags = 69402624, lo_data = 0, lo_witness = 0xffffff80006cf400},
      rw_lock = 1}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
      ta_priority = 0, ta_func = 0xffffffff80737ce9 <do_link_state_change>,
      ta_context = 0xfffffe0003001800}, if_addr_mtx = {lock_object = {
        lo_name = 0xffffffff80accbc0 "if_addr_mtx", lo_flags = 16973824,
        lo_data = 0, lo_witness = 0xffffff80006c8b80}, mtx_lock = 4},
    if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {
      tqh_first = 0xfffffe0003007b20, tqh_last = 0xfffffe0003007b28},
    if_pf_kif = 0xfffffe000588b400, if_lagg = 0x0, if_description = 0x0,
    if_fib = 0, if_alloctype = 6 '\006', if_cspare = "\000\000", if_ispare = {0,
      0, 0, 0}, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
 (kgdb) up
 #11 0xffffffff8074b53d in pfil_run_hooks (ph=) at /usr/src/sys/net/pfil.c:85
 85				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 (kgdb) list
 80		KASSERT(ph->ph_nhooks >= 0, ("Pfil hook count dropped < 0"));
 81		for (pfh = pfil_hook_get(dir, ph); pfh != NULL;
 82		     pfh = TAILQ_NEXT(pfh, pfil_link)) {
 83			if (pfh->pfil_func != NULL) {
 84				ASSERT_HOST_BYTE_ORDER(m);
 85				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 86				    inp);
 87				if (rv != 0 || m == NULL)
 88					break;
 89				ASSERT_HOST_BYTE_ORDER(m);
 (kgdb) p *pfh
 $2 = {pfil_link = {tqe_next = 0xfffffe00058c5980,
      tqe_prev = 0xfffffe0005821b00},
    pfil_func = 0xffffffff80779c33 <ipfw_check_hook>, pfil_arg = 0x0}
 (kgdb) p pfh->pfil_func
 $3 = (int (*)(void *, struct mbuf **, struct ifnet *, int, struct inpcb
       *)) 0xffffffff80779c33 <ipfw_check_hook>
 (kgdb) p pf_check_out
 $4 = {int (void *, struct mbuf **, struct ifnet *, int, struct inpcb
       *)} 0xffffffff8032d39a <pf_check_out>
 (kgdb) p fr_check_wrapper
 $5 = {int (void *, struct mbuf **, struct ifnet *,
      int)} 0xffffffff802fc303 <fr_check_wrapper>
 (kgdb) p ipfw_check_hook
 $6 = {int (void *, struct mbuf **, struct ifnet *, int, struct inpcb
       *)} 0xffffffff80779c33 <ipfw_check_hook>
 (kgdb)
 
 #### kgdb.out_assert_new
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPvfhRSPOsGF+KA+MRAlM/AKClrSdzDyqSgechCL/RKRtj6KHpVQCfQtCL
 PQk+XB5xpajaVmaGba7wD7s=
 =J22z
 -----END PGP SIGNATURE-----
 --3469798045-664628730-1337849937=:89783--

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@FreeBSD.org, freebsd-pf@FreeBSD.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Thu, 24 May 2012 15:50:04 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Thu, 24 May 2012, Daniel Hartmeier wrote:
 
 > On Thu, May 24, 2012 at 09:10:04AM +0000, Joerg Pulz wrote:
 >
 >>  panic: ipfw_check_hook:281 ASSERT_HOST_BYTE_ORDER 45056 176
 >>  ipfw_check_hook() at ipfw_check_hook+0x511
 >>  pfil_run_hooks() at pfil_run_hooks+0xf1
 >>  ip_output() at ip_output+0x6de
 >>  ip_forward() at ip_forward+0x19e
 >>  ip_input() at ip_input+0x680
 >>  swi_net() at swi_net+0x15a
 >
 > OK, this convinces me that the problem is in ipfw.
 >
 > You enabled it with
 >
 > options         IPFIREWALL
 > options         IPFIREWALL_VERBOSE
 > options         IPFIREWALL_VERBOSE_LIMIT=100
 > options         IPFIREWALL_DEFAULT_TO_ACCEPT
 >
 > but say you're not using it?
 >
 > The above will actually enable ipfw's packet inspection with a default
 > pass rule. And a non-trivial amount of code runs, unlike pf (and
 > ipfilter), which must first be enabled (like with pfctl -e) first.
 >
 > Could you rebuild a kernel without the above options, just to confirm
 > the theory that the problem is related to ipfw?
 >
 > We can try to find the problem within ipfw, maybe asking the ipfw
 > developers for help.
 
 Daniel,
 
 exactly, ipfw was enabled with the above kernel options but not configured 
 to filter or do anything but the DEFAULT_TO_ACCEPT.
 I've rebuilt the kernel without IPFIREWALL options. The system is running 
 now for about three and a half hours.
 Time will show if this solved our problem.
 I'm still wondering why these panics showed up in irregular unreproducable 
 intervals.
 
 Thanks for writing to the ipfw list. I'm really interested in tracking 
 this further down to fix it forever, so nobody will stumble over it again.
 
 Thanks for all your help. Feel free to contact me if you have new ideas or 
 things i should try.
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPvjyPSPOsGF+KA+MRAqgqAJ0Z8uuoOLHpbEcUTSrg1oXgNu7sowCfem2Z
 r8rPTyO39GMo9qJa10z+zzM=
 =pq7s
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Fri, 25 May 2012 09:25:38 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Thu, 24 May 2012, Joerg Pulz wrote:
 
 > Daniel,
 >
 > exactly, ipfw was enabled with the above kernel options but not configured
 > to filter or do anything but the DEFAULT_TO_ACCEPT.
 > I've rebuilt the kernel without IPFIREWALL options. The system is running
 > now for about three and a half hours.
 > Time will show if this solved our problem.
 > I'm still wondering why these panics showed up in irregular unreproducable
 > intervals.
 >
 > Thanks for writing to the ipfw list. I'm really interested in tracking
 > this further down to fix it forever, so nobody will stumble over it again.
 >
 > Thanks for all your help. Feel free to contact me if you have new ideas or
 > things i should try.
 
 Daniel,
 
 the system is still running without panic, but i found the following log 
 entries from last night:
 
 May 24 23:28:57 charon kernel: pf_route: m0->m_len < sizeof(struct ip)
 May 24 23:28:57 charon kernel: pf_route: m0->m_len < sizeof(struct ip)
 
 Do you think that this may be related to the panics?
 I've found this error message two times in contrib/pf/net/pf.c.
 I can't say which of them or both have printed the message.
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPvzP1SPOsGF+KA+MRAngoAJ4wk4PSjEtYvpCak2H8Qze8GaUbfwCgg2dq
 2sQgy+3qWttRKxCj/WctPvY=
 =ejhQ
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Fri, 25 May 2012 13:25:32 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Fri, 25 May 2012, Daniel Hartmeier wrote:
 
 > On Fri, May 25, 2012 at 07:30:16AM +0000, Joerg Pulz wrote:
 >
 >>  the system is still running without panic, but i found the following log
 >>  entries from last night:
 >>
 >>  May 24 23:28:57 charon kernel: pf_route: m0->m_len < sizeof(struct ip)
 >>  May 24 23:28:57 charon kernel: pf_route: m0->m_len < sizeof(struct ip)
 >>
 >>  Do you think that this may be related to the panics?
 >>  I've found this error message two times in contrib/pf/net/pf.c.
 >>  I can't say which of them or both have printed the message.
 >
 > Yes, this could possibly explain it.
 >
 > All pfil consumers assume that the IP header is one continuous memory
 > region in the mbuf, without verifying this or correcting it (with
 > m_pullup() or such) if wrong.
 >
 > pf: pf_check_in() in sys/contrib/pf/net/pf_ioctl.c
 > 	h = mtod(*m, struct ip *);
 > 	access h->ip_len
 >
 > ipfw: ipfw_check_hook() in sys/netinet/ipfw/ip_fw_pfil.c
 > 	SET_NET_IPLEN(mtod(*m0, struct ip *));
 >
 > ipfilter: fr_check_wrapper() in sys/contrib/ipfilter/netinet/ip_fil_freebsd.c
 > 	struct ip *ip = mtod(*mp, struct ip *);
 > 	access ip->ip_hl
 >
 > Hence, the caller of pfil_run_hooks() must ensure this before the call.
 > ip_input() does, but there are operations that might violate the
 > condition again.
 >
 > If the IP header is not continuous in the first mbuf, doing
 >
 > 	struct ip *ip = mtod(m, struct ip *);
 > 	ip->ip_len = ntohs(ip->ip_len);
 >
 > will read (and write) unrelated memory.
 >
 > If, later, something does call m_pullup(), ip_len will get 'replaced'
 > again. This could lead to some byte order swaps getting undone.
 >
 > I'm not sure what the proper action is here, i.e. should we be surprised
 > that an mbuf with such a small m_len is found, and track down how it
 > was produced, or should the pfil code simply expect this?
 >
 > I'd probably add a sanity check to pfil_run_hooks(), like
 >
 > --- sys/net/pfil.c      23 Sep 2011 00:51:37 -0000      1.19.2.1
 > +++ sys/net/pfil.c      25 May 2012 09:10:27 -0000
 > @@ -46,6 +46,8 @@
 >
 > #include <net/if.h>
 > #include <net/pfil.h>
 > +#include <netinet/in.h>
 > +#include <netinet/ip.h>
 >
 > static struct mtx pfil_global_lock;
 >
 > @@ -74,15 +76,21 @@
 >        struct mbuf *m = *mp;
 >        int rv = 0;
 >
 > +       if (m->m_pkthdr.len < sizeof(struct ip) ||
 > +           m->m_len < sizeof(struct ip))
 > +               panic("pfil_run_hooks: m->m_pkthdr.len %d, m->m_len %d",
 > +                   (int)m->m_pkthdr.len, (int)m->m_len);
 >        PFIL_RLOCK(ph, &rmpt);
 >        KASSERT(ph->ph_nhooks >= 0, ("Pfil hook count dropped < 0"));
 >        for (pfh = pfil_hook_get(dir, ph); pfh != NULL;
 >             pfh = TAILQ_NEXT(pfh, pfil_link)) {
 >                if (pfh->pfil_func != NULL) {
 > +                       ASSERT_HOST_BYTE_ORDER(m);
 >                        rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 >                            inp);
 >                        if (rv != 0 || m == NULL)
 >                                break;
 > +                       ASSERT_HOST_BYTE_ORDER(m);
 >                }
 >        }
 >        PFIL_RUNLOCK(ph, &rmpt);
 >
 > Then when it will panic (instead of just the pf_route() message), the
 > stack trace could help.
 >
 > This might require several iterations, adding such checks all around the
 > place. The cause might be some mbuf operation done in ipsec, for some
 > edge case, explaining why it occurs so rarely...
 >
 > If I could easily reproduce it locally, I'd probably do it, but it's
 > your machine that crashes all the time, so it's your call :)
 
 Daniel,
 
 thanks for your explaining.
 
 As i said, i will do everything to track this down to the real bottom to 
 get it fixed forever.
 
 Your proposed changes are in and the system is running with the rebuilt 
 kernel. Now it's time to wait and see what happens.
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPv2wvSPOsGF+KA+MRAp1DAKCp2KZdPKBdsR5PUbK3ixqXqFdi9ACfbfvd
 vez+bq9X5lMjbdysCXy+stU=
 =tvtF
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: freebsd-pf@freebsd.org, bug-followup@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad fragment handling?)
Date: Sun, 27 May 2012 20:27:58 +0200

 Daniel Hartmeier <daniel@benzedrine.cx> wrote:
 
 >On Fri, May 25, 2012 at 07:30:16AM +0000, Joerg Pulz wrote:
 >
 >>  the system is still running without panic, but i found the following
 >log 
 >>  entries from last night:
 >>  
 >>  May 24 23:28:57 charon kernel: pf_route: m0->m_len < sizeof(struct
 >ip)
 >>  May 24 23:28:57 charon kernel: pf_route: m0->m_len < sizeof(struct
 >ip)
 >>  
 >>  Do you think that this may be related to the panics?
 >>  I've found this error message two times in contrib/pf/net/pf.c.
 >>  I can't say which of them or both have printed the message.
 >
 >Yes, this could possibly explain it.
 >
 >All pfil consumers assume that the IP header is one continuous memory
 >region in the mbuf, without verifying this or correcting it (with
 >m_pullup() or such) if wrong.
 >
 >pf: pf_check_in() in sys/contrib/pf/net/pf_ioctl.c
 >	h = mtod(*m, struct ip *);
 >	access h->ip_len
 >
 >ipfw: ipfw_check_hook() in sys/netinet/ipfw/ip_fw_pfil.c
 >	SET_NET_IPLEN(mtod(*m0, struct ip *));
 >
 >ipfilter: fr_check_wrapper() in
 >sys/contrib/ipfilter/netinet/ip_fil_freebsd.c
 >	struct ip *ip = mtod(*mp, struct ip *);
 >	access ip->ip_hl
 >
 >Hence, the caller of pfil_run_hooks() must ensure this before the call.
 >ip_input() does, but there are operations that might violate the
 >condition again.
 >
 >If the IP header is not continuous in the first mbuf, doing
 >
 >	struct ip *ip = mtod(m, struct ip *);
 >	ip->ip_len = ntohs(ip->ip_len);
 >
 >will read (and write) unrelated memory.
 >
 >If, later, something does call m_pullup(), ip_len will get 'replaced'
 >again. This could lead to some byte order swaps getting undone.
 >
 >I'm not sure what the proper action is here, i.e. should we be
 >surprised
 >that an mbuf with such a small m_len is found, and track down how it
 >was produced, or should the pfil code simply expect this?
 >
 >I'd probably add a sanity check to pfil_run_hooks(), like
 >
 >--- sys/net/pfil.c      23 Sep 2011 00:51:37 -0000      1.19.2.1
 >+++ sys/net/pfil.c      25 May 2012 09:10:27 -0000
 >@@ -46,6 +46,8 @@
 >
 > #include <net/if.h>
 > #include <net/pfil.h>
 >+#include <netinet/in.h>
 >+#include <netinet/ip.h>
 >
 > static struct mtx pfil_global_lock;
 >
 >@@ -74,15 +76,21 @@
 >        struct mbuf *m = *mp;
 >        int rv = 0;
 >
 >+       if (m->m_pkthdr.len < sizeof(struct ip) ||
 >+           m->m_len < sizeof(struct ip))
 >+               panic("pfil_run_hooks: m->m_pkthdr.len %d, m->m_len
 >%d",
 >+                   (int)m->m_pkthdr.len, (int)m->m_len);
 >        PFIL_RLOCK(ph, &rmpt);
 >        KASSERT(ph->ph_nhooks >= 0, ("Pfil hook count dropped < 0"));
 >        for (pfh = pfil_hook_get(dir, ph); pfh != NULL;
 >             pfh = TAILQ_NEXT(pfh, pfil_link)) {
 >                if (pfh->pfil_func != NULL) {
 >+                       ASSERT_HOST_BYTE_ORDER(m);
 >                    rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 >                            inp);
 >                        if (rv != 0 || m == NULL)
 >                                break;
 >+                       ASSERT_HOST_BYTE_ORDER(m);
 >                }
 >        }
 >        PFIL_RUNLOCK(ph, &rmpt);
 >
 >Then when it will panic (instead of just the pf_route() message), the
 >stack trace could help.
 >
 >This might require several iterations, adding such checks all around
 >the
 >place. The cause might be some mbuf operation done in ipsec, for some
 >edge case, explaining why it occurs so rarely...
 >
 >If I could easily reproduce it locally, I'd probably do it, but it's
 >your machine that crashes all the time, so it's your call :)
 >
 >Daniel
 
 Daniel,
 
 i've seen 12 more "pf_route: m0->m_len < sizeof(struct ip)" messages since the system is running after adding your patch, but no panic.
 Is there another place in the code where i can add an additional check?
 
 Kind regards
 Jörg

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Fri, 1 Jun 2012 10:25:39 +0200 (CEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-380680488-1338539143=:89783
 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 8BIT
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Tue, 29 May 2012, Daniel Hartmeier wrote:
 
 > On Sun, May 27, 2012 at 06:30:09PM +0000, Joerg Pulz wrote:
 >
 >>  i've seen 12 more "pf_route: m0->m_len < sizeof(struct ip)" messages since the system is running after adding your patch, but no panic.
 >>  Is there another place in the code where i can add an additional check?
 >
 > Yes, the following patch adds more checks to pf.
 
 Daniel,
 
 after several days waiting for a panic since i applied your new patch, it 
 finally happend last night.
 
 Below is the kgdb(1) output. I tried to print as much as possible to give 
 you the most informations.
 
 Hope this helps to find the cuase of the trouble or at least to get a bit 
 closer.
 
 #### kgdb.out_len
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: pf_test: 1: m->m_pkthdr.len 176, m->m_len 0
 cpuid = 1
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic+0x182
 pf_test() at pf_test+0x259
 pf_check_out() at pf_check_out+0x71
 pfil_run_hooks() at pfil_run_hooks+0x113
 ip_output() at ip_output+0x6de
 ip_forward() at ip_forward+0x19e
 ip_input() at ip_input+0x680
 swi_net() at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 Dumping 588 out of 4077 MB:..3%..11%..22%..33%..41%..52%..63%..71%..82%..93%
 
 Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_mirror.ko
 Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipmi.ko
 #0  doadump (textdump=0) at pcpu.h:224
 224		__asm("movq %%gs:0,%0" : "=r" (td));
 (kgdb) up 10
 #10 0xffffffff80326737 in pf_test (dir=2, ifp=0xfffffe0003001800,
      m0=0xffffff80002418e8, eh=0x0, inp=0x0)
      at /usr/src/sys/contrib/pf/net/pf.c:6725
 6725			panic("pf_test: 1: m->m_pkthdr.len %d, m->m_len %d",
 (kgdb) list
 6720			goto done;
 6721		}
 6722 
 6723		if (m->m_pkthdr.len < sizeof(struct ip) ||
 6724		    m->m_len < sizeof(struct ip))
 6725			panic("pf_test: 1: m->m_pkthdr.len %d, m->m_len %d",
 6726			    (int)m->m_pkthdr.len, (int)m->m_len);
 6727 
 6728	#ifdef __FreeBSD__
 6729		if (m->m_flags & M_SKIP_FIREWALL) {
 (kgdb) p *m
 $1 = {m_hdr = {mh_next = 0xfffffe01671a0700, mh_nextpkt = 0x0,
      mh_data = 0xfffffe0064823774 "E", mh_len = 0, mh_flags = 66, mh_type = 1,
      pad = ""}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
          header = 0x0, len = 176, flowid = 0, csum_flags = 768,
          csum_data = 16922, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
          tags = {slh_first = 0xfffffe00644820a0}}, MH_dat = {MH_ext = {
            ext_buf = 0x38200ec0045 <Address 0x38200ec0045 out of bounds>,
            ext_free = 0x38200b00045, ext_arg1 = 0xd7d59754b1600478,
            ext_arg2 = 0xb000004557b3bb81, ext_size = 62620,
            ref_cnt = 0x1b2a8c002079b0a, ext_type = -1242365181},
          MH_databuf = "E\000\000\202\003\000\000E\000\000\202\003\000\000x\004`T\227\201WE\000\000\234\000\000\177\001\031\022\n\233\a\002\001\003\003\000\000\000\000E\000\000\235&\000\000>\021\r\001\n\233\a\002\0005A\000\211\203\016\212\205\200\000\001\000\001\000\002\000\002"}},
      M_databuf = "\000\030\000\003\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\003\000\000\032B\000\000\000\000\000\000ޠ Hd\000E\000\000\202\003\000\000E\000\000\202\003\000\000x\004`T\227\201WE\000\000\234\000\000\177\001\031\022\n\233\a\002\001\003\003\000\000\000\000E\000\000\235&\000\000>\021\r\001\n\233\a\002\0005A\000\211\203\016\212\205\200\000\001\000\001\000\002\000\002"...}}
 (kgdb) p *ifp
 $2 = {if_softc = 0xffffff80007a9000, if_l2com = 0xfffffe000300b200,
    if_vnet = 0x0, if_link = {tqe_next = 0xfffffe0003002000,
      tqe_prev = 0xfffffe0003003818},
    if_xname = "bge0", '\0' <repeats 11 times>,
    if_dname = 0xfffffe00028f0758 "bge", if_dunit = 0, if_refcount = 1,
    if_addrhead = {tqh_first = 0xfffffe000300a000,
      tqh_last = 0xfffffe0005a940b8}, if_pcount = 0, if_carp = 0x0,
    if_bpf = 0xfffffe0005062400, if_index = 5, if_index_reserved = 0,
    if_vlantrunk = 0x0, if_flags = 34819, if_capabilities = 524443,
    if_capenable = 524443, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
      ifi_type = 6 '\006', ifi_physical = 0 '\0', ifi_addrlen = 6 '\006',
      ifi_hdrlen = 18 '\022', ifi_link_state = 2 '\002',
      ifi_spare_char1 = 0 '\0', ifi_spare_char2 = 0 '\0',
      ifi_datalen = 152 '\230', ifi_mtu = 1500, ifi_metric = 0,
      ifi_baudrate = 1000000000, ifi_ipackets = 4678659, ifi_ierrors = 0,
      ifi_opackets = 2594069, ifi_oerrors = 0, ifi_collisions = 0,
      ifi_ibytes = 598927432, ifi_obytes = 2837994361, ifi_imcasts = 2432290,
      ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 3,
      ifi_epoch = 1, ifi_lastchange = {tv_sec = 1338284854, tv_usec = 622823}},
    if_multiaddrs = {tqh_first = 0xfffffe0005bdb080,
      tqh_last = 0xfffffe00058ff080}, if_amcount = 0,
    if_output = 0xffffffff8073d2f5 <ether_output>,
    if_input = 0xffffffff8073c8cb <ether_input>,
    if_start = 0xffffffff803c2b67 <bge_start>,
    if_ioctl = 0xffffffff803c8d9a <bge_ioctl>,
    if_init = 0xffffffff803c8d54 <bge_init>,
    if_resolvemulti = 0xffffffff8073c28d <ether_resolvemulti>,
    if_qflush = 0xffffffff807350b2 <if_qflush>,
    if_transmit = 0xffffffff80734f7e <if_transmit>, if_reassign = 0,
    if_home_vnet = 0x0, if_addr = 0xfffffe000300a000, if_llsoftc = 0x0,
    if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0,
      ifq_maxlen = 511, ifq_drops = 0, ifq_mtx = {lock_object = {
          lo_name = 0xfffffe0003001828 "bge0", lo_flags = 16973824, lo_data = 0,
          lo_witness = 0xffffff80006cf480}, mtx_lock = 4}, ifq_drv_head = 0x0,
      ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 511, altq_type = 0,
      altq_flags = 1, altq_disc = 0x0, altq_ifp = 0xfffffe0003001800,
      altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0,
      altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0},
    if_broadcastaddr = 0xffffffff80ad06c0 "", if_bridge = 0x0,
    if_label = 0x0, if_prefixhead = {tqh_first = 0x0,
      tqh_last = 0xfffffe0003001a78}, if_afdata = {0x0, 0x0, 0xfffffe0005821a20,
      0x0 <repeats 25 times>, 0xfffffe0005815940, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, if_afdata_initialized = 2, if_afdata_lock = {
      lock_object = {lo_name = 0xffffffff80acf95a "if_afdata",
        lo_flags = 69402624, lo_data = 0, lo_witness = 0xffffff80006cf400},
      rw_lock = 1}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
      ta_priority = 0, ta_func = 0xffffffff80737559 <do_link_state_change>,
      ta_context = 0xfffffe0003001800}, if_addr_mtx = {lock_object = {
        lo_name = 0xffffffff80ac1a20 "if_addr_mtx", lo_flags = 16973824,
        lo_data = 0, lo_witness = 0xffffff80006c8b80}, mtx_lock = 4},
    if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {
      tqh_first = 0xfffffe0003007b20, tqh_last = 0xfffffe0003007b28},
    if_pf_kif = 0xfffffe0005888400, if_lagg = 0x0, if_description = 0x0,
    if_fib = 0, if_alloctype = 6 '\006', if_cspare = "\000\000", if_ispare = {0,
      0, 0, 0}, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
 (kgdb) p pd
 $3 = {lookup = {done = 0, uid = 0, gid = 0, pid = 0}, tot_len = 0, hdr = {
      tcp = 0x0, udp = 0x0, icmp = 0x0, icmp6 = 0x0, any = 0x0}, nat_rule = 0x0,
    eh = 0x0, src = 0x0, dst = 0x0, sport = 0x0, dport = 0x0,
    pf_mtag = 0xfffffe00644f9358, p_len = 0, ip_sum = 0x0, proto_sum = 0x0,
    flags = 0, af = 0 '\0', proto = 0 '\0', tos = 0 '\0', dir = 0 '\0',
    sidx = 0 '\0', didx = 0 '\0'}
 (kgdb) p pf_status
 $4 = {counters = {9415424, 0, 0, 0, 0, 0, 0, 0, 3464, 0, 27, 0, 0, 0, 0},
    lcounters = {0, 0, 0, 0, 0, 0, 0}, fcounters = {12630228, 74172, 74158},
    scounters = {0, 0, 0}, pcounters = {{{0, 0, 0}, {0, 0, 0}}, {{0, 0, 0}, {0,
          0, 0}}}, bcounters = {{0, 0}, {0, 0}}, stateid = 5747889684957176252,
    running = 1, states = 14, src_nodes = 0, since = 1338284855, debug = 1,
    hostid = 3046117155, ifname = '\0' <repeats 15 times>,
    pf_chksum = "qu\205<0h\021vi\203"}
 (kgdb) p pf_status.running
 $5 = 1
 (kgdb) up
 #11 0xffffffff8032cc7b in pf_check_out (arg=)
      at /usr/src/sys/contrib/pf/net/pf_ioctl.c:4184
 4184		chk = pf_test(PF_OUT, ifp, m, NULL, inp);
 (kgdb) list
 4179			h = mtod(*m, struct ip *);
 4180			HTONS(h->ip_len);
 4181			HTONS(h->ip_off);
 4182		}
 4183		CURVNET_SET(ifp->if_vnet);
 4184		chk = pf_test(PF_OUT, ifp, m, NULL, inp);
 4185		CURVNET_RESTORE();
 4186		if (chk && *m) {
 4187			m_freem(*m);
 4188			*m = NULL;
 (kgdb) up
 #12 0xffffffff8074adcf in pfil_run_hooks (ph=) at /usr/src/sys/net/pfil.c:89
 89				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 (kgdb) list
 84		KASSERT(ph->ph_nhooks >= 0, ("Pfil hook count dropped < 0"));
 85		for (pfh = pfil_hook_get(dir, ph); pfh != NULL;
 86		     pfh = TAILQ_NEXT(pfh, pfil_link)) {
 87			if (pfh->pfil_func != NULL) {
 88				ASSERT_HOST_BYTE_ORDER(m);
 89				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 90				    inp);
 91				if (rv != 0 || m == NULL)
 92					break;
 93				ASSERT_HOST_BYTE_ORDER(m);
 (kgdb) p *pfh
 $6 = {pfil_link = {tqe_next = 0x0, tqe_prev = 0xfffffe0005821b00},
    pfil_func = 0xffffffff8032cc0a <pf_check_out>, pfil_arg = 0x0}
 (kgdb) up
 #13 0xffffffff80776b3a in ip_output (m=0xfffffe0064823700, opt=)
      at /usr/src/sys/netinet/ip_output.c:512
 512		error = pfil_run_hooks(&V_inet_pfil_hook, &m, ifp, PFIL_OUT, inp);
 (kgdb) list
 507			goto passout;
 508 
 509		/* Run through list of hooks for output packets. */
 510		odst.s_addr = ip->ip_dst.s_addr;
 511		ASSERT_HOST_BYTE_ORDER(m);
 512		error = pfil_run_hooks(&V_inet_pfil_hook, &m, ifp, PFIL_OUT, inp);
 513		if (error != 0 || m == NULL)
 514			goto done;
 515 
 516		ip = mtod(m, struct ip *);
 (kgdb) p *ip
 $7 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 45056,
    ip_id = 62620, ip_off = 0, ip_ttl = 127 '\177', ip_p = 1 '\001',
    ip_sum = 4633, ip_src = {s_addr = 34052874}, ip_dst = {s_addr = 28485824}}
 (kgdb) p flags
 $8 = 1
 (kgdb) p mtu
 $9 = 1500
 (kgdb) p *ia
 $10 = {ia_ifa = {ifa_addr = 0xfffffe0005a09338,
      ifa_dstaddr = 0xfffffe0005a09348, ifa_netmask = 0xfffffe0005a09358,
      if_data = {ifi_type = 0 '\0', ifi_physical = 0 '\0', ifi_addrlen = 0 '\0',
        ifi_hdrlen = 0 '\0', ifi_link_state = 0 '\0', ifi_spare_char1 = 0 '\0',
        ifi_spare_char2 = 0 '\0', ifi_datalen = 0 '\0', ifi_mtu = 0,
        ifi_metric = 0, ifi_baudrate = 0, ifi_ipackets = 4447700,
        ifi_ierrors = 0, ifi_opackets = 2591860, ifi_oerrors = 0,
        ifi_collisions = 0, ifi_ibytes = 608432458, ifi_obytes = 2801425920,
        ifi_imcasts = 0, ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0,
        ifi_hwassist = 0, ifi_epoch = 0, ifi_lastchange = {tv_sec = 0,
          tv_usec = 0}}, ifa_ifp = 0xfffffe0003001800, ifa_link = {
        tqe_next = 0xfffffe0005a94000, tqe_prev = 0xfffffe000300a0b8},
      ifa_rtrequest = 0, ifa_flags = 5, ifa_refcnt = 6, ifa_metric = 0,
      ifa_claim_addr = 0, ifa_mtx = {lock_object = {
          lo_name = 0xffffffff80ad4634 "ifaddr", lo_flags = 16973824,
          lo_data = 0, lo_witness = 0xffffff80006c8980}, mtx_lock = 4}},
    ia_subnet = 2176561920, ia_subnetmask = 4294967040, ia_hash = {
      le_next = 0x0, le_prev = 0xfffffe000587f8c8}, ia_link = {
      tqe_next = 0xfffffe0005902c00, tqe_prev = 0xfffffe0005902928}, ia_addr = {
      sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 0, sin_addr = {
        s_addr = 1471396737}, sin_zero = "\000\000\000\000\000\000\000"},
    ia_dstaddr = {sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 0,
      sin_addr = {s_addr = 4289969025},
      sin_zero = "\000\000\000\000\000\000\000"}, ia_sockmask = {
      sin_len = 7 '\a', sin_family = 2 '\002', sin_port = 0, sin_addr = {
        s_addr = 16777215}, sin_zero = "\000\000\000\000\000\000\000"}}
 (kgdb) p *dst
 $11 = {sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 0, sin_addr = {
      s_addr = 4273191809}, sin_zero = "\000\000\000\000\000\000\000"}
 (kgdb)
 
 #### kgdb.out_len
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPyHyGSPOsGF+KA+MRAmr4AJ91yi1whfweG8Dkue7zi0Lvcsdn4gCfScX0
 L8tV5u5gLMelsZX43e6yo6M=
 =VzIz
 -----END PGP SIGNATURE-----
 --3469798045-380680488-1338539143=:89783--

From: =?ISO-8859-1?Q?Ermal_Lu=E7i?= <eri@freebsd.org>
To: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
Cc: Daniel Hartmeier <daniel@benzedrine.cx>, bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Fri, 1 Jun 2012 15:21:09 +0200

 On Fri, Jun 1, 2012 at 10:25 AM, Joerg Pulz <Joerg.Pulz@frm2.tum.de> wrote:
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA1
 >
 >
 > On Tue, 29 May 2012, Daniel Hartmeier wrote:
 >
 >> On Sun, May 27, 2012 at 06:30:09PM +0000, Joerg Pulz wrote:
 >>
 >>> =C2=A0i've seen 12 more "pf_route: m0->m_len < sizeof(struct ip)" messa=
 ges
 >>> since the system is running after adding your patch, but no panic.
 >>> =C2=A0Is there another place in the code where i can add an additional =
 check?
 >>
 >>
 >> Yes, the following patch adds more checks to pf.
 >
 >
 > Daniel,
 >
 > after several days waiting for a panic since i applied your new patch, it
 > finally happend last night.
 >
 > Below is the kgdb(1) output. I tried to print as much as possible to give
 > you the most informations.
 >
 > Hope this helps to find the cuase of the trouble or at least to get a bit
 > closer.
 >
 > #### kgdb.out_len
 >
 >
 > GNU gdb 6.1.1 [FreeBSD]
 > Copyright 2004 Free Software Foundation, Inc.
 > GDB is free software, covered by the GNU General Public License, and you =
 are
 > welcome to change it and/or distribute copies of it under certain
 > conditions.
 > Type "show copying" to see the conditions.
 > There is absolutely no warranty for GDB. =C2=A0Type "show warranty" for d=
 etails.
 > This GDB was configured as "amd64-marcel-freebsd"...
 >
 > Unread portion of the kernel message buffer:
 > panic: pf_test: 1: m->m_pkthdr.len 176, m->m_len 0
 >
 > cpuid =3D 1
 > KDB: stack backtrace:
 > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 > kdb_backtrace() at kdb_backtrace+0x37
 > panic() at panic+0x182
 > pf_test() at pf_test+0x259
 > pf_check_out() at pf_check_out+0x71
 > pfil_run_hooks() at pfil_run_hooks+0x113
 >
 > ip_output() at ip_output+0x6de
 > ip_forward() at ip_forward+0x19e
 
 It is quite strange that you do not have a pfil_run_hooks() here as well!
 Maybe you are running ipsec, if so i would expect that to show in the trace=
 !?
 
 Can you describe the setup you have in more detail to understand what
 interactions are happening with the stack?
 
 > ip_input() at ip_input+0x680
 > swi_net() at swi_net+0x15a
 > intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 > ithread_loop() at ithread_loop+0xaf
 > fork_exit() at fork_exit+0x12a
 > fork_trampoline() at fork_trampoline+0xe
 > - --- trap 0, rip =3D 0, rsp =3D 0xffffff8000241d00, rbp =3D 0 ---
 > KDB: enter: panic
 > Dumping 588 out of 4077 MB:..3%..11%..22%..33%..41%..52%..63%..71%..82%..=
 93%
 >
 >
 > Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
 > /boot/kernel/geom_mirror.ko.symbols...done.
 > done.
 > Loaded symbols for /boot/kernel/geom_mirror.ko
 > Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
 > /boot/kernel/ipmi.ko.symbols...done.
 > done.
 > Loaded symbols for /boot/kernel/ipmi.ko
 > #0 =C2=A0doadump (textdump=3D0) at pcpu.h:224
 > 224 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 __asm("movq %%gs:0,%0" : "=
 =3Dr" (td));
 > (kgdb) up 10
 > #10 0xffffffff80326737 in pf_test (dir=3D2, ifp=3D0xfffffe0003001800,
 > =C2=A0 =C2=A0m0=3D0xffffff80002418e8, eh=3D0x0, inp=3D0x0)
 > =C2=A0 =C2=A0at /usr/src/sys/contrib/pf/net/pf.c:6725
 > 6725 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0panic("pf_test: 1: m->m_pkthdr.len %d, m->m_len %d",
 > (kgdb) list
 > 6720 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0goto done;
 > 6721 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
 > 6722 6723 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (m->m_pkthd=
 r.len < sizeof(struct ip) ||
 > 6724 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0m->m_len < si=
 zeof(struct ip))
 > 6725 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0panic("pf_test: 1: m->m_pkthdr.len %d, m->m_len %d",
 > 6726 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0 =C2=A0 =C2=A0(int)m->m_pkthdr.len, (int)m->m_len);
 > 6727 6728 =C2=A0 =C2=A0 =C2=A0 #ifdef __FreeBSD__
 > 6729 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (m->m_flags & M_SKIP_FIR=
 EWALL) {
 > (kgdb) p *m
 > $1 =3D {m_hdr =3D {mh_next =3D 0xfffffe01671a0700, mh_nextpkt =3D 0x0,
 > =C2=A0 =C2=A0mh_data =3D 0xfffffe0064823774 "E", mh_len =3D 0, mh_flags =
 =3D 66, mh_type =3D 1,
 >
 > =C2=A0 =C2=A0pad =3D "=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E"}, M_dat =3D {=
 MH =3D {MH_pkthdr =3D {rcvif =3D 0xfffffe0003001800,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0header =3D 0x0, len =3D 176, flowid =3D 0, csu=
 m_flags =3D 768,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0csum_data =3D 16922, tso_segsz =3D 0, PH_vt =
 =3D {vt_vtag =3D 0, vt_nrecs =3D
 > 0},
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0tags =3D {slh_first =3D 0xfffffe00644820a0}}, =
 MH_dat =3D {MH_ext =3D {
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ext_buf =3D 0x38200ec0045 <Address 0x38=
 200ec0045 out of bounds>,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ext_free =3D 0x38200b00045, ext_arg1 =
 =3D 0xd7d59754b1600478,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ext_arg2 =3D 0xb000004557b3bb81, ext_si=
 ze =3D 62620,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ref_cnt =3D 0x1b2a8c002079b0a, ext_type=
  =3D -1242365181},
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0MH_databuf =3D
 > "E\000=C4=97\000\202\003\000\000E\000=C2=B0\000\202\003\000\000x\004`=C4=
 =85T\227=C3=95=C5=A8\201=C5=A7=C4=A3WE\000\000=C2=B0\234=C3=B4\000\000\177\=
 001\031\022\n\233\a\002=C4=80=C4=BB=C4=93\001\003\003=C3=B3=C4=A9\000\000\0=
 00\000E\000\000\235&=C3=BC\000\000>\021=C5=85\r=C4=80=C4=BB=C4=93\001\n\233=
 \a\002\0005=C3=85A\000\211\203\016=C5=86\212\205\200\000\001\000\001\000\00=
 2\000\002=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=
 =C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=
 =9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=
 =C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=
 =9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=
 =C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E"}},
 > =C2=A0 =C2=A0M_databuf =3D
 > "\000\030\000\003\000=C3=BE=C4=B8=C4=B8\000\000\000\000\000\000\000\000=
 =C2=B0\000\000\000\000\000\000\000\000\003\000\000\032B\000\000\000\000\000=
 \000=C3=9E=C4=80=C2=AD=C3=9E
 > Hd\000=C3=BE=C4=B8=C4=B8E\000=C4=97\000\202\003\000\000E\000=C2=B0\000\20=
 2\003\000\000x\004`=C4=85T\227=C3=95=C5=A8\201=C5=A7=C4=A3WE\000\000=C2=B0\=
 234=C3=B4\000\000\177\001\031\022\n\233\a\002=C4=80=C4=BB=C4=93\001\003\003=
 =C3=B3=C4=A9\000\000\000\000E\000\000\235&=C3=BC\000\000>\021=C5=85\r=C4=80=
 =C4=BB=C4=93\001\n\233\a\002\0005=C3=85A\000\211\203\016=C5=86\212\205\200\=
 000\001\000\001\000\002\000\002=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=
 =C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=
 =9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=
 =C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=
 =9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E=C3=9E=C4=80=C2=AD=C3=9E"...}}
 > (kgdb) p *ifp
 > $2 =3D {if_softc =3D 0xffffff80007a9000, if_l2com =3D 0xfffffe000300b200,
 >
 > =C2=A0if_vnet =3D 0x0, if_link =3D {tqe_next =3D 0xfffffe0003002000,
 > =C2=A0 =C2=A0tqe_prev =3D 0xfffffe0003003818},
 > =C2=A0if_xname =3D "bge0", '\0' <repeats 11 times>,
 > =C2=A0if_dname =3D 0xfffffe00028f0758 "bge", if_dunit =3D 0, if_refcount =
 =3D 1,
 >
 > =C2=A0if_addrhead =3D {tqh_first =3D 0xfffffe000300a000,
 > =C2=A0 =C2=A0tqh_last =3D 0xfffffe0005a940b8}, if_pcount =3D 0, if_carp =
 =3D 0x0,
 > =C2=A0if_bpf =3D 0xfffffe0005062400, if_index =3D 5, if_index_reserved =
 =3D 0,
 >
 > =C2=A0if_vlantrunk =3D 0x0, if_flags =3D 34819, if_capabilities =3D 52444=
 3,
 > =C2=A0if_capenable =3D 524443, if_linkmib =3D 0x0, if_linkmiblen =3D 0, i=
 f_data =3D {
 > =C2=A0 =C2=A0ifi_type =3D 6 '\006', ifi_physical =3D 0 '\0', ifi_addrlen =
 =3D 6 '\006',
 > =C2=A0 =C2=A0ifi_hdrlen =3D 18 '\022', ifi_link_state =3D 2 '\002',
 > =C2=A0 =C2=A0ifi_spare_char1 =3D 0 '\0', ifi_spare_char2 =3D 0 '\0',
 > =C2=A0 =C2=A0ifi_datalen =3D 152 '\230', ifi_mtu =3D 1500, ifi_metric =3D=
  0,
 > =C2=A0 =C2=A0ifi_baudrate =3D 1000000000, ifi_ipackets =3D 4678659, ifi_i=
 errors =3D 0,
 > =C2=A0 =C2=A0ifi_opackets =3D 2594069, ifi_oerrors =3D 0, ifi_collisions =
 =3D 0,
 > =C2=A0 =C2=A0ifi_ibytes =3D 598927432, ifi_obytes =3D 2837994361, ifi_imc=
 asts =3D 2432290,
 >
 > =C2=A0 =C2=A0ifi_omcasts =3D 0, ifi_iqdrops =3D 0, ifi_noproto =3D 0, ifi=
 _hwassist =3D 3,
 > =C2=A0 =C2=A0ifi_epoch =3D 1, ifi_lastchange =3D {tv_sec =3D 1338284854, =
 tv_usec =3D 622823}},
 > =C2=A0if_multiaddrs =3D {tqh_first =3D 0xfffffe0005bdb080,
 > =C2=A0 =C2=A0tqh_last =3D 0xfffffe00058ff080}, if_amcount =3D 0,
 > =C2=A0if_output =3D 0xffffffff8073d2f5 <ether_output>,
 > =C2=A0if_input =3D 0xffffffff8073c8cb <ether_input>,
 > =C2=A0if_start =3D 0xffffffff803c2b67 <bge_start>,
 > =C2=A0if_ioctl =3D 0xffffffff803c8d9a <bge_ioctl>,
 > =C2=A0if_init =3D 0xffffffff803c8d54 <bge_init>,
 > =C2=A0if_resolvemulti =3D 0xffffffff8073c28d <ether_resolvemulti>,
 > =C2=A0if_qflush =3D 0xffffffff807350b2 <if_qflush>,
 > =C2=A0if_transmit =3D 0xffffffff80734f7e <if_transmit>, if_reassign =3D 0=
 ,
 >
 > =C2=A0if_home_vnet =3D 0x0, if_addr =3D 0xfffffe000300a000, if_llsoftc =
 =3D 0x0,
 > =C2=A0if_drv_flags =3D 64, if_snd =3D {ifq_head =3D 0x0, ifq_tail =3D 0x0=
 , ifq_len =3D 0,
 > =C2=A0 =C2=A0ifq_maxlen =3D 511, ifq_drops =3D 0, ifq_mtx =3D {lock_objec=
 t =3D {
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0lo_name =3D 0xfffffe0003001828 "bge0", lo_flag=
 s =3D 16973824, lo_data =3D
 > 0,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0lo_witness =3D 0xffffff80006cf480}, mtx_lock =
 =3D 4}, ifq_drv_head =3D 0x0,
 > =C2=A0 =C2=A0ifq_drv_tail =3D 0x0, ifq_drv_len =3D 0, ifq_drv_maxlen =3D =
 511, altq_type =3D 0,
 > =C2=A0 =C2=A0altq_flags =3D 1, altq_disc =3D 0x0, altq_ifp =3D 0xfffffe00=
 03001800,
 > =C2=A0 =C2=A0altq_enqueue =3D 0, altq_dequeue =3D 0, altq_request =3D 0, =
 altq_clfier =3D 0x0,
 > =C2=A0 =C2=A0altq_classify =3D 0, altq_tbr =3D 0x0, altq_cdnr =3D 0x0},
 > =C2=A0if_broadcastaddr =3D 0xffffffff80ad06c0 "=C4=B8=C4=B8=C4=B8=C4=B8=
 =C4=B8=C4=B8", if_bridge =3D 0x0,
 >
 > =C2=A0if_label =3D 0x0, if_prefixhead =3D {tqh_first =3D 0x0,
 > =C2=A0 =C2=A0tqh_last =3D 0xfffffe0003001a78}, if_afdata =3D {0x0, 0x0,
 > 0xfffffe0005821a20,
 > =C2=A0 =C2=A00x0 <repeats 25 times>, 0xfffffe0005815940, 0x0, 0x0, 0x0, 0=
 x0, 0x0, 0x0,
 >
 > =C2=A0 =C2=A00x0, 0x0, 0x0}, if_afdata_initialized =3D 2, if_afdata_lock =
 =3D {
 > =C2=A0 =C2=A0lock_object =3D {lo_name =3D 0xffffffff80acf95a "if_afdata",
 >
 > =C2=A0 =C2=A0 =C2=A0lo_flags =3D 69402624, lo_data =3D 0, lo_witness =3D =
 0xffffff80006cf400},
 > =C2=A0 =C2=A0rw_lock =3D 1}, if_linktask =3D {ta_link =3D {stqe_next =3D =
 0x0}, ta_pending =3D 0,
 > =C2=A0 =C2=A0ta_priority =3D 0, ta_func =3D 0xffffffff80737559 <do_link_s=
 tate_change>,
 >
 > =C2=A0 =C2=A0ta_context =3D 0xfffffe0003001800}, if_addr_mtx =3D {lock_ob=
 ject =3D {
 > =C2=A0 =C2=A0 =C2=A0lo_name =3D 0xffffffff80ac1a20 "if_addr_mtx", lo_flag=
 s =3D 16973824,
 >
 > =C2=A0 =C2=A0 =C2=A0lo_data =3D 0, lo_witness =3D 0xffffff80006c8b80}, mt=
 x_lock =3D 4},
 > =C2=A0if_clones =3D {le_next =3D 0x0, le_prev =3D 0x0}, if_groups =3D {
 > =C2=A0 =C2=A0tqh_first =3D 0xfffffe0003007b20, tqh_last =3D 0xfffffe00030=
 07b28},
 > =C2=A0if_pf_kif =3D 0xfffffe0005888400, if_lagg =3D 0x0, if_description =
 =3D 0x0,
 >
 > =C2=A0if_fib =3D 0, if_alloctype =3D 6 '\006', if_cspare =3D "\000\000", =
 if_ispare =3D
 > {0,
 > =C2=A0 =C2=A00, 0, 0}, if_pspare =3D {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, =
 0x0}}
 > (kgdb) p pd
 > $3 =3D {lookup =3D {done =3D 0, uid =3D 0, gid =3D 0, pid =3D 0}, tot_len=
  =3D 0, hdr =3D {
 > =C2=A0 =C2=A0tcp =3D 0x0, udp =3D 0x0, icmp =3D 0x0, icmp6 =3D 0x0, any =
 =3D 0x0}, nat_rule =3D
 > 0x0,
 > =C2=A0eh =3D 0x0, src =3D 0x0, dst =3D 0x0, sport =3D 0x0, dport =3D 0x0,
 > =C2=A0pf_mtag =3D 0xfffffe00644f9358, p_len =3D 0, ip_sum =3D 0x0, proto_=
 sum =3D 0x0,
 > =C2=A0flags =3D 0, af =3D 0 '\0', proto =3D 0 '\0', tos =3D 0 '\0', dir =
 =3D 0 '\0',
 > =C2=A0sidx =3D 0 '\0', didx =3D 0 '\0'}
 > (kgdb) p pf_status
 > $4 =3D {counters =3D {9415424, 0, 0, 0, 0, 0, 0, 0, 3464, 0, 27, 0, 0, 0,=
  0},
 > =C2=A0lcounters =3D {0, 0, 0, 0, 0, 0, 0}, fcounters =3D {12630228, 74172=
 , 74158},
 > =C2=A0scounters =3D {0, 0, 0}, pcounters =3D {{{0, 0, 0}, {0, 0, 0}}, {{0=
 , 0, 0}, {0,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A00, 0}}}, bcounters =3D {{0, 0}, {0, 0}}, state=
 id =3D 5747889684957176252,
 > =C2=A0running =3D 1, states =3D 14, src_nodes =3D 0, since =3D 1338284855=
 , debug =3D 1,
 > =C2=A0hostid =3D 3046117155, ifname =3D '\0' <repeats 15 times>,
 > =C2=A0pf_chksum =3D "qu=C3=8E\205<0=C2=AD=C2=A0h=C5=A1\021=C5=A7=C5=ABvi\=
 203"}
 > (kgdb) p pf_status.running
 > $5 =3D 1
 > (kgdb) up
 > #11 0xffffffff8032cc7b in pf_check_out (arg=3D)
 > =C2=A0 =C2=A0at /usr/src/sys/contrib/pf/net/pf_ioctl.c:4184
 > 4184 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0chk =3D pf_test(PF_OUT, ifp=
 , m, NULL, inp);
 > (kgdb) list
 > 4179 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0h =3D mtod(*m, struct ip *);
 > 4180 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0HTONS(h->ip_len);
 > 4181 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0HTONS(h->ip_off);
 > 4182 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0}
 > 4183 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CURVNET_SET(ifp->if_vnet);
 > 4184 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0chk =3D pf_test(PF_OUT, ifp=
 , m, NULL, inp);
 > 4185 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CURVNET_RESTORE();
 > 4186 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (chk && *m) {
 > 4187 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0m_freem(*m);
 > 4188 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0*m =3D NULL;
 > (kgdb) up
 > #12 0xffffffff8074adcf in pfil_run_hooks (ph=3D) at /usr/src/sys/net/pfil=
 .c:89
 > 89 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rv =3D (*pfh->pfil_func)(pfh->pfil_arg, &=
 m,
 > ifp, dir,
 > (kgdb) list
 > 84 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0KASSERT(ph->ph_nhooks =
 >=3D 0, ("Pfil hook count dropped <
 > 0"));
 > 85 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0for (pfh =3D pfil_hook=
 _get(dir, ph); pfh !=3D NULL;
 > 86 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 pfh =3D=
  TAILQ_NEXT(pfh, pfil_link)) {
 > 87 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0if (pfh->pfil_func !=3D NULL) {
 > 88 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ASSERT_HOST_BYTE_ORDER(m);
 > 89 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rv =3D (*pfh->pfil_func)(pfh->pfil_arg, &=
 m,
 > ifp, dir,
 > 90 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0inp);
 > 91 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (rv !=3D 0 || m =3D=3D NULL)
 > 92 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0break;
 > 93 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ASSERT_HOST_BYTE_ORDER(m);
 > (kgdb) p *pfh
 > $6 =3D {pfil_link =3D {tqe_next =3D 0x0, tqe_prev =3D 0xfffffe0005821b00}=
 ,
 > =C2=A0pfil_func =3D 0xffffffff8032cc0a <pf_check_out>, pfil_arg =3D 0x0}
 > (kgdb) up
 > #13 0xffffffff80776b3a in ip_output (m=3D0xfffffe0064823700, opt=3D)
 > =C2=A0 =C2=A0at /usr/src/sys/netinet/ip_output.c:512
 >
 > 512 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 error =3D pfil_run_hooks(&V=
 _inet_pfil_hook, &m, ifp, PFIL_OUT,
 > inp);
 > (kgdb) list
 > 507 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
  goto passout;
 > 508 509 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /* Run through list of hooks for outp=
 ut packets. */
 >
 > 510 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 odst.s_addr =3D ip->ip_dst.=
 s_addr;
 > 511 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ASSERT_HOST_BYTE_ORDER(m);
 > 512 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 error =3D pfil_run_hooks(&V=
 _inet_pfil_hook, &m, ifp, PFIL_OUT,
 > inp);
 > 513 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (error !=3D 0 || m =3D=
 =3D NULL)
 > 514 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
  goto done;
 > 515 516 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ip =3D mtod(m, struct ip *);
 > (kgdb) p *ip
 > $7 =3D {ip_hl =3D 5 '\005', ip_v =3D 4 '\004', ip_tos =3D 0 '\0', ip_len =
 =3D 45056,
 > =C2=A0ip_id =3D 62620, ip_off =3D 0, ip_ttl =3D 127 '\177', ip_p =3D 1 '\=
 001',
 > =C2=A0ip_sum =3D 4633, ip_src =3D {s_addr =3D 34052874}, ip_dst =3D {s_ad=
 dr =3D 28485824}}
 > (kgdb) p flags
 > $8 =3D 1
 > (kgdb) p mtu
 > $9 =3D 1500
 > (kgdb) p *ia
 > $10 =3D {ia_ifa =3D {ifa_addr =3D 0xfffffe0005a09338,
 > =C2=A0 =C2=A0ifa_dstaddr =3D 0xfffffe0005a09348, ifa_netmask =3D 0xfffffe=
 0005a09358,
 > =C2=A0 =C2=A0if_data =3D {ifi_type =3D 0 '\0', ifi_physical =3D 0 '\0', i=
 fi_addrlen =3D 0
 > '\0',
 > =C2=A0 =C2=A0 =C2=A0ifi_hdrlen =3D 0 '\0', ifi_link_state =3D 0 '\0', ifi=
 _spare_char1 =3D 0 '\0',
 > =C2=A0 =C2=A0 =C2=A0ifi_spare_char2 =3D 0 '\0', ifi_datalen =3D 0 '\0', i=
 fi_mtu =3D 0,
 > =C2=A0 =C2=A0 =C2=A0ifi_metric =3D 0, ifi_baudrate =3D 0, ifi_ipackets =
 =3D 4447700,
 > =C2=A0 =C2=A0 =C2=A0ifi_ierrors =3D 0, ifi_opackets =3D 2591860, ifi_oerr=
 ors =3D 0,
 > =C2=A0 =C2=A0 =C2=A0ifi_collisions =3D 0, ifi_ibytes =3D 608432458, ifi_o=
 bytes =3D 2801425920,
 > =C2=A0 =C2=A0 =C2=A0ifi_imcasts =3D 0, ifi_omcasts =3D 0, ifi_iqdrops =3D=
  0, ifi_noproto =3D 0,
 > =C2=A0 =C2=A0 =C2=A0ifi_hwassist =3D 0, ifi_epoch =3D 0, ifi_lastchange =
 =3D {tv_sec =3D 0,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0tv_usec =3D 0}}, ifa_ifp =3D 0xfffffe000300180=
 0, ifa_link =3D {
 > =C2=A0 =C2=A0 =C2=A0tqe_next =3D 0xfffffe0005a94000, tqe_prev =3D 0xfffff=
 e000300a0b8},
 > =C2=A0 =C2=A0ifa_rtrequest =3D 0, ifa_flags =3D 5, ifa_refcnt =3D 6, ifa_=
 metric =3D 0,
 > =C2=A0 =C2=A0ifa_claim_addr =3D 0, ifa_mtx =3D {lock_object =3D {
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0lo_name =3D 0xffffffff80ad4634 "ifaddr", lo_fl=
 ags =3D 16973824,
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0lo_data =3D 0, lo_witness =3D 0xffffff80006c89=
 80}, mtx_lock =3D 4}},
 > =C2=A0ia_subnet =3D 2176561920, ia_subnetmask =3D 4294967040, ia_hash =3D=
  {
 > =C2=A0 =C2=A0le_next =3D 0x0, le_prev =3D 0xfffffe000587f8c8}, ia_link =
 =3D {
 > =C2=A0 =C2=A0tqe_next =3D 0xfffffe0005902c00, tqe_prev =3D 0xfffffe000590=
 2928}, ia_addr =3D
 > {
 > =C2=A0 =C2=A0sin_len =3D 16 '\020', sin_family =3D 2 '\002', sin_port =3D=
  0, sin_addr =3D {
 > =C2=A0 =C2=A0 =C2=A0s_addr =3D 1471396737}, sin_zero =3D "\000\000\000\00=
 0\000\000\000"},
 > =C2=A0ia_dstaddr =3D {sin_len =3D 16 '\020', sin_family =3D 2 '\002', sin=
 _port =3D 0,
 > =C2=A0 =C2=A0sin_addr =3D {s_addr =3D 4289969025},
 > =C2=A0 =C2=A0sin_zero =3D "\000\000\000\000\000\000\000"}, ia_sockmask =
 =3D {
 > =C2=A0 =C2=A0sin_len =3D 7 '\a', sin_family =3D 2 '\002', sin_port =3D 0,=
  sin_addr =3D {
 > =C2=A0 =C2=A0 =C2=A0s_addr =3D 16777215}, sin_zero =3D "\000\000\000\000\=
 000\000\000"}}
 > (kgdb) p *dst
 > $11 =3D {sin_len =3D 16 '\020', sin_family =3D 2 '\002', sin_port =3D 0, =
 sin_addr =3D
 > {
 > =C2=A0 =C2=A0s_addr =3D 4273191809}, sin_zero =3D "\000\000\000\000\000\0=
 00\000"}
 > (kgdb)
 >
 > #### kgdb.out_len
 >
 >
 > - -- The beginning is the most important part of the work.
 > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
 =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-Plato
 > -----BEGIN PGP SIGNATURE-----
 > Version: GnuPG v2.0.18 (FreeBSD)
 >
 > iD8DBQFPyHyGSPOsGF+KA+MRAmr4AJ91yi1whfweG8Dkue7zi0Lvcsdn4gCfScX0
 > L8tV5u5gLMelsZX43e6yo6M=3D
 > =3DVzIz
 > -----END PGP SIGNATURE-----
 > _______________________________________________
 > freebsd-pf@freebsd.org mailing list
 > http://lists.freebsd.org/mailman/listinfo/freebsd-pf
 > To unsubscribe, send any mail to "freebsd-pf-unsubscribe@freebsd.org"
 >
 
 
 
 --=20
 Ermal

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>,
        =?ISO-8859-15?Q?Ermal_Lu=E7i?= <eri@freebsd.org>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Mon, 4 Jun 2012 08:40:16 +0200 (CEST)

   This message is in MIME format.  The first part should be readable text,
   while the remaining parts are likely unreadable without MIME-aware tools.
 
 --3469798045-1136351326-1338792020=:89783
 Content-Type: TEXT/PLAIN; charset=UTF-8; format=flowed
 Content-Transfer-Encoding: 8BIT
 
 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Fri, 1 Jun 2012, Ermal Luçi wrote:
 
 > On Fri, Jun 1, 2012 at 10:25 AM, Joerg Pulz <Joerg.Pulz@frm2.tum.de> wrote:
 >> -----BEGIN PGP SIGNED MESSAGE-----
 >> Hash: SHA1
 >>
 >>
 >> On Tue, 29 May 2012, Daniel Hartmeier wrote:
 >>
 >>> On Sun, May 27, 2012 at 06:30:09PM +0000, Joerg Pulz wrote:
 >>>
 >>>>  i've seen 12 more "pf_route: m0->m_len < sizeof(struct ip)" messages
 >>>> since the system is running after adding your patch, but no panic.
 >>>>  Is there another place in the code where i can add an additional check?
 >>>
 >>>
 >>> Yes, the following patch adds more checks to pf.
 >>
 >>
 >> Daniel,
 >>
 >> after several days waiting for a panic since i applied your new patch, it
 >> finally happend last night.
 >>
 >> Below is the kgdb(1) output. I tried to print as much as possible to give
 >> you the most informations.
 >>
 >> Hope this helps to find the cuase of the trouble or at least to get a bit
 >> closer.
 >>
 >> #### kgdb.out_len
 >> pf_test() at pf_test+0x259
 >> pf_check_out() at pf_check_out+0x71
 >> pfil_run_hooks() at pfil_run_hooks+0x113
 >>
 >> ip_output() at ip_output+0x6de
 >> ip_forward() at ip_forward+0x19e
 >
 > It is quite strange that you do not have a pfil_run_hooks() here as well!
 > Maybe you are running ipsec, if so i would expect that to show in the trace!?
 >
 > Can you describe the setup you have in more detail to understand what
 > interactions are happening with the stack?
 
 Ermal,
 
 for detailed information please take a look at PR: kern/168190
 Everything is described there.
 
 Meanwhile, i got five more panics. Four of them look exactly like the 
 first 
 one:
  	panic: pf_test: 1: m->m_pkthdr.len 176, m->m_len 0
 
 and the last one shows:
  	panic: pf_test: 1: m->m_pkthdr.len 310, m->m_len 0
 
 All panics with "m->m_pkthdr.len 176, m->m_len 0" show "bge0" as ifname, 
 which is the external interface.
 The last panic with "m->m_pkthdr.len 310, m->m_len 0" shows "bge1" as 
 ifname, which is the internal interface.
 
 
 All dumps look like the first one, only interface counters, addresses 
 (memory and involved src and dst) differ.
 
 For the sake of completeness, here is the output of the last dump.
 
 Any further ideas?
 
 Kind regards
 Joerg
 
 
 #### kgdb.out_len6
 
 GNU gdb 6.1.1 [FreeBSD]
 Copyright 2004 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "amd64-marcel-freebsd"...
 
 Unread portion of the kernel message buffer:
 panic: pf_test: 1: m->m_pkthdr.len 310, m->m_len 0
 cpuid = 0
 KDB: stack backtrace:
 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
 kdb_backtrace() at kdb_backtrace+0x37
 panic() at panic+0x182
 pf_test() at pf_test+0x259
 pf_check_out() at pf_check_out+0x71
 pfil_run_hooks() at pfil_run_hooks+0x113
 ip_output() at ip_output+0x6de
 ip_forward() at ip_forward+0x19e
 ip_input() at ip_input+0x680
 swi_net() at swi_net+0x15a
 intr_event_execute_handlers() at intr_event_execute_handlers+0x66
 ithread_loop() at ithread_loop+0xaf
 fork_exit() at fork_exit+0x12a
 fork_trampoline() at fork_trampoline+0xe
 - --- trap 0, rip = 0, rsp = 0xffffff8000241d00, rbp = 0 ---
 KDB: enter: panic
 Dumping 714 out of 4077 MB:..3%..12%..21%..32%..41%..52%..61%..72%..81%..92%
 
 Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from /boot/kernel/geom_mirror.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/geom_mirror.ko
 Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
 done.
 Loaded symbols for /boot/kernel/ipmi.ko
 #0  doadump (textdump=0) at pcpu.h:224
 224		__asm("movq %%gs:0,%0" : "=r" (td));
 (kgdb) up 10
 #10 0xffffffff80326737 in pf_test (dir=2, ifp=0xfffffe0003002000,
      m0=0xffffff80002418e8, eh=0x0, inp=0x0)
      at /usr/src/sys/contrib/pf/net/pf.c:6725
 6725			panic("pf_test: 1: m->m_pkthdr.len %d, m->m_len %d",
 (kgdb) list
 6720			goto done;
 6721		}
 6722 
 6723		if (m->m_pkthdr.len < sizeof(struct ip) ||
 6724		    m->m_len < sizeof(struct ip))
 6725			panic("pf_test: 1: m->m_pkthdr.len %d, m->m_len %d",
 6726			    (int)m->m_pkthdr.len, (int)m->m_len);
 6727 
 6728	#ifdef __FreeBSD__
 6729		if (m->m_flags & M_SKIP_FIREWALL) {
 (kgdb) p *m
 $1 = {m_hdr = {mh_next = 0xfffffe0005a80800, mh_nextpkt = 0x0,
      mh_data = 0xfffffe001a8bb174 "E", mh_len = 0, mh_flags = 66, mh_type = 1,
      pad = "­ÞÞÀ­Þ"}, M_dat = {MH = {MH_pkthdr = {rcvif = 0xfffffe0003001800,
          header = 0x0, len = 310, flowid = 0, csum_flags = 768,
          csum_data = 26821, tso_segsz = 0, PH_vt = {vt_vtag = 0, vt_nrecs = 0},
          tags = {slh_first = 0xfffffe00058bc580}}, MH_dat = {MH_ext = {
            ext_buf = 0x3765016c0045 <Address 0x3765016c0045 out of bounds>,
            ext_free = 0x376501360045, ext_arg1 = 0x4e79875d91110437,
            ext_arg2 = 0x3601004557b3bb81, ext_size = 3584,
            ref_cnt = 0x240119ac05079b0a, ext_type = -239336701},
          MH_databuf = "E\000l\001e7\000\000E\0006\001e7\000\0007\004\021\221]\207yN\201»³WE\000\0016\000\016\000\000\177\001zÜ\n\233\a\005¬\031\001$\003\003¼ñ\000\000\000\000E\000\001\032\f\213\000\000>\021°k¬\031\001$\n\233\a\005\0005öf\001\006«û\200[\201\200\000\001\000\001\000\003\000\006ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"}},
      M_databuf = "\000\030\000\003\000þÿÿ\000\000\000\000\000\000\000\0006\001\000\000\000\000\000\000\000\003\000\000Åh\000\000\000\000\000\000ÞÀ­Þ\200Å\213\005\000þÿÿE\000l\001e7\000\000E\0006\001e7\000\0007\004\021\221]\207yN\201»³WE\000\0016\000\016\000\000\177\001zÜ\n\233\a\005¬\031\001$\003\003¼ñ\000\000\000\000E\000\001\032\f\213\000\000>\021°k¬\031\001$\n\233\a\005\0005öf\001\006«û\200[\201\200\000\001\000\001\000\003\000\006ÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞ ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­ÞÞÀ­Þ"...}}
 (kgdb) p *ifp
 $2 = {if_softc = 0xffffff80007b1000, if_l2com = 0xfffffe000300ba40,
    if_vnet = 0x0, if_link = {tqe_next = 0xfffffe0003001000,
      tqe_prev = 0xfffffe0003001818},
    if_xname = "bge1", '\0' <repeats 11 times>,
    if_dname = 0xfffffe00028f0758 "bge", if_dunit = 1, if_refcount = 1,
    if_addrhead = {tqh_first = 0xfffffe0003009800,
      tqh_last = 0xfffffe00059236b8}, if_pcount = 0, if_carp = 0x0,
    if_bpf = 0xfffffe00050a4880, if_index = 6, if_index_reserved = 0,
    if_vlantrunk = 0x0, if_flags = 34819, if_capabilities = 524443,
    if_capenable = 524443, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
      ifi_type = 6 '\006', ifi_physical = 0 '\0', ifi_addrlen = 6 '\006',
      ifi_hdrlen = 18 '\022', ifi_link_state = 2 '\002',
      ifi_spare_char1 = 0 '\0', ifi_spare_char2 = 0 '\0',
      ifi_datalen = 152 '\230', ifi_mtu = 1500, ifi_metric = 0,
      ifi_baudrate = 1000000000, ifi_ipackets = 354876, ifi_ierrors = 0,
      ifi_opackets = 141821, ifi_oerrors = 0, ifi_collisions = 0,
      ifi_ibytes = 106681967, ifi_obytes = 12317395, ifi_imcasts = 169195,
      ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 3,
      ifi_epoch = 1, ifi_lastchange = {tv_sec = 1338708223, tv_usec = 13051}},
    if_multiaddrs = {tqh_first = 0xfffffe0005926b40,
      tqh_last = 0xfffffe00058b8d00}, if_amcount = 0,
    if_output = 0xffffffff8073d2f5 <ether_output>,
    if_input = 0xffffffff8073c8cb <ether_input>,
    if_start = 0xffffffff803c2b67 <bge_start>,
    if_ioctl = 0xffffffff803c8d9a <bge_ioctl>,
    if_init = 0xffffffff803c8d54 <bge_init>,
    if_resolvemulti = 0xffffffff8073c28d <ether_resolvemulti>,
    if_qflush = 0xffffffff807350b2 <if_qflush>,
    if_transmit = 0xffffffff80734f7e <if_transmit>, if_reassign = 0,
    if_home_vnet = 0x0, if_addr = 0xfffffe0003009800, if_llsoftc = 0x0,
    if_drv_flags = 64, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0,
      ifq_maxlen = 511, ifq_drops = 0, ifq_mtx = {lock_object = {
          lo_name = 0xfffffe0003002028 "bge1", lo_flags = 16973824, lo_data = 0,
          lo_witness = 0xffffff80006cf480}, mtx_lock = 4}, ifq_drv_head = 0x0,
      ifq_drv_tail = 0x0, ifq_drv_len = 0, ifq_drv_maxlen = 511, altq_type = 0,
      altq_flags = 1, altq_disc = 0x0, altq_ifp = 0xfffffe0003002000,
      altq_enqueue = 0, altq_dequeue = 0, altq_request = 0, altq_clfier = 0x0,
      altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0},
    if_broadcastaddr = 0xffffffff80ad06c0 "ÿÿÿÿÿÿ", if_bridge = 0x0,
    if_label = 0x0, if_prefixhead = {tqh_first = 0x0,
      tqh_last = 0xfffffe0003002278}, if_afdata = {0x0, 0x0, 0xfffffe0005821a00,
      0x0 <repeats 25 times>, 0xfffffe0005815880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
      0x0, 0x0, 0x0}, if_afdata_initialized = 2, if_afdata_lock = {
      lock_object = {lo_name = 0xffffffff80acf95a "if_afdata",
        lo_flags = 69402624, lo_data = 0, lo_witness = 0xffffff80006cf400},
      rw_lock = 1}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
      ta_priority = 0, ta_func = 0xffffffff80737559 <do_link_state_change>,
      ta_context = 0xfffffe0003002000}, if_addr_mtx = {lock_object = {
        lo_name = 0xffffffff80ac1a20 "if_addr_mtx", lo_flags = 16973824,
        lo_data = 0, lo_witness = 0xffffff80006c8b80}, mtx_lock = 4},
    if_clones = {le_next = 0x0, le_prev = 0x0}, if_groups = {
      tqh_first = 0xfffffe0005065ae0, tqh_last = 0xfffffe0005065ae8},
    if_pf_kif = 0xfffffe0005888300, if_lagg = 0x0, if_description = 0x0,
    if_fib = 0, if_alloctype = 6 '\006', if_cspare = "\000\000", if_ispare = {0,
      0, 0, 0}, if_pspare = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
 (kgdb) p pd
 $3 = {lookup = {done = 0, uid = 0, gid = 0, pid = 0}, tot_len = 0, hdr = {
      tcp = 0x0, udp = 0x0, icmp = 0x0, icmp6 = 0x0, any = 0x0}, nat_rule = 0x0,
    eh = 0x0, src = 0x0, dst = 0x0, sport = 0x0, dport = 0x0,
    pf_mtag = 0xfffffe001a43dbd8, p_len = 0, ip_sum = 0x0, proto_sum = 0x0,
    flags = 0, af = 0 '\0', proto = 0 '\0', tos = 0 '\0', dir = 0 '\0',
    sidx = 0 '\0', didx = 0 '\0'}
 (kgdb) p pf_status
 $4 = {counters = {634320, 0, 0, 0, 0, 0, 0, 0, 320, 0, 6, 0, 0, 0, 0},
    lcounters = {0, 0, 0, 0, 0, 0, 0}, fcounters = {719058, 4120, 4008},
    scounters = {0, 0, 0}, pcounters = {{{0, 0, 0}, {0, 0, 0}}, {{0, 0, 0}, {0,
          0, 0}}}, bcounters = {{0, 0}, {0, 0}}, stateid = 5749708040966246424,
    running = 1, states = 112, src_nodes = 0, since = 1338708224, debug = 1,
    hostid = 4123334744, ifname = '\0' <repeats 15 times>,
    pf_chksum = "quÎ\205<0­ hº\021»¾vi\203"}
 (kgdb) p pf_status.running
 $5 = 1
 (kgdb) up
 #11 0xffffffff8032cc7b in pf_check_out (arg=)
      at /usr/src/sys/contrib/pf/net/pf_ioctl.c:4184
 4184		chk = pf_test(PF_OUT, ifp, m, NULL, inp);
 (kgdb) list
 4179			h = mtod(*m, struct ip *);
 4180			HTONS(h->ip_len);
 4181			HTONS(h->ip_off);
 4182		}
 4183		CURVNET_SET(ifp->if_vnet);
 4184		chk = pf_test(PF_OUT, ifp, m, NULL, inp);
 4185		CURVNET_RESTORE();
 4186		if (chk && *m) {
 4187			m_freem(*m);
 4188			*m = NULL;
 (kgdb) up
 #12 0xffffffff8074adcf in pfil_run_hooks (ph=) at /usr/src/sys/net/pfil.c:89
 89				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 (kgdb) list
 84		KASSERT(ph->ph_nhooks >= 0, ("Pfil hook count dropped < 0"));
 85		for (pfh = pfil_hook_get(dir, ph); pfh != NULL;
 86		     pfh = TAILQ_NEXT(pfh, pfil_link)) {
 87			if (pfh->pfil_func != NULL) {
 88				ASSERT_HOST_BYTE_ORDER(m);
 89				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
 90				    inp);
 91				if (rv != 0 || m == NULL)
 92					break;
 93				ASSERT_HOST_BYTE_ORDER(m);
 (kgdb) p *pfh
 $6 = {pfil_link = {tqe_next = 0x0, tqe_prev = 0xfffffe0005821b00},
    pfil_func = 0xffffffff8032cc0a <pf_check_out>, pfil_arg = 0x0}
 (kgdb) up
 #13 0xffffffff80776b3a in ip_output (m=0xfffffe001a8bb100, opt=)
      at /usr/src/sys/netinet/ip_output.c:512
 512		error = pfil_run_hooks(&V_inet_pfil_hook, &m, ifp, PFIL_OUT, inp);
 (kgdb) list
 507			goto passout;
 508 
 509		/* Run through list of hooks for output packets. */
 510		odst.s_addr = ip->ip_dst.s_addr;
 511		ASSERT_HOST_BYTE_ORDER(m);
 512		error = pfil_run_hooks(&V_inet_pfil_hook, &m, ifp, PFIL_OUT, inp);
 513		if (error != 0 || m == NULL)
 514			goto done;
 515 
 516		ip = mtod(m, struct ip *);
 (kgdb) p *ip
 $7 = {ip_hl = 5 '\005', ip_v = 4 '\004', ip_tos = 0 '\0', ip_len = 13825,
    ip_id = 3584, ip_off = 0, ip_ttl = 127 '\177', ip_p = 1 '\001',
    ip_sum = 56442, ip_src = {s_addr = 84384522}, ip_dst = {s_addr = 604051884}}
 (kgdb) p flags
 $8 = 1
 (kgdb) p mtu
 $9 = 1500
 (kgdb) p *ia
 $10 = {ia_ifa = {ifa_addr = 0xfffffe000592f338,
      ifa_dstaddr = 0xfffffe000592f348, ifa_netmask = 0xfffffe000592f358,
      if_data = {ifi_type = 0 '\0', ifi_physical = 0 '\0', ifi_addrlen = 0 '\0',
        ifi_hdrlen = 0 '\0', ifi_link_state = 0 '\0', ifi_spare_char1 = 0 '\0',
        ifi_spare_char2 = 0 '\0', ifi_datalen = 0 '\0', ifi_mtu = 0,
        ifi_metric = 0, ifi_baudrate = 0, ifi_ipackets = 6077, ifi_ierrors = 0,
        ifi_opackets = 2883, ifi_oerrors = 0, ifi_collisions = 0,
        ifi_ibytes = 823732, ifi_obytes = 309266, ifi_imcasts = 0,
        ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0,
        ifi_epoch = 0, ifi_lastchange = {tv_sec = 0, tv_usec = 0}},
      ifa_ifp = 0xfffffe0003002000, ifa_link = {tqe_next = 0xfffffe0005923600,
        tqe_prev = 0xfffffe00030098b8}, ifa_rtrequest = 0, ifa_flags = 5,
      ifa_refcnt = 7, ifa_metric = 0, ifa_claim_addr = 0, ifa_mtx = {
        lock_object = {lo_name = 0xffffffff80ad4634 "ifaddr",
          lo_flags = 16973824, lo_data = 0, lo_witness = 0xffffff80006c8980},
        mtx_lock = 4}}, ia_subnet = 2887319808, ia_subnetmask = 4294967040,
    ia_hash = {le_next = 0x0, le_prev = 0xfffffe000587ff70}, ia_link = {
      tqe_next = 0x0, tqe_prev = 0xfffffe00058e1d28}, ia_addr = {
      sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 0, sin_addr = {
        s_addr = 2533431724}, sin_zero = "\000\000\000\000\000\000\000"},
    ia_dstaddr = {sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 0,
      sin_addr = {s_addr = 4278262188},
      sin_zero = "\000\000\000\000\000\000\000"}, ia_sockmask = {
      sin_len = 7 '\a', sin_family = 2 '\002', sin_port = 0, sin_addr = {
        s_addr = 16777215}, sin_zero = "\000\000\000\000\000\000\000"}}
 (kgdb) p *dst
 $11 = {sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 0, sin_addr = {
      s_addr = 604051884}, sin_zero = "\000\000\000\000\000\000\000"}
 (kgdb)
 
 #### kgdb.out_len6
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPzFhTSPOsGF+KA+MRAtVJAJ4oHrqC8Y4O0q0vHjgnhJ7wrVlE7ACfSkc0
 6juyQmrvcT+m74JdfH30tlE=
 =ayEo
 -----END PGP SIGNATURE-----
 --3469798045-1136351326-1338792020=:89783--

From: Daniel Hartmeier <daniel@benzedrine.cx>
To: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad fragment handling?)
Date: Mon, 4 Jun 2012 08:53:44 +0200

 --8t9RHnE3ZwKMSgU+
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On Fri, Jun 01, 2012 at 10:25:39AM +0200, Joerg Pulz wrote:
 
 > panic: pf_test: 1: m->m_pkthdr.len 176, m->m_len 0
 > pf_test() at pf_test+0x259
 > pf_check_out() at pf_check_out+0x71
 > pfil_run_hooks() at pfil_run_hooks+0x113
 > ip_output() at ip_output+0x6de
 > ip_forward() at ip_forward+0x19e
 > ip_input() at ip_input+0x680
 > swi_net() at swi_net+0x15a
 
 The interesting part is in pfil_rule_hooks:
 
 > #12 0xffffffff8074adcf in pfil_run_hooks (ph=) at /usr/src/sys/net/pfil.c:89
 > 89				rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, 
 > ifp, dir,
 > (kgdb) p *pfh
 > $6 = {pfil_link = {tqe_next = 0x0, tqe_prev = 0xfffffe0005821b00},
 >   pfil_func = 0xffffffff8032cc0a <pf_check_out>, pfil_arg = 0x0}
 
 There is a check on entry, which didn't trigger, so the mbuf was fine
 when the function was called.
 
 We're in the second pass of the loop, there seem to be (at least) two
 registered hooks, with pf being called second. What is the first one?
 You disabled ipfw, so my guess is ipfilter is first.
 
 Can you try to print *tqe_prev in the pfil_run_hooks frame?
 
 Now, the question is whether the first hook modifies the mbuf, or if
 it's pf on the way seen in your stack trace.
 
 I added further checks (before and after each hook), see the updated
 patch of pfil.c below.
 
 You could also disable ipfilter (so the module isn't loaded at all, and
 no pfil hook is registered).
 
 I'm reading more mbuf functions to find out what might leave the first
 chunk of an mbuf with m_len 0 (possibly some m_adj() call?), and from
 where it might be called.
 
 Kind regards,
 Daniel
 
 --8t9RHnE3ZwKMSgU+
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: attachment; filename="pfil.diff"
 
 Index: sys/net/pfil.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/net/pfil.c,v
 retrieving revision 1.19.2.1
 diff -u -r1.19.2.1 pfil.c
 --- sys/net/pfil.c	23 Sep 2011 00:51:37 -0000	1.19.2.1
 +++ sys/net/pfil.c	4 Jun 2012 06:39:46 -0000
 @@ -46,6 +46,8 @@
  
  #include <net/if.h>
  #include <net/pfil.h>
 +#include <netinet/in.h>
 +#include <netinet/ip.h>
  
  static struct mtx pfil_global_lock;
  
 @@ -74,15 +76,31 @@
  	struct mbuf *m = *mp;
  	int rv = 0;
  
 +	if (m->m_pkthdr.len < sizeof(struct ip) ||
 +	    m->m_len < sizeof(struct ip))
 +		panic("pfil_run_hooks: 1: m->m_pkthdr.len %d, m->m_len %d",
 +		    (int)m->m_pkthdr.len, (int)m->m_len);
  	PFIL_RLOCK(ph, &rmpt);
  	KASSERT(ph->ph_nhooks >= 0, ("Pfil hook count dropped < 0"));
  	for (pfh = pfil_hook_get(dir, ph); pfh != NULL;
  	     pfh = TAILQ_NEXT(pfh, pfil_link)) {
  		if (pfh->pfil_func != NULL) {
 +			ASSERT_HOST_BYTE_ORDER(m);
 +			if (m->m_pkthdr.len < sizeof(struct ip) ||
 +			    m->m_len < sizeof(struct ip))
 +				panic("pfil_run_hooks: 2: m->m_pkthdr.len %d, "
 +				    "m->m_len %d", (int)m->m_pkthdr.len,
 +				    (int)m->m_len);
  			rv = (*pfh->pfil_func)(pfh->pfil_arg, &m, ifp, dir,
  			    inp);
  			if (rv != 0 || m == NULL)
  				break;
 +			ASSERT_HOST_BYTE_ORDER(m);
 +			if (m->m_pkthdr.len < sizeof(struct ip) ||
 +			    m->m_len < sizeof(struct ip))
 +				panic("pfil_run_hooks: 3: m->m_pkthdr.len %d, "
 +				    "m->m_len %d", (int)m->m_pkthdr.len,
 +				    (int)m->m_len);
  		}
  	}
  	PFIL_RUNLOCK(ph, &rmpt);
 
 --8t9RHnE3ZwKMSgU+--

From: Daniel Hartmeier <daniel@benzedrine.cx>
To: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad fragment handling?)
Date: Mon, 4 Jun 2012 12:08:29 +0200

 I think I found what might happen in your case. When reading the
 ipfilter hook previously, I thought that it would return quickly due to
 fr_running not being enabled, but it's not an 'enabled' flag as set
 manually with pfctl -e!
 
 If you compile in ipfilter (as your kernel config does), it will run
 a lot of code, even if you don't enable it in rc.conf and configure a
 ruleset.
 
 I noticed that in all your traces so far, the packets always have ip_p=1
 (ICMP) and a non-minimal length.
 
 There's a very suspicious m_pulldown() call here
 
 	fr_check_wrapper()
 	fr_check()
 	fr_makefrip()
 	frpr_ipv4hdr()
 	frpr_icmp()
 	  for ICMP_SOURCEQUENCH|REDIRECT|TIMEXCEED|PARAMPROB
 		fr_coalesce()
 			fr_pullup()
 			  for len > MHLEN (around what, 150?)
 				m_pulldown()
 
 where fr_pullup() subsequently even does this
 
                 while (M_LEN(m) == 0) {
                         m = m->m_next;
                 }
 
 i.e. it seems to explicitely deal with the case where the mbuf starts
 with m_len==0 segments.
 
 The problem is, nothing else does, neither other pfil consumers (like
 ipfw or pf) nor bridge or any other networking code I can find.
 This could also explain the effects in ipfw (not updating byte order)
 and pf (panic you originally reported).
 
 There are barely any other m_pulldown() calls in the kernel, especially
 with offp=NULL, and its source is full of warning comments, see
 /usr/src/sys/kern/uipc_mbuf2.c. So I'm not sure if it's being used
 wrongly (ignoring its return value certainly looks wrong), or if it
 shouldn't be used at all.
 
 You can possibly trigger the problem by provoking an ICMP TTL exceeded
 error of size 150-200, either with traceroute -I packetlen or by
 manually crafting it (with dnet from ports or such).
 
 Alternatively, I suspect the problem will no longer show when you
 disable ipfilter.
 
 Daniel

From: Daniel Hartmeier <daniel@benzedrine.cx>
To: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad fragment handling?)
Date: Mon, 4 Jun 2012 12:25:44 +0200

 Here's a patch that directly tests this theory.
 
 If correct, it will replace the panics with simple log messages that
 show when ipfilter left an m_len==0 mbuf.
 
 Daniel
 
 Index: sys/contrib/ipfilter/netinet/ip_fil_freebsd.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/contrib/ipfilter/netinet/ip_fil_freebsd.c,v
 retrieving revision 1.20.4.1
 diff -u -r1.20.4.1 ip_fil_freebsd.c
 --- sys/contrib/ipfilter/netinet/ip_fil_freebsd.c       23 Sep 2011 00:51:37 -0000      1.20.4.1
 +++ sys/contrib/ipfilter/netinet/ip_fil_freebsd.c       4 Jun 2012 10:19:04 -0000
 @@ -182,8 +182,18 @@
  static int
  fr_check_wrapper(void *arg, struct mbuf **mp, struct ifnet *ifp, int dir)
  {
 +       int r;
 +
         struct ip *ip = mtod(*mp, struct ip *);
 -       return fr_check(ip, ip->ip_hl << 2, ifp, (dir == PFIL_OUT), mp);
 +       ASSERT_HOST_BYTE_ORDER(*mp);
 +       r = fr_check(ip, ip->ip_hl << 2, ifp, (dir == PFIL_OUT), mp);
 +       if (*mp != NULL && (*mp)->m_pkthdr.len >= sizeof(struct ip) &&
 +           (*mp)->m_len < sizeof(struct ip)) {
 +               printf("fr_check_wrapper: m_len %d fixed\n",
 +                   (int)(*mp)->m_len);
 +               *mp = m_pullup(*mp, sizeof(struct ip));
 +       }
 +       return r;
  }
 
  # ifdef USE_INET6
 

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Mon, 4 Jun 2012 13:50:46 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Mon, 4 Jun 2012, Daniel Hartmeier wrote:
 
 > Here's a patch that directly tests this theory.
 >
 > If correct, it will replace the panics with simple log messages that
 > show when ipfilter left an m_len==0 mbuf.
 
 Daniel,
 
 thanks for all your help and the detailed informations.
 The system is now running with your latest patches.
 I was unable to reproduce the panics with netstat(1). Will take a look at 
 dnet later.
 
 I will keep you informed about new panics or log messages out of 
 fr_check_wrapper().
 
 Kind regards
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPzKEZSPOsGF+KA+MRAufHAKCJO7tHOUr1tcHEzxarTkQaGdnZfgCguqOP
 1eFhcJJ5y8KSc0jxK25TIX8=
 =3EA1
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Tue, 5 Jun 2012 08:48:48 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Mon, 4 Jun 2012, Daniel Hartmeier wrote:
 
 > Here's a patch that directly tests this theory.
 >
 > If correct, it will replace the panics with simple log messages that
 > show when ipfilter left an m_len==0 mbuf.
 
 Daniel,
 
 seems that your patch fixed it.
 I've seen the following log entry:
 
 Jun  5 02:15:33 charon kernel: fr_check_wrapper: m_len 0 fixed
 
 No panic and everything is running smooth.
 I will go and recompile the kernel with all the IPFIREWALL options 
 reenabled to make sure that the byte ordering problem does not appear.
 
 I will report back.
 
 Thanks for your help.
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPzavTSPOsGF+KA+MRArY+AJ43yqTeJ6hb+uCM7xZ8FWTztCz69ACgg1Wx
 yVCCuNUO0ipvlbPwa0jzZjM=
 =MGzr
 -----END PGP SIGNATURE-----

From: Joerg Pulz <Joerg.Pulz@frm2.tum.de>
To: Daniel Hartmeier <daniel@benzedrine.cx>
Cc: bug-followup@freebsd.org, freebsd-pf@freebsd.org
Subject: Re: kern/168190: [pf] panic when using pf and route-to (maybe: bad
 fragment handling?)
Date: Tue, 5 Jun 2012 14:41:20 +0200 (CEST)

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 
 On Tue, 5 Jun 2012, Joerg Pulz wrote:
 
 > On Mon, 4 Jun 2012, Daniel Hartmeier wrote:
 >
 >> Here's a patch that directly tests this theory.
 >> 
 >> If correct, it will replace the panics with simple log messages that
 >> show when ipfilter left an m_len==0 mbuf.
 >
 > Daniel,
 >
 > seems that your patch fixed it.
 > I've seen the following log entry:
 >
 > Jun  5 02:15:33 charon kernel: fr_check_wrapper: m_len 0 fixed
 >
 > No panic and everything is running smooth.
 > I will go and recompile the kernel with all the IPFIREWALL options
 > reenabled to make sure that the byte ordering problem does not appear.
 >
 > I will report back.
 
 Daniel,
 
 as promised here is my report.
 
 Your patch resolved all problems and panics.
 I've seen two log entries since i reenabled all IPFIREWALL options.
 
 Jun  5 14:07:19 charon kernel: fr_check_wrapper: m_len 0 fixed
 Jun  5 14:07:19 charon kernel: fr_check_wrapper: m_len 0 fixed
 
 No panics or other messages. Everything is running fine now.
 
 Thanks again
 Joerg
 
 - -- 
 The beginning is the most important part of the work.
  				-Plato
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.18 (FreeBSD)
 
 iD8DBQFPzf5zSPOsGF+KA+MRAvC3AJsFAEf8axpmvfu3VPUiaFprhIT6KwCfSKMI
 J1Ywq6NYvDeHHXiVjuWSRWw=
 =k6q6
 -----END PGP SIGNATURE-----
>Unformatted:
