From nobody@FreeBSD.org  Mon Sep 16 06:12:42 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id 47E3A20C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 16 Sep 2013 06:12:42 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from oldred.freebsd.org (oldred.freebsd.org [8.8.178.121])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id 272F92F07
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 16 Sep 2013 06:12:42 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id r8G6CfIQ006658
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 16 Sep 2013 06:12:41 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id r8G6CfMZ006649;
	Mon, 16 Sep 2013 06:12:41 GMT
	(envelope-from nobody)
Message-Id: <201309160612.r8G6CfMZ006649@oldred.freebsd.org>
Date: Mon, 16 Sep 2013 06:12:41 GMT
From: Berend de Ber <berend@pobox.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Very frequent (at most 3 hours) kernel trap 12
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         182141
>Category:       kern
>Synopsis:       [pf] crash in pf_test_rule
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    glebius
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Sep 16 06:20:00 UTC 2013
>Closed-Date:    Wed Jan 22 10:45:37 UTC 2014
>Last-Modified:  Wed Jan 22 10:45:37 UTC 2014
>Originator:     Berend de Ber
>Release:        FreeBSD 9.2-PRELEASE
>Organization:
Xplain Hosting
>Environment:
FreeBSD bmach.nederware.nl 9.2-PRERELEASE FreeBSD 9.2-PRERELEASE #25: Mon Sep 16 15:55:10 NZST 2013     root@bmach.nederware.nl:/usr/obj/usr/src/sys/BMACH  i386
>Description:
	

    Sep 16 17:26:24 bmach kernel: Fatal trap 12: page fault while in kernel mode
    Sep 16 17:26:24 bmach kernel: cpuid = 0; apic id = 00
    Sep 16 17:26:24 bmach kernel: fault virtual address     = 0x10
    Sep 16 17:26:24 bmach kernel: fault code                = supervisor read, page not present
    Sep 16 17:26:24 bmach kernel: instruction pointer       = 0x20:0xc05343be
    Sep 16 17:26:24 bmach kernel: stack pointer             = 0x28:0xf0764798
    Sep 16 17:26:24 bmach kernel: frame pointer             = 0x28:0xf0764798
    Sep 16 17:26:24 bmach kernel: stack pointer             = 0x28:0xf0764798
    Sep 16 17:26:24 bmach kernel: frame pointer             = 0x28:0xf0764798
    Sep 16 17:26:24 bmach kernel: code segment              = base 0x0, limit 0xfffff, type 0x1b
    Sep 16 17:26:24 bmach kernel: = DPL 0, pres 1, def32 1, gran 1
    Sep 16 17:26:24 bmach kernel: processor eflags  = interrupt enabled, resume, IOPL = 0
    Sep 16 17:26:24 bmach kernel: current process           = 2330 (named)
    Sep 16 17:26:24 bmach kernel: trap number               = 12
    Sep 16 17:26:24 bmach kernel: panic: page fault
    Sep 16 17:26:24 bmach kernel: cpuid = 0
    Sep 16 17:26:24 bmach kernel: KDB: stack backtrace:
    Sep 16 17:26:24 bmach kernel: #0 0xc0b27b78 at kdb_backtrace+0x4b
    Sep 16 17:26:24 bmach kernel: #1 0xc0af0b39 at panic+0x16f
    Sep 16 17:26:24 bmach kernel: #2 0xc0f8cb62 at trap_fatal+0x324
    Sep 16 17:26:24 bmach kernel: #3 0xc0f8cc47 at trap_pfault+0xd9
    Sep 16 17:26:24 bmach kernel: #4 0xc0f8dfa9 at trap+0x571
    Sep 16 17:26:24 bmach kernel: #5 0xc0f77c8c at calltrap+0x6
    Sep 16 17:26:24 bmach kernel: #6 0xc053f9ab at pf_test_rule+0x1be5
    Sep 16 17:26:24 bmach kernel: #7 0xc054212a at pf_test+0x84d
    Sep 16 17:26:24 bmach kernel: #8 0xc0547fa8 at pf_check_out+0x5c
    Sep 16 17:26:24 bmach kernel: #9 0xc0bb8e84 at pfil_run_hooks+0x74
    Sep 16 17:26:24 bmach kernel: #10 0xc0c1caba at ip_output+0x85d
    ep 16 17:26:24 bmach kernel: #9 0xc0bb8e84 at pfil_run_hooks+0x74
    Sep 16 17:26:24 bmach kernel: #10 0xc0c1caba at ip_output+0x85d
    Sep 16 17:26:24 bmach kernel: #11 0xc0c9df13 at udp_send+0x93c
    Sep 16 17:26:24 bmach kernel: #12 0xc0b5bb39 at sosend_dgram+0x467
    Sep 16 17:26:24 bmach kernel: #13 0xc0b57157 at sosend+0x3f
    Sep 16 17:26:24 bmach kernel: #14 0xc0b5e7c3 at kern_sendit+0x1ca
    Sep 16 17:26:24 bmach kernel: #15 0xc0b5e9e1 at sendit+0xad
    Sep 16 17:26:24 bmach kernel: #16 0xc0b5ea6f at sys_sendmsg+0x6b
    Sep 16 17:26:24 bmach kernel: #17 0xc0f8d41c at syscall+0x446
    Sep 16 17:26:24 bmach kernel: Uptime: 56m23s


>How-To-Repeat:
Custom kernel, but this is basically GENERIC + some extras like IPFW/PF.

Happens very frequently.

Server did run FreeBSD 8.2-STABLE before without much problems.
>Fix:


>Release-Note:
>Audit-Trail:

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Tue, 17 Sep 2013 10:07:05 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --6962t8n17TKfX9ETwspm5Cb5QOw05VJXU
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 The next one:
 
 Sep 17 10:01:27 bmach kernel: Fatal trap 12: page fault while in kernel m=
 ode
 Sep 17 10:01:27 bmach kernel: cpuid =3D 1; apic id =3D 01
 Sep 17 10:01:27 bmach kernel: fault virtual address     =3D 0x10
 Sep 17 10:01:27 bmach kernel: fault code                =3D supervisor
 read, page not present
 Sep 17 10:01:27 bmach kernel: instruction pointer       =3D 0x20:0xc05343=
 be
 Sep 17 10:01:27 bmach kernel: stack pointer             =3D 0x28:0xf07677=
 98
 Sep 17 10:01:27 bmach kernel: frame pointer             =3D 0x28:0xf07677=
 98
 Sep 17 10:01:27 bmach kernel: code segment              =3D base 0x0,
 limit 0xfffff, type 0x1b
 Sep 17 10:01:27 bmach kernel: =3D DPL 0, pres 1, def32 1, gran 1
 Sep 17 10:01:27 bmach kernel: processor eflags  =3D interrupt enabled,
 resume, IOPL =3D 0
 Sep 17 10:01:27 bmach kernel: current process           =3D 2343 (named)
 Sep 17 10:01:27 bmach kernel: trap number               =3D 12
 Sep 17 10:01:27 bmach kernel: panic: page fault
 Sep 17 10:01:27 bmach kernel: cpuid =3D 1
 Sep 17 10:01:27 bmach kernel: KDB: stack backtrace:
 Sep 17 10:01:27 bmach kernel: #0 0xc0b27b78 at kdb_backtrace+0x4b
 Sep 17 10:01:27 bmach kernel: #1 0xc0af0b39 at panic+0x16f
 Sep 17 10:01:27 bmach kernel: #2 0xc0f8cb62 at trap_fatal+0x324
 Sep 17 10:01:27 bmach kernel: #3 0xc0f8cc47 at trap_pfault+0xd9
 Sep 17 10:01:27 bmach kernel: #4 0xc0f8dfa9 at trap+0x571
 Sep 17 10:01:27 bmach kernel: #5 0xc0f77c8c at calltrap+0x6
 Sep 17 10:01:27 bmach kernel: #6 0xc053f9ab at pf_test_rule+0x1be5
 Sep 17 10:01:27 bmach kernel: #7 0xc054212a at pf_test+0x84d
 Sep 17 10:01:27 bmach kernel: #8 0xc0547fa8 at pf_check_out+0x5c
 Sep 17 10:01:27 bmach kernel: #9 0xc0bb8e84 at pfil_run_hooks+0x74
 Sep 17 10:01:27 bmach kernel: #10 0xc0c1caba at ip_output+0x85d
 Sep 17 10:01:27 bmach kernel: #11 0xc0c9df13 at udp_send+0x93c
 Sep 17 10:01:27 bmach kernel: #12 0xc0b5bb39 at sosend_dgram+0x467
 Sep 17 10:01:27 bmach kernel: #13 0xc0b57157 at sosend+0x3f
 Sep 17 10:01:27 bmach kernel: #14 0xc0b5e7c3 at kern_sendit+0x1ca
 Sep 17 10:01:27 bmach kernel: #15 0xc0b5e9e1 at sendit+0xad
 Sep 17 10:01:27 bmach kernel: #16 0xc0b5ea6f at sys_sendmsg+0x6b
 Sep 17 10:01:27 bmach kernel: #17 0xc0f8d41c at syscall+0x446
 Sep 17 10:01:27 bmach kernel: Uptime: 16h33m57s
 
 
 --6962t8n17TKfX9ETwspm5Cb5QOw05VJXU
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSN4EJAAoJEKOfeD48G3g56VsP/2VzrJvURJcBG97eodCeMN4T
 ShwYlMVJyWXPOP2Yl+vRqURz2B5bFPfcFFvyNLGgvlx8eSuQPo8NeRLmBgxYJGiH
 hi3iM84p/IscfEK6LY84PL3vB23T7BKd14LduAGAcL4UoYC1wDBy/aRm0IqVE5K1
 KUAH2IvEEoIZ1yRUVTpXDwnMqqkgQ1pToY9sZVL9XQcsadGBJgX1kROvkWhSlSLj
 EaDokvPY9JkTdFEPjraIbaZsFBmjIkpnm43xABfkqoRETtR9KNrW3EiPZHwMFSof
 QDRAp7IY6jDvfaO+r8B7svD+zJNwGqCNqHbQUCmO3Eu+AzEuplP7CQ6q4CheNBVr
 EiwNsDt0DM7if7oQgiMJfyH+PQrxTh92WSKru9eKfvjJ6+wnD9usPnAkBk/E+dnF
 FU9kX1YLQR1+K+5yd1yPgrMP9qRSKjrqGyUlZmBYQz9B23Kz/i/Sr4pdkH+JJzh4
 SKuFNkP4jdLG5QSIphtEW9/pm2Lt5XGnu9wVnRaZHknrysNJmP7vO6lSddHXpiYt
 zMATbBYatl8ynYtZUwmcnRegUkfubqh0yo/hneuuJYuziUKAdDFlvEpvnZLWz5HT
 Q8kLbbNHc5gECyakmuANvW2DxHViwh5C+RUxJrEs6j0vLry3QLifwU54FtD6DQWV
 UZUMlPW9A+q1QPckVU58
 =08C9
 -----END PGP SIGNATURE-----
 
 --6962t8n17TKfX9ETwspm5Cb5QOw05VJXU--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Wed, 18 Sep 2013 07:49:28 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --T0EEtiDBJWIFpejXndaGIXnIJBWDbf9Kx
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 And another one.
 
 Sep 17 21:53:09 bmach kernel: Fatal trap 12: page fault while in kernel m=
 ode
 Sep 17 21:53:09 bmach kernel: cpuid =3D 1; apic id =3D 01
 Sep 17 21:53:09 bmach kernel: fault virtual address     =3D 0x10
 Sep 17 21:53:09 bmach kernel: fault code                =3D supervisor
 read, page not present
 Sep 17 21:53:09 bmach kernel: instruction pointer       =3D 0x20:0xc05343=
 be
 Sep 17 21:53:09 bmach kernel: stack pointer             =3D 0x28:0xf07647=
 98
 Sep 17 21:53:09 bmach kernel: frame pointer             =3D 0x28:0xf07647=
 98
 Sep 17 21:53:09 bmach kernel: code segment              =3D base 0x0,
 limit 0xfffff, type 0x1b
 Sep 17 21:53:09 bmach kernel: =3D DPL 0, pres 1, def32 1, gran 1
 Sep 17 21:53:09 bmach kernel: processor eflags  =3D interrupt enabled,
 resume, IOPL =3D 0
 Sep 17 21:53:09 bmach kernel: current process           =3D 2343 (named)
 Sep 17 21:53:09 bmach kernel: trap number               =3D 12
 Sep 17 21:53:09 bmach kernel: panic: page fault
 Sep 17 21:53:09 bmach kernel: cpuid =3D 1
 Sep 17 21:53:09 bmach kernel: KDB: stack backtrace:
 Sep 17 21:53:09 bmach kernel: #0 0xc0b27b78 at kdb_backtrace+0x4b
 Sep 17 21:53:09 bmach kernel: #1 0xc0af0b39 at panic+0x16f
 Sep 17 21:53:09 bmach kernel: #2 0xc0f8cb62 at trap_fatal+0x324
 Sep 17 21:53:09 bmach kernel: #3 0xc0f8cc47 at trap_pfault+0xd9
 Sep 17 21:53:09 bmach kernel: #4 0xc0f8dfa9 at trap+0x571
 Sep 17 21:53:09 bmach kernel: #5 0xc0f77c8c at calltrap+0x6
 Sep 17 21:53:09 bmach kernel: #6 0xc053f9ab at pf_test_rule+0x1be5
 Sep 17 21:53:09 bmach kernel: #7 0xc054212a at pf_test+0x84d
 Sep 17 21:53:09 bmach kernel: #8 0xc0547fa8 at pf_check_out+0x5c
 Sep 17 21:53:09 bmach kernel: #9 0xc0bb8e84 at pfil_run_hooks+0x74
 Sep 17 21:53:09 bmach kernel: #10 0xc0c1caba at ip_output+0x85d
 Sep 17 21:53:09 bmach kernel: #11 0xc0c9df13 at udp_send+0x93c
 Sep 17 21:53:09 bmach kernel: #12 0xc0b5bb39 at sosend_dgram+0x467
 Sep 17 21:53:09 bmach kernel: #13 0xc0b57157 at sosend+0x3f
 Sep 17 21:53:09 bmach kernel: #14 0xc0b5e7c3 at kern_sendit+0x1ca
 Sep 17 21:53:09 bmach kernel: #15 0xc0b5e9e1 at sendit+0xad
 Sep 17 21:53:09 bmach kernel: #16 0xc0b5ea6f at sys_sendmsg+0x6b
 Sep 17 21:53:09 bmach kernel: #17 0xc0f8d41c at syscall+0x446
 Sep 17 21:53:09 bmach kernel: Uptime: 11h50m23s
 
 
 Any pointers greatly appreciated.
 
 
 --T0EEtiDBJWIFpejXndaGIXnIJBWDbf9Kx
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSOLJdAAoJEKOfeD48G3g5J8AP/j8pq4NaLuPMz8ABg0SuDcdE
 6/vcPUvqKugLUmrXgxuirsi2IG1VyBZxECyUVAYV1OjP9hl6UOPAYDyjFcYzG0FL
 LkxyMTLFxz5bNgtBVyXkt9E5zLSNax5SLhCeBupoZR2XFeyzhlaNG2kQWFHDe3qT
 MlFYUseuonge9D1/ZhTpb3K7g5mpgzresfXaurJtGolDAPdYPEUzaFwn5PsC5sXO
 4POqvQ7OkZJWBOHI9mnW2/J50Y1IjHmeqlsx+p9wbghTo0GivxYpXQTPwn1ESe6r
 rNa40649NAqN9LFPsJM9a11M5JJ5595UPqXESTGr+zw9a8Lf3tPhsj4LTDPvBRzl
 rxpL4eewEA1hqBuEUsha4ovS+XRUh5W8jkGvflVq7degJxlouKgXIeQuI/7LoHCT
 t/mdXNiiRl+v0g9Qj2frDQU07R9v3eD6mFiIQDzBQeNOCATeql11NpNd+upAOlhp
 FbgQgDY1/EVXz3Y46RAQ3hmnVVYhmjJQ/lMaad1Fn80E+vW5Tps1tFJOMUQ1X+A0
 gt6wEBdP3HsPjE/lwJBvBe8EQGfV8LI6jzDV2Y0vwKvbB1YrqaFNMIiu19ha3W8q
 PbOrDrw46L96mJWrH8Rka+Yp050RKeFdGr0yzWdc+bVjvF8I9ol9UWtUdtIMLelR
 DqmU5VAXhj+edakp4iQZ
 =gibp
 -----END PGP SIGNATURE-----
 
 --T0EEtiDBJWIFpejXndaGIXnIJBWDbf9Kx--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Wed, 18 Sep 2013 21:39:35 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --9qfsKgCMUbdjOVQRWvcgIrPbUjrRqdolN
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 As requested by freebsdhelp: revision updated to is latest (retrieved
 using cvsup): default release=3Dcvs tag=3DRELENG_9
 
 /var/run/dmesg.boot:
 
 fxp0: Ethernet address: 00:90:27:17:93:5d
 fwohci0: <Texas Instruments TSB43AB22/A> mem
 0xe3344000-0xe33447ff,0xe3340000-0xe3343fff irq 19 at device 3.0 on pci7
 fwohci0: OHCI version 1.10 (ROM=3D0)
 fwohci0: No. of Isochronous channels is 4.
 fwohci0: EUI64 00:90:27:00:01:df:6e:98
 fwohci0: Phy 1394a available S400, 2 ports.
 fwohci0: Link S400, max_rec 2048 bytes.
 firewire0: <IEEE1394(FireWire) bus> on fwohci0
 fwe0: <Ethernet over FireWire> on firewire0
 if_fwe0: Fake Ethernet address: 02:90:27:df:6e:98
 fwe0: Ethernet address: 02:90:27:df:6e:98
 fwip0: <IP over FireWire> on firewire0
 fwip0: Firewire address: 00:90:27:00:01:df:6e:98 @ 0xfffe00000000, S400,
 maxrec 2048
 dcons_crom0: <dcons configuration ROM> on firewire0
 dcons_crom0: bus_addr 0x145c000
 fwohci0: Initiate bus reset
 fwohci0: fwohci_intr_core: BUS reset
 fwohci0: fwohci_intr_core: node_id=3D0x00000000, SelfID Count=3D1,
 CYCLEMASTER mode
 isab0: <PCI-ISA bridge> at device 31.0 on pci0
 isa0: <ISA bus> on isab0
 ahci0: <Intel ICH8 AHCI SATA controller> port
 0x5408-0x540f,0x5414-0x5417,0x5400-0x5407,0x5410-0x5413,0x5020-0x503f
 mem 0xe3525000-0xe35257ff irq 19 at device 31.2 on pci0
 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported
 ahcich0: <AHCI channel> at channel 0 on ahci0
 ahcich1: <AHCI channel> at channel 1 on ahci0
 ahcich4: <AHCI channel> at channel 4 on ahci0
 ahcich5: <AHCI channel> at channel 5 on ahci0
 pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
 ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 on acpi0
 ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
 ppc0: FIFO with 16/16/8 bytes threshold
 ppbus0: <Parallel port bus> on ppc0
 plip0: <PLIP network interface> on ppbus0
 lpt0: <Printer> on ppbus0
 lpt0: Interrupt-driven port
 ppi0: <Parallel I/O> on ppbus0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 0xcf800-0xd07ff,0xd0800-0xd0fff pnpid
 ORM0000 on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=3D0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0=
 
 ata0: <ATA channel> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
 ata1: <ATA channel> at port 0x170-0x177,0x376 irq 15 on isa0
 est0: <Enhanced SpeedStep Frequency Control> on cpu0
 p4tcc0: <CPU Frequency Thermal Control> on cpu0
 est1: <Enhanced SpeedStep Frequency Control> on cpu1
 p4tcc1: <CPU Frequency Thermal Control> on cpu1
 firewire0: 1 nodes, maxhop <=3D 0 cable IRM irm(0)  (me)
 firewire0: bus manager 0
 Timecounters tick every 1.000 msec
 hdacc0: <Sigmatel STAC9227X HDA CODEC> at cad 2 on hdac0
 hdaa0: <Sigmatel STAC9227X Audio Function Group> at nid 1 on hdacc0
 pcm0: <Sigmatel STAC9227X (Analog 5.1+HP/2.0)> at nid 13,15,14,10 and 11
 on hdaa0
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 usbus3: 12Mbps Full Speed USB v1.0
 usbus4: 12Mbps Full Speed USB v1.0
 usbus5: 12Mbps Full Speed USB v1.0
 usbus6: 480Mbps High Speed USB v2.0
 ugen0.1: <Intel> at usbus0
 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 ugen1.1: <Intel> at usbus1
 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 ugen2.1: <Intel> at usbus2
 uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 ugen3.1: <Intel> at usbus3
 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
 ugen4.1: <Intel> at usbus4
 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
 ugen5.1: <Intel> at usbus5
 uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
 ugen6.1: <Intel> at usbus6
 uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
 uhub0: 2 ports with 2 removable, self powered
 uhub1: 2 ports with 2 removable, self powered
 uhub3: 2 ports with 2 removable, self powered
 uhub4: 2 ports with 2 removable, self powered
 uhub5: 2 ports with 2 removable, self powered
 uhub2: 4 ports with 4 removable, self powered
 uhub6: 6 ports with 6 removable, self powered
 ugen2.2: <Hewlett-Packard> at usbus2
 ulpt0: <Printer> on usbus2
 ulpt0: using bi-directional mode
 ugen3.2: <vendor 0x0001> at usbus3
 uhid0: <vendor 0x0001 product 0x0000, class 0/0, rev 1.00/1.00, addr 2>
 on usbus3
 ugen2.3: <Sunplus Technology Inc.> at usbus2
 umass0: <Bulk Only Interface> on usbus2
 umass0:  SCSI over Bulk-Only; quirks =3D 0x4101
 umass0:10:0:-1: Attached to scbus10
 sa0 at ahc1 bus 0 scbus2 target 6 lun 0
 sa0: <SEAGATE DAT    9SP40-000 910B> Removable Sequential Access SCSI-3
 device
 sa0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
 ada0 at ata2 bus 0 scbus0 target 0 lun 0
 ada0: <ST3160815A 3.AAD> ATA-7 device
 ada0: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
 ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
 ada0: Previously was known as ad4
 cd0 at ata2 bus 0 scbus0 target 1 lun 0
 cd0: <Optiarc DVD RW AD-7170A 1.85> Removable CD-ROM SCSI-0 device
 cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
 cd0: cd present [292819 x 2048 byte records]
 da0 at umass-sim0 bus 0 scbus10 target 0 lun 0
 da0: <Hitachi HTS721010G9SA00 > Fixed Direct Access SCSI-2 device
 da0: 40.000MB/s transfers
 da0: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C)
 ada1 at ahcich0 bus 0 scbus3 target 0 lun 0
 ada1: <ST500DM002-1BD142 KC44> ATA-8 SATA 3.x device
 ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 ada1: Command Queueing enabled
 ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
 ada1: Previously was known as ad6
 SMP: AP CPU #1 Launched!
 Timecounter "TSC-low" frequency 15609752 Hz quality 1000
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 0 0 4 77 d2 0 0 1 0
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 Trying to mount root from cd9660:/dev/iso9660/FREEBSD_INSTALL [ro]...
 WARNING: / was not properly dismounted
 Waiting (max 60 seconds) for system process `vnlru' to stop...done
 Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
 Waiting (max 60 seconds) for system process `syncer' to stop...
 Syncing disks, vnodes remaining...0 done
 All buffers synced.
 Uptime: 13m53s
 usbus0: Controller shutdown
 uhub0: at usbus0, port 1, addr 1 (disconnected)
 usbus0: Controller shutdown complete
 usbus1: Controller shutdown
 uhub1: at usbus1, port 1, addr 1 (disconnected)
 usbus1: Controller shutdown complete
 usbus2: Controller shutdown
 uhub2: at usbus2, port 1, addr 1 (disconnected)
 ugen2.2: <Hewlett-Packard> at usbus2 (disconnected)
 ulpt0: at uhub2, port 3, addr 2 (disconnected)
 ugen2.3: <Sunplus Technology Inc.> at usbus2 (disconnected)
 umass0: at uhub2, port 4, addr 3 (disconnected)
 (da0:umass-sim0:0:0:0): lost device - 0 outstanding, 1 refs
 (pass4:(da0:umass-sim0:0:umass-sim0:0:0:0:0): passdevgonecb: devfs entry
 is gone
 0): removing device entry
 usbus2: Controller shutdown complete
 usbus3: Controller shutdown
 uhub3: at usbus3, port 1, addr 1 (disconnected)
 ugen3.2: <vendor 0x0001> at usbus3 (disconnected)
 uhid0: at uhub3, port 2, addr 2 (disconnected)
 usbus3: Controller shutdown complete
 usbus4: Controller shutdown
 uhub4: at usbus4, port 1, addr 1 (disconnected)
 usbus4: Controller shutdown complete
 usbus5: Controller shutdown
 uhub5: at usbus5, port 1, addr 1 (disconnected)
 usbus5: Controller shutdown complete
 usbus6: Controller shutdown
 uhub6: at usbus6, port 1, addr 1 (disconnected)
 usbus6: Controller shutdown complete
 Rebooting...
 cpu_reset: Stopping other CPUs
 Copyright (c) 1992-2013 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 9.2-PRERELEASE #26: Wed Sep 18 11:37:44 NZST 2013
     root@bmach.nederware.nl:/usr/obj/usr/src/sys/BMACH i386
 gcc version 4.2.1 20070831 patched [FreeBSD]
 link_elf: symbol ata_controlcmd undefined
 KLD file atapicam.ko - could not finalize loading
 CPU: Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz (1998.04-MHz
 686-class CPU)
   Origin =3D "GenuineIntel"  Id =3D 0x6f2  Family =3D 0x6  Model =3D 0xf
 Stepping =3D 2
 
 Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG=
 E,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=3D0xe39d<SSE3,DTES64,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM>=
 
   AMD Features=3D0x20100000<NX,LM>
   AMD Features2=3D0x1<LAHF>
   TSC: P-state invariant, performance statistics
 real memory  =3D 4294967296 (4096 MB)
 avail memory =3D 3399114752 (3241 MB)
 Event timer "LAPIC" quality 400
 ACPI APIC Table: <INTEL  DP965LT >
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 FreeBSD/SMP: 1 package(s) x 2 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
 ioapic0: Changing APIC ID to 2
 ioapic0 <Version 2.0> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 cryptosoft0: <software crypto> on motherboard
 acpi0: <INTEL DP965LT> on motherboard
 acpi0: Power Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
 Event timer "RTC" frequency 32768 Hz quality 0
 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
 Timecounter "i8254" frequency 1193182 Hz quality 0
 Event timer "i8254" frequency 1193182 Hz quality 100
 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed03fff on acpi0
 Timecounter "HPET" frequency 14318180 Hz quality 950
 Event timer "HPET" frequency 14318180 Hz quality 450
 Event timer "HPET1" frequency 14318180 Hz quality 440
 Event timer "HPET2" frequency 14318180 Hz quality 440
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
 acpi_button0: <Sleep Button> on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
 pci1: <ACPI PCI bus> on pcib1
 vgapci0: <VGA-compatible display> port 0x4000-0x407f mem
 0xe2000000-0xe2ffffff,0xd0000000-0xdfffffff,0xe0000000-0xe1ffffff irq 16
 at device 0.0 on pci1
 pci0: <simple comms> at device 3.0 (no driver attached)
 em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x50e0-0x50ff mem
 0xe3500000-0xe351ffff,0xe3524000-0xe3524fff irq 20 at device 25.0 on pci0=
 
 em0: Using an MSI interrupt
 em0: Ethernet address: 00:19:d1:e8:c7:b3
 uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0x50c0-0x50df irq
 16 at device 26.0 on pci0
 uhci0: LegSup =3D 0x2f00
 usbus0 on uhci0
 uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0x50a0-0x50bf irq
 21 at device 26.1 on pci0
 uhci1: LegSup =3D 0x2f00
 usbus1 on uhci1
 ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem
 0xe3525c00-0xe3525fff irq 18 at device 26.7 on pci0
 usbus2: EHCI version 1.0
 usbus2 on ehci0
 hdac0: <Intel 82801H HDA Controller> mem 0xe3520000-0xe3523fff irq 22 at
 device 27.0 on pci0
 pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0
 pci2: <ACPI PCI bus> on pcib2
 pcib3: <ACPI PCI-PCI bridge> at device 28.1 on pci0
 pci3: <ACPI PCI bus> on pcib3
 atapci0: <Marvell 88SE6101 UDMA133 controller> port
 0x3018-0x301f,0x3024-0x3027,0x3010-0x3017,0x3020-0x3023,0x3000-0x300f
 mem 0xe3400000-0xe34001ff irq 17 at device 0.0 on pci3
 ata2: <ATA channel> at channel 0 on atapci0
 pcib4: <ACPI PCI-PCI bridge> at device 28.2 on pci0
 pci4: <ACPI PCI bus> on pcib4
 pcib5: <ACPI PCI-PCI bridge> at device 28.3 on pci0
 pci5: <ACPI PCI bus> on pcib5
 pcib6: <ACPI PCI-PCI bridge> at device 28.4 on pci0
 pci6: <ACPI PCI bus> on pcib6
 uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0x5080-0x509f irq
 23 at device 29.0 on pci0
 uhci2: LegSup =3D 0x2f00
 usbus3 on uhci2
 uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0x5060-0x507f irq
 19 at device 29.1 on pci0
 uhci3: LegSup =3D 0x2f00
 usbus4 on uhci3
 uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0x5040-0x505f irq
 18 at device 29.2 on pci0
 uhci4: LegSup =3D 0x2f00
 usbus5 on uhci4
 ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem
 0xe3525800-0xe3525bff irq 23 at device 29.7 on pci0
 usbus6: EHCI version 1.0
 usbus6 on ehci1
 pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
 pci7: <ACPI PCI bus> on pcib7
 em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port
 0x2000-0x203f mem 0xe3320000-0xe333ffff,0xe3300000-0xe331ffff irq 21 at
 device 0.0 on pci7
 em1: Ethernet address: 00:1b:21:51:e1:e0
 pcib8: <PCI-PCI bridge> at device 1.0 on pci7
 pci8: <PCI bus> on pcib8
 ahc0: <Adaptec 3940 Ultra SCSI adapter> port 0x1400-0x14ff mem
 0xe3201000-0xe3201fff irq 22 at device 4.0 on pci8
 aic7880: Ultra Wide Channel A, SCSI Id=3D7, 16/253 SCBs
 ahc1: <Adaptec 3940 Ultra SCSI adapter> port 0x1000-0x10ff mem
 0xe3200000-0xe3200fff irq 21 at device 5.0 on pci8
 aic7880: Ultra Wide Channel B, SCSI Id=3D0, 16/253 SCBs
 fxp0: <Intel 82558 Pro/100 Ethernet> port 0x2040-0x205f mem
 0xe3000000-0xe3000fff,0xe3100000-0xe31fffff irq 18 at device 2.0 on pci7
 miibus0: <MII bus> on fxp0
 inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
 fxp0: Ethernet address: 00:90:27:17:93:5d
 pci7: <serial bus, FireWire> at device 3.0 (no driver attached)
 isab0: <PCI-ISA bridge> at device 31.0 on pci0
 isa0: <ISA bus> on isab0
 ahci0: <Intel ICH8 AHCI SATA controller> port
 0x5408-0x540f,0x5414-0x5417,0x5400-0x5407,0x5410-0x5413,0x5020-0x503f
 mem 0xe3525000-0xe35257ff irq 19 at device 31.2 on pci0
 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported
 ahcich0: <AHCI channel> at channel 0 on ahci0
 ahcich1: <AHCI channel> at channel 1 on ahci0
 ahcich4: <AHCI channel> at channel 4 on ahci0
 ahcich5: <AHCI channel> at channel 5 on ahci0
 pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
 ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 on acpi0
 ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
 ppc0: FIFO with 16/16/8 bytes threshold
 ppbus0: <Parallel port bus> on ppc0
 plip0: <PLIP network interface> on ppbus0
 lpt0: <Printer> on ppbus0
 lpt0: Interrupt-driven port
 ppi0: <Parallel I/O> on ppbus0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 ichwd0 on isa0
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 0xcf800-0xd07ff,0xd0800-0xd0fff pnpid
 ORM0000 on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=3D0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0=
 
 ata0: <ATA channel> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
 ata1: <ATA channel> at port 0x170-0x177,0x376 irq 15 on isa0
 coretemp0: <CPU On-Die Thermal Sensors> on cpu0
 est0: <Enhanced SpeedStep Frequency Control> on cpu0
 p4tcc0: <CPU Frequency Thermal Control> on cpu0
 coretemp1: <CPU On-Die Thermal Sensors> on cpu1
 est1: <Enhanced SpeedStep Frequency Control> on cpu1
 p4tcc1: <CPU Frequency Thermal Control> on cpu1
 Timecounters tick every 1.000 msec
 vboxdrv: fAsync=3D0 offMin=3D0x1e0 offMax=3D0x46a
 IPsec: Initialized Security Association Processing.
 ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny,
 logging disabled
 DUMMYNET 0 with IPv6 initialized (100409)
 load_dn_sched dn_sched FIFO loaded
 load_dn_sched dn_sched PRIO loaded
 load_dn_sched dn_sched QFQ loaded
 load_dn_sched dn_sched RR loaded
 load_dn_sched dn_sched WF2Q+ loaded
 hdacc0: <Sigmatel STAC9227X HDA CODEC> at cad 2 on hdac0
 hdaa0: <Sigmatel STAC9227X Audio Function Group> at nid 1 on hdacc0
 pcm0: <Sigmatel STAC9227X (Analog 5.1+HP/2.0)> at nid 13,15,14,10 and 11
 on hdaa0
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 usbus3: 12Mbps Full Speed USB v1.0
 usbus4: 12Mbps Full Speed USB v1.0
 usbus5: 12Mbps Full Speed USB v1.0
 usbus6: 480Mbps High Speed USB v2.0
 ugen0.1: <Intel> at usbus0
 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 ugen1.1: <Intel> at usbus1
 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 ugen2.1: <Intel> at usbus2
 uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 ugen3.1: <Intel> at usbus3
 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
 ugen4.1: <Intel> at usbus4
 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
 ugen5.1: <Intel> at usbus5
 uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
 ugen6.1: <Intel> at usbus6
 uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
 uhub0: 2 ports with 2 removable, self powered
 uhub1: 2 ports with 2 removable, self powered
 uhub3: 2 ports with 2 removable, self powered
 uhub4: 2 ports with 2 removable, self powered
 uhub5: 2 ports with 2 removable, self powered
 uhub2: 4 ports with 4 removable, self powered
 uhub6: 6 ports with 6 removable, self powered
 ugen2.2: <Hewlett-Packard> at usbus2
 ulpt0: <Printer> on usbus2
 ulpt0: using bi-directional mode
 ugen3.2: <vendor 0x0001> at usbus3
 uhid0: <vendor 0x0001 product 0x0000, class 0/0, rev 1.00/1.00, addr 2>
 on usbus3
 ugen2.3: <Sunplus Technology Inc.> at usbus2
 umass0: <Bulk Only Interface> on usbus2
 umass0:  SCSI over Bulk-Only; quirks =3D 0x4101
 umass0:9:0:-1: Attached to scbus9
 sa0 at ahc1 bus 0 scbus2 target 6 lun 0
 sa0: <SEAGATE DAT    9SP40-000 910B> Removable Sequential Access SCSI-3
 device
 sa0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
 ada0 at ata2 bus 0 scbus0 target 0 lun 0
 ada0: da0 at umass-sim0 bus 0 scbus9 target 0 lun 0
 <ST3160815A 3.AAD> ATA-7 device
 ada0: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
 ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
 ada0: Previously was known as ad4
 cd0 at ata2 bus 0 scbus0 target 1 lun 0
 cd0: <Optiarc DVD RW AD-7170A 1.85> Removable CD-ROM SCSI-0 device
 cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
 cd0: cd present [292819 x 2048 byte records]
 da0: <Hitachi HTS721010G9SA00 > Fixed Direct Access SCSI-2 device
 da0: 40.000MB/s transfers
 da0: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C)
 da0: quirks=3D0x2<NO_6_BYTE>
 ada1 at ahcich0 bus 0 scbus3 target 0 lun 0
 ada1: <ST500DM002-1BD142 KC44> ATA-8 SATA 3.x device
 ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 ada1: Command Queueing enabled
 ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
 ada1: quirks=3D0x1<4K>
 ada1: Previously was known as ad6
 SMP: AP CPU #1 Launched!
 Timecounter "TSC" frequency 1998041860 Hz quality 1000
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 Trying to mount root from ufs:/dev/ad4s1a [rw,noatime]...
 WARNING: /tmp was not properly dismounted
 WARNING: /usr was not properly dismounted
 WARNING: /usr: mount pending error: blocks 580 files 28
 WARNING: /var was not properly dismounted
 WARNING: /home was not properly dismounted
 tun0: link state changed to UP
 tun1: link state changed to UP
 WARNING: attempt to domain_add(netgraph) after domainfinalize()
 
 
 Fatal trap 12: page fault while in kernel mode
 cpuid =3D 1; apic id =3D 01
 fault virtual address	=3D 0x10
 fault code		=3D supervisor read, page not present
 instruction pointer	=3D 0x20:0xc05343be
 stack pointer	        =3D 0x28:0xf0767798
 frame pointer	        =3D 0x28:0xf0767798
 code segment		=3D base 0x0, limit 0xfffff, type 0x1b
 			=3D DPL 0, pres 1, def32 1, gran 1
 processor eflags	=3D interrupt enabled, resume, IOPL =3D 0
 current process		=3D 2315 (named)
 trap number		=3D 12
 panic: page fault
 cpuid =3D 1
 KDB: stack backtrace:
 #0 0xc0b27b58 at kdb_backtrace+0x4b
 #1 0xc0af0b19 at panic+0x16f
 #2 0xc0f8cb22 at trap_fatal+0x324
 #3 0xc0f8cc07 at trap_pfault+0xd9
 #4 0xc0f8df69 at trap+0x571
 #5 0xc0f77c4c at calltrap+0x6
 #6 0xc053f9ab at pf_test_rule+0x1be5
 #7 0xc054212a at pf_test+0x84d
 #8 0xc0547fa8 at pf_check_out+0x5c
 #9 0xc0bb8e64 at pfil_run_hooks+0x74
 #10 0xc0c1ca7a at ip_output+0x85d
 #11 0xc0c9ded3 at udp_send+0x93c
 #12 0xc0b5bb19 at sosend_dgram+0x467
 #13 0xc0b57137 at sosend+0x3f
 #14 0xc0b5e7a3 at kern_sendit+0x1ca
 #15 0xc0b5e9c1 at sendit+0xad
 #16 0xc0b5ea4f at sys_sendmsg+0x6b
 #17 0xc0f8d3dc at syscall+0x446
 Uptime: 7m38s
 Automatic reboot in 15 seconds - press a key on the console to abort
 Rebooting...
 cpu_reset: Restarting BSP
 cpu_reset_proxy: Stopped CPU 1
 Copyright (c) 1992-2013 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 9.2-PRERELEASE #26: Wed Sep 18 11:37:44 NZST 2013
     root@bmach.nederware.nl:/usr/obj/usr/src/sys/BMACH i386
 gcc version 4.2.1 20070831 patched [FreeBSD]
 link_elf: symbol ata_controlcmd undefined
 KLD file atapicam.ko - could not finalize loading
 CPU: Intel(R) Core(TM)2 CPU          4400  @ 2.00GHz (1998.04-MHz
 686-class CPU)
   Origin =3D "GenuineIntel"  Id =3D 0x6f2  Family =3D 0x6  Model =3D 0xf
 Stepping =3D 2
 
 Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG=
 E,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=3D0xe39d<SSE3,DTES64,MON,DS_CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM>=
 
   AMD Features=3D0x20100000<NX,LM>
   AMD Features2=3D0x1<LAHF>
   TSC: P-state invariant, performance statistics
 real memory  =3D 4294967296 (4096 MB)
 avail memory =3D 3399114752 (3241 MB)
 Event timer "LAPIC" quality 400
 ACPI APIC Table: <INTEL  DP965LT >
 FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 FreeBSD/SMP: 1 package(s) x 2 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
 ioapic0: Changing APIC ID to 2
 ioapic0 <Version 2.0> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 cryptosoft0: <software crypto> on motherboard
 acpi0: <INTEL DP965LT> on motherboard
 acpi0: Power Button (fixed)
 cpu0: <ACPI CPU> on acpi0
 cpu1: <ACPI CPU> on acpi0
 atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
 Event timer "RTC" frequency 32768 Hz quality 0
 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
 Timecounter "i8254" frequency 1193182 Hz quality 0
 Event timer "i8254" frequency 1193182 Hz quality 100
 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed03fff on acpi0
 Timecounter "HPET" frequency 14318180 Hz quality 950
 Event timer "HPET" frequency 14318180 Hz quality 450
 Event timer "HPET1" frequency 14318180 Hz quality 440
 Event timer "HPET2" frequency 14318180 Hz quality 440
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
 acpi_button0: <Sleep Button> on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
 pci1: <ACPI PCI bus> on pcib1
 vgapci0: <VGA-compatible display> port 0x4000-0x407f mem
 0xe2000000-0xe2ffffff,0xd0000000-0xdfffffff,0xe0000000-0xe1ffffff irq 16
 at device 0.0 on pci1
 pci0: <simple comms> at device 3.0 (no driver attached)
 em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x50e0-0x50ff mem
 0xe3500000-0xe351ffff,0xe3524000-0xe3524fff irq 20 at device 25.0 on pci0=
 
 em0: Using an MSI interrupt
 em0: Ethernet address: 00:19:d1:e8:c7:b3
 uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0x50c0-0x50df irq
 16 at device 26.0 on pci0
 uhci0: LegSup =3D 0x2f00
 usbus0 on uhci0
 uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0x50a0-0x50bf irq
 21 at device 26.1 on pci0
 uhci1: LegSup =3D 0x2f00
 usbus1 on uhci1
 ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem
 0xe3525c00-0xe3525fff irq 18 at device 26.7 on pci0
 usbus2: EHCI version 1.0
 usbus2 on ehci0
 hdac0: <Intel 82801H HDA Controller> mem 0xe3520000-0xe3523fff irq 22 at
 device 27.0 on pci0
 pcib2: <ACPI PCI-PCI bridge> at device 28.0 on pci0
 pci2: <ACPI PCI bus> on pcib2
 pcib3: <ACPI PCI-PCI bridge> at device 28.1 on pci0
 pci3: <ACPI PCI bus> on pcib3
 atapci0: <Marvell 88SE6101 UDMA133 controller> port
 0x3018-0x301f,0x3024-0x3027,0x3010-0x3017,0x3020-0x3023,0x3000-0x300f
 mem 0xe3400000-0xe34001ff irq 17 at device 0.0 on pci3
 ata2: <ATA channel> at channel 0 on atapci0
 pcib4: <ACPI PCI-PCI bridge> at device 28.2 on pci0
 pci4: <ACPI PCI bus> on pcib4
 pcib5: <ACPI PCI-PCI bridge> at device 28.3 on pci0
 pci5: <ACPI PCI bus> on pcib5
 pcib6: <ACPI PCI-PCI bridge> at device 28.4 on pci0
 pci6: <ACPI PCI bus> on pcib6
 uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0x5080-0x509f irq
 23 at device 29.0 on pci0
 uhci2: LegSup =3D 0x2f00
 usbus3 on uhci2
 uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0x5060-0x507f irq
 19 at device 29.1 on pci0
 uhci3: LegSup =3D 0x2f00
 usbus4 on uhci3
 uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0x5040-0x505f irq
 18 at device 29.2 on pci0
 uhci4: LegSup =3D 0x2f00
 usbus5 on uhci4
 ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem
 0xe3525800-0xe3525bff irq 23 at device 29.7 on pci0
 usbus6: EHCI version 1.0
 usbus6 on ehci1
 pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
 pci7: <ACPI PCI bus> on pcib7
 em1: <Intel(R) PRO/1000 Legacy Network Connection 1.0.6> port
 0x2000-0x203f mem 0xe3320000-0xe333ffff,0xe3300000-0xe331ffff irq 21 at
 device 0.0 on pci7
 em1: Ethernet address: 00:1b:21:51:e1:e0
 pcib8: <PCI-PCI bridge> at device 1.0 on pci7
 pci8: <PCI bus> on pcib8
 ahc0: <Adaptec 3940 Ultra SCSI adapter> port 0x1400-0x14ff mem
 0xe3201000-0xe3201fff irq 22 at device 4.0 on pci8
 aic7880: Ultra Wide Channel A, SCSI Id=3D7, 16/253 SCBs
 ahc1: <Adaptec 3940 Ultra SCSI adapter> port 0x1000-0x10ff mem
 0xe3200000-0xe3200fff irq 21 at device 5.0 on pci8
 aic7880: Ultra Wide Channel B, SCSI Id=3D0, 16/253 SCBs
 fxp0: <Intel 82558 Pro/100 Ethernet> port 0x2040-0x205f mem
 0xe3000000-0xe3000fff,0xe3100000-0xe31fffff irq 18 at device 2.0 on pci7
 miibus0: <MII bus> on fxp0
 inphy0: <i82555 10/100 media interface> PHY 1 on miibus0
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow
 fxp0: Ethernet address: 00:90:27:17:93:5d
 pci7: <serial bus, FireWire> at device 3.0 (no driver attached)
 isab0: <PCI-ISA bridge> at device 31.0 on pci0
 isa0: <ISA bus> on isab0
 ahci0: <Intel ICH8 AHCI SATA controller> port
 0x5408-0x540f,0x5414-0x5417,0x5400-0x5407,0x5410-0x5413,0x5020-0x503f
 mem 0xe3525000-0xe35257ff irq 19 at device 31.2 on pci0
 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported
 ahcich0: <AHCI channel> at channel 0 on ahci0
 ahcich1: <AHCI channel> at channel 1 on ahci0
 ahcich4: <AHCI channel> at channel 4 on ahci0
 ahcich5: <AHCI channel> at channel 5 on ahci0
 pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
 ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 on acpi0
 ppc0: Generic chipset (ECP/PS2/NIBBLE) in COMPATIBLE mode
 ppc0: FIFO with 16/16/8 bytes threshold
 ppbus0: <Parallel port bus> on ppc0
 plip0: <PLIP network interface> on ppbus0
 lpt0: <Printer> on ppbus0
 lpt0: Interrupt-driven port
 ppi0: <Parallel I/O> on ppbus0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 ichwd0 on isa0
 pmtimer0 on isa0
 orm0: <ISA Option ROMs> at iomem 0xcf800-0xd07ff,0xd0800-0xd0fff pnpid
 ORM0000 on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=3D0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0=
 
 ata0: <ATA channel> at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
 ata1: <ATA channel> at port 0x170-0x177,0x376 irq 15 on isa0
 coretemp0: <CPU On-Die Thermal Sensors> on cpu0
 est0: <Enhanced SpeedStep Frequency Control> on cpu0
 p4tcc0: <CPU Frequency Thermal Control> on cpu0
 coretemp1: <CPU On-Die Thermal Sensors> on cpu1
 est1: <Enhanced SpeedStep Frequency Control> on cpu1
 p4tcc1: <CPU Frequency Thermal Control> on cpu1
 Timecounters tick every 1.000 msec
 vboxdrv: fAsync=3D0 offMin=3D0x1e0 offMax=3D0x438
 IPsec: Initialized Security Association Processing.
 ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to deny,
 logging disabled
 DUMMYNET 0 with IPv6 initialized (100409)
 load_dn_sched dn_sched FIFO loaded
 load_dn_sched dn_sched PRIO loaded
 load_dn_sched dn_sched QFQ loaded
 load_dn_sched dn_sched RR loaded
 load_dn_sched dn_sched WF2Q+ loaded
 hdacc0: <Sigmatel STAC9227X HDA CODEC> at cad 2 on hdac0
 hdaa0: <Sigmatel STAC9227X Audio Function Group> at nid 1 on hdacc0
 pcm0: <Sigmatel STAC9227X (Analog 5.1+HP/2.0)> at nid 13,15,14,10 and 11
 on hdaa0
 usbus0: 12Mbps Full Speed USB v1.0
 usbus1: 12Mbps Full Speed USB v1.0
 usbus2: 480Mbps High Speed USB v2.0
 usbus3: 12Mbps Full Speed USB v1.0
 usbus4: 12Mbps Full Speed USB v1.0
 usbus5: 12Mbps Full Speed USB v1.0
 usbus6: 480Mbps High Speed USB v2.0
 ugen0.1: <Intel> at usbus0
 uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
 ugen1.1: <Intel> at usbus1
 uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
 ugen2.1: <Intel> at usbus2
 uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
 ugen3.1: <Intel> at usbus3
 uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
 ugen4.1: <Intel> at usbus4
 uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
 ugen5.1: <Intel> at usbus5
 uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
 ugen6.1: <Intel> at usbus6
 uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
 uhub0: 2 ports with 2 removable, self powered
 uhub1: 2 ports with 2 removable, self powered
 uhub3: 2 ports with 2 removable, self powered
 uhub4: 2 ports with 2 removable, self powered
 uhub5: 2 ports with 2 removable, self powered
 uhub2: 4 ports with 4 removable, self powered
 uhub6: 6 ports with 6 removable, self powered
 ugen2.2: <Hewlett-Packard> at usbus2
 ulpt0: <Printer> on usbus2
 ulpt0: using bi-directional mode
 ugen3.2: <vendor 0x0001> at usbus3
 uhid0: <vendor 0x0001 product 0x0000, class 0/0, rev 1.00/1.00, addr 2>
 on usbus3
 ugen2.3: <Sunplus Technology Inc.> at usbus2
 umass0: <Bulk Only Interface> on usbus2
 umass0:  SCSI over Bulk-Only; quirks =3D 0x4101
 umass0:9:0:-1: Attached to scbus9
 sa0 at ahc1 bus 0 scbus2 target 6 lun 0
 sa0: <SEAGATE DAT    9SP40-000 910B> Removable Sequential Access SCSI-3
 device
 sa0: 40.000MB/s transfers (20.000MHz, offset 8, 16bit)
 ada0 at ata2 bus 0 scbus0 target 0 lun 0
 ada0: <ST3160815A 3.AAD> ATA-7 device
 ada0: 100.000MB/s transfers (UDMA5, PIO 8192bytes)
 ada0: 152627MB (312581808 512 byte sectors: 16H 63S/T 16383C)
 ada0: Previously was known as ad4
 cd0 at ata2 bus 0 scbus0 target 1 lun 0
 cd0: <Optiarc DVD RW AD-7170A 1.85> Removable CD-ROM SCSI-0 device
 cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
 cd0: cd present [292819 x 2048 byte records]
 da0 at umass-sim0 bus 0 scbus9 target 0 lun 0
 da0: <Hitachi HTS721010G9SA00 > Fixed Direct Access SCSI-2 device
 da0: 40.000MB/s transfers
 da0: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C)
 da0: quirks=3D0x2<NO_6_BYTE>
 ada1 at ahcich0 bus 0 scbus3 target 0 lun 0
 ada1: <ST500DM002-1BD142 KC44> ATA-8 SATA 3.x device
 ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
 ada1: Command Queueing enabled
 ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
 ada1: quirks=3D0x1<4K>
 ada1: Previously was known as ad6
 SMP: AP CPU #1 Launched!
 Timecounter "TSC" frequency 1998042600 Hz quality 1000
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 (cd0:ata2:0:1:0): READ(10). CDB: 28 00 00 04 77 d2 00 00 01 00
 (cd0:ata2:0:1:0): CAM status: SCSI Status Error
 (cd0:ata2:0:1:0): SCSI status: Check Condition
 (cd0:ata2:0:1:0): SCSI sense: ILLEGAL REQUEST asc:64,0 (Illegal mode for
 this track)
 (cd0:ata2:0:1:0): Info: 0x477d2
 (cd0:ata2:0:1:0): Error 6, Unretryable error
 (cd0:ata2:0:1:0): cddone: got error 0x6 back
 Trying to mount root from ufs:/dev/ad4s1a [rw,noatime]...
 WARNING: / was not properly dismounted
 WARNING: /tmp was not properly dismounted
 WARNING: /usr was not properly dismounted
 WARNING: /usr: mount pending error: blocks 0 files 4
 WARNING: /var was not properly dismounted
 WARNING: /var: mount pending error: blocks 4 files 1
 WARNING: /home was not properly dismounted
 
 
 --9qfsKgCMUbdjOVQRWvcgIrPbUjrRqdolN
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSOXTYAAoJEKOfeD48G3g5H/4P/22oT7+FWNE6OxcP/ID44ain
 uYdPVupPq1fax2QZ7A09zpPE0yaU0ycjrPFSqE4g8ktc2PXtf8G/V27OpnhOmool
 yG7kVZ9cMOV8Q4jxVb305cJeuU5jJGFGeQMCnoExXLVPKHn/SjjOD0X20mUZu3Pk
 lE8Tm+J1uWfnx7z10fT9G55vQ+6SAE5tklh0wJh9WH1xCs1UABtPhk2wz9n2HJmf
 Kw+VWcfBDAso7b8R6tCGGmFb9JLCxBiv6xWZ/it65qz8SwrmQSLFgC7kHIP4ZJJ1
 TW5XQ9Mt3GW2eiKWLdUf1jezC6+MadYUHyw+cPZHaEJCdwXQckTz43wfRbXjYs9Y
 suUwwtvW9vIsgDnxGbWgfYX5vBR1OoP7bUYlK7qLMjEzny6XR4JWUW5EjhiDEjye
 SrEkRNQd3wxEHyeJdOkf3Q+s5CRe8C+nE5R/XY52GNeJFUm0j7vxr4zVGpvJ6I0g
 p2o2YWXMvDvubZJ0VciSvFw7wF+6FREygFOZw1hXMZJv2+hUCvzCebcceEZiAfMe
 CpKS+odFToCTVgOAfeFBMDJ51VTVq0NpTiRSe2aBbfSUP/K233QhR17cEw0oHzTe
 RZrrO3RHm4/ObhMdwEDZIMH4a0j+koBX+Krr9ElujPwxt6qxQD/Q2+YH+3RdM8Wh
 7Ou+HJtSsLtLkl/YAG6V
 =eNEY
 -----END PGP SIGNATURE-----
 
 --9qfsKgCMUbdjOVQRWvcgIrPbUjrRqdolN--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Fri, 20 Sep 2013 14:37:31 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --V4dXVa9gNlomRqwWaMlK1RqOHDXnCFQUB
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Same bug, but with rebuilt setup. Changed my root from ufs to zfs. Went
 from i368 to amd62. My kernel is
 https://svn0.us-west.freebsd.org/base/releng/9.1
 
 Got exactly similar message.
 
 Sep 20 14:31:59 bmach kernel: Fatal trap 12: page fault while in kernel m=
 ode
 Sep 20 14:31:59 bmach kernel: cpuid =3D 0; apic id =3D 00
 Sep 20 14:31:59 bmach kernel: fault virtual address	=3D 0x10
 Sep 20 14:31:59 bmach kernel: fault code		=3D supervisor read data, page
 not present
 Sep 20 14:31:59 bmach kernel: instruction pointer	=3D 0x20:0xffffffff8174=
 f8b8
 Sep 20 14:31:59 bmach kernel: stack pointer	        =3D
 0x28:0xffffff81193bf380
 Sep 20 14:31:59 bmach kernel: frame pointer	        =3D
 0x28:0xffffff81193bf390
 Sep 20 14:31:59 bmach kernel: code segment		=3D base 0x0, limit 0xfffff,
 type 0x1b
 Sep 20 14:31:59 bmach kernel: =3D DPL 0, pres 1, long 1, def32 0, gran 1
 Sep 20 14:31:59 bmach kernel: processor eflags	=3D interrupt enabled,
 resume, IOPL =3D 0
 Sep 20 14:31:59 bmach kernel: current process		=3D 2128 (named)
 Sep 20 14:31:59 bmach kernel: trap number		=3D 12
 Sep 20 14:31:59 bmach kernel: panic: page fault
 Sep 20 14:31:59 bmach kernel: cpuid =3D 0
 Sep 20 14:31:59 bmach kernel: KDB: stack backtrace:
 Sep 20 14:31:59 bmach kernel: #0 0xffffffff8092a076 at kdb_backtrace+0x66=
 
 Sep 20 14:31:59 bmach kernel: #1 0xffffffff808f41cd at panic+0x1cd
 Sep 20 14:31:59 bmach kernel: #2 0xffffffff80be8350 at trap_fatal+0x290
 Sep 20 14:31:59 bmach kernel: #3 0xffffffff80be868d at trap_pfault+0x1ed
 Sep 20 14:31:59 bmach kernel: #4 0xffffffff80be8cae at trap+0x3ce
 Sep 20 14:31:59 bmach kernel: #5 0xffffffff80bd32bf at calltrap+0x8
 Sep 20 14:31:59 bmach kernel: #6 0xffffffff81759e6f at pf_test_rule+0xf3f=
 
 Sep 20 14:31:59 bmach kernel: #7 0xffffffff8175d644 at pf_test+0xe34
 Sep 20 14:31:59 bmach kernel: #8 0xffffffff817651c1 at pf_check_out+0x41
 Sep 20 14:31:59 bmach kernel: #9 0xffffffff809b800e at pfil_run_hooks+0x9=
 e
 Sep 20 14:31:59 bmach kernel: #10 0xffffffff80a14806 at ip_output+0x416
 Sep 20 14:31:59 bmach kernel: #11 0xffffffff80a91909 at udp_send+0x509
 Sep 20 14:31:59 bmach kernel: #12 0xffffffff8095ec67 at sosend_dgram+0x2d=
 7
 Sep 20 14:31:59 bmach kernel: #13 0xffffffff80962f53 at kern_sendit+0x1a3=
 
 Sep 20 14:31:59 bmach kernel: #14 0xffffffff809631ec at sendit+0xdc
 Sep 20 14:31:59 bmach kernel: #15 0xffffffff80963277 at sys_sendmsg+0x87
 Sep 20 14:31:59 bmach kernel: #16 0xffffffff80be7bf6 at amd64_syscall+0x5=
 46
 Sep 20 14:31:59 bmach kernel: #17 0xffffffff80bd35a7 at Xfast_syscall+0xf=
 7
 Sep 20 14:31:59 bmach kernel: Uptime: 6h24m26s
 
 
 My kernel is an exact copy of GENERIC + these additions:
 
 
 options         MROUTING                # Multicast routing
 
 # ALTQ
 options         ALTQ
 options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
 options         ALTQ_RED        # Random Early Detection (RED)
 options         ALTQ_RIO        # RED In/Out
 options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
 options         ALTQ_CDNR
 options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
 #options         ALTQ_NOPCC      # Required if the TSC is unusable
 
 
 --V4dXVa9gNlomRqwWaMlK1RqOHDXnCFQUB
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSO7TsAAoJEKOfeD48G3g5aQMP/2+ixh0ogvW7c6tTc0neizBf
 /7qS76pT0QeaktLZrboAfzC0fRWjhtOGdfpNymxVT/q+JKGM8ZoHD9jcCyYLAdyi
 yv2Uo0Lxf5JbdjRkLmSKcHWRmzdNGSrs+r7w/bbQrReX1W/cuw4t8XfBQ/klOtey
 K8NtzUMXwMBQqRI3bIm8IVOrmean2o3TgXUVH8y5eY92YZ4HDyktCqGd3Uyw+mEV
 ZhrfHHIe9hcD8ajob+3ab6A4I6oVwE+hvEqj3L8whTOqZWQmvGaYL2SFZlrgzipE
 2BY37FIFJCCgflDJfrFZsMvCUtDLOtDvbRONRqSo9ILAtu+gTKDetGYLXJ5e9fEN
 etYy/F5F+0yapxj2ZmjpI/pVJSTas8slW54dcMqznuOePpouzkHvHsjdU+59NT5w
 KQQZjhtvbE/kXn3evDijHqiXG2mj84i1+o4ygH2HZXvtqPnM5+z1f/hZpXytH5uG
 l4n2uByEURwTnaTbMlKflzAXQHRU3yuIC1QDKA+cG9/w6Q3Kk7Zqx/xgKJHR93RS
 2194JuD1RHmwbSZ7sHiMRSnsDcjgfXKZRADnVlCv/eSVb41bQBiCFv7b/GFyftuX
 IGPMRFcOBXNPnbeOQkFIE0tbi28YwZmlOvLeRkO6Fds8PpAE+g8iaELut6bHOUNx
 EOLb/vo/jfIax+wmnV4B
 =tYA2
 -----END PGP SIGNATURE-----
 
 --V4dXVa9gNlomRqwWaMlK1RqOHDXnCFQUB--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Sat, 21 Sep 2013 16:12:48 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --gGcoUPca7xBFhjNC9DneAswx6ElsA5plv
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 This bug seems related to
 http://www.freebsd.org/cgi/query-pr.cgi?pr=3D173659&cat=3D
 
 
 --gGcoUPca7xBFhjNC9DneAswx6ElsA5plv
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSPRzAAAoJEKOfeD48G3g5GMAP/1jztLNljN8FGcZM/QyBc5bD
 PBxO0CzRpDVW7glh50qyL7ZtbJZeogv43Vx4JNw01k4NpKMX01XfDKzXub0T0vMe
 JTVGApNzRZnRmK7ru5jCPcSrBhtnVn+59Xi2oyyiH7gb5f+iKr7QbGoAG14ut1t9
 w0aX4kzPc1rtS72UPDdwz4T9haDfWJ4ryBmmFYWu0J82rbtwjEvedhE0GdrogsFW
 CqemftqdQ/rpbc7HYiGt55w956Vu3iOMmB0dAOAUqI+n9MxLFHc/leac+XMIDeDJ
 j8LzPtTTMlJHoA0QLQkjsSaV0ndSbC7jBUTv1y730h0bDh4xUaOlCNQP4qvGJklx
 lvuXaMvrbsSjmRIYpTDS6SqNUFez6UJJcGvVKJOW6uetL/enrSWKWwNJs4MdE4P2
 w3eoJF+i9jEWuHNTcgJjYnXaHw9kKE8BCt4mTwjzm1Txb91jI3xyzH5h6/a3jHuX
 byILw1jIhq2/W8u12SXe5gk/kKXC9ERGlbz/dpHfmK4xgpoyTeRxtuqd+ffv5PBy
 LBIRqUu25h32qrE1wxIZWT4EDWFeDXJ/q1ltc93Rm7xy/9ddY7ZsvEYZEGRvk70o
 Xa03fKMpVCKmdtm+VwMdA4ID+xWYP9NvTQzZL2ap9BSF22uhyUXo75UD8MWoOCIL
 rdaKjUQFrF4VJrnqqlLk
 =MV4g
 -----END PGP SIGNATURE-----
 
 --gGcoUPca7xBFhjNC9DneAswx6ElsA5plv--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Mon, 23 Sep 2013 16:56:09 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --wakNJ7tli9VR2rpMBX1smw7IHnHGSu6Wh
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Have enabled:
 
 options         INVARIANTS
 options         INVARIANT_SUPPORT
 options         WITNESS
 options         DEBUG_MEMGUARD
 
 log dump is now:
 
 Sep 23 16:48:15 bmach syslogd: kernel boot file is /boot/kernel/kernel
 Sep 23 16:48:15 bmach kernel: Kernel page fault with the following
 non-sleepable locks held:
 Sep 23 16:48:15 bmach kernel: exclusive sleep mutex pf task mtx (pf task
 mtx) r =3D 0 (0xffffffff8190b7c0) locked @
 /usr/src/9.1/sys/modules/pf/../../contrib/pf/net/pf.c:6621
 Sep 23 16:48:15 bmach kernel: shared rm PFil hook read/write mutex (PFil
 hook read/write mutex) r =3D 0 (0xffffffff81419ee8) locked @
 /usr/src/9.1/sys/net/pfil.c:77
 Sep 23 16:48:15 bmach kernel: shared rw udpinp (udpinp) r =3D 0
 (0xfffffe008122d168) locked @ /usr/src/9.1/sys/netinet/udp_usrreq.c:975
 Sep 23 16:48:15 bmach kernel: KDB: stack backtrace:
 Sep 23 16:48:15 bmach kernel: #0 0xffffffff8090d386 at kdb_backtrace+0x66=
 
 Sep 23 16:48:15 bmach kernel: #1 0xffffffff8092141e at
 _witness_debugger+0x2e
 Sep 23 16:48:15 bmach kernel: #2 0xffffffff80922206 at witness_warn+0x2c6=
 
 Sep 23 16:48:15 bmach kernel: #3 0xffffffff80bba27a at trap_pfault+0x6a
 Sep 23 16:48:15 bmach kernel: #4 0xffffffff80bbaa10 at trap+0x440
 Sep 23 16:48:15 bmach kernel: #5 0xffffffff80ba4cdf at calltrap+0x8
 Sep 23 16:48:15 bmach kernel: #6 0xffffffff818ec11f at pf_test_rule+0xf2f=
 
 Sep 23 16:48:15 bmach kernel: #7 0xffffffff818ef944 at pf_test+0xe74
 Sep 23 16:48:15 bmach kernel: #8 0xffffffff818f76f1 at pf_check_out+0x41
 Sep 23 16:48:15 bmach kernel: #9 0xffffffff8099b802 at pfil_run_hooks+0xd=
 2
 Sep 23 16:48:15 bmach kernel: #10 0xffffffff809f340f at ip_output+0x55f
 Sep 23 16:48:15 bmach kernel: #11 0xffffffff80a6c4c5 at udp_send+0x4d5
 Sep 23 16:48:15 bmach kernel: #12 0xffffffff80945a0e at sosend_dgram+0x21=
 e
 Sep 23 16:48:15 bmach kernel: #13 0xffffffff8094cd44 at kern_sendit+0x1b4=
 
 Sep 23 16:48:15 bmach kernel: #14 0xffffffff8094cfac at sendit+0xdc
 Sep 23 16:48:15 bmach kernel: #15 0xffffffff8094d037 at sys_sendmsg+0x87
 Sep 23 16:48:15 bmach kernel: #16 0xffffffff80bb979a at amd64_syscall+0x2=
 fa
 Sep 23 16:48:15 bmach kernel: #17 0xffffffff80ba4fc7 at Xfast_syscall+0xf=
 7
 Sep 23 16:48:15 bmach kernel:
 Sep 23 16:48:15 bmach kernel:
 Sep 23 16:48:15 bmach kernel: Fatal trap 12: page fault while in kernel m=
 ode
 Sep 23 16:48:15 bmach kernel: cpuid =3D 0; apic id =3D 00
 Sep 23 16:48:15 bmach kernel: fault virtual address	=3D 0x10
 Sep 23 16:48:15 bmach kernel: fault code		=3D supervisor read data, page
 not present
 Sep 23 16:48:15 bmach kernel: instruction pointer	=3D 0x20:0xffffffff818e=
 1888
 Sep 23 16:48:15 bmach kernel: stack pointer	        =3D
 0x28:0xffffff82fc485370
 Sep 23 16:48:15 bmach kernel: frame pointer	        =3D
 0x28:0xffffff82fc485380
 Sep 23 16:48:15 bmach kernel: code segment		=3D base 0x0, limit 0xfffff,
 type 0x1b
 Sep 23 16:48:15 bmach kernel: =3D DPL 0, pres 1, long 1, def32 0, gran 1
 Sep 23 16:48:15 bmach kernel: processor eflags	=3D interrupt enabled,
 resume, IOPL =3D 0
 Sep 23 16:48:15 bmach kernel: current process		=3D 2199 (named)
 Sep 23 16:48:15 bmach kernel: trap number		=3D 12
 Sep 23 16:48:15 bmach kernel: panic: page fault
 Sep 23 16:48:15 bmach kernel: cpuid =3D 0
 Sep 23 16:48:15 bmach kernel: KDB: stack backtrace:
 Sep 23 16:48:15 bmach kernel: #0 0xffffffff8090d386 at kdb_backtrace+0x66=
 
 Sep 23 16:48:15 bmach kernel: #1 0xffffffff808d6e68 at panic+0x1d8
 Sep 23 16:48:15 bmach kernel: #2 0xffffffff80bba0c0 at trap_fatal+0x290
 Sep 23 16:48:15 bmach kernel: #3 0xffffffff80bba28a at trap_pfault+0x7a
 Sep 23 16:48:15 bmach kernel: #4 0xffffffff80bbaa10 at trap+0x440
 Sep 23 16:48:15 bmach kernel: #5 0xffffffff80ba4cdf at calltrap+0x8
 Sep 23 16:48:15 bmach kernel: #6 0xffffffff818ec11f at pf_test_rule+0xf2f=
 
 Sep 23 16:48:15 bmach kernel: #7 0xffffffff818ef944 at pf_test+0xe74
 Sep 23 16:48:15 bmach kernel: #8 0xffffffff818f76f1 at pf_check_out+0x41
 Sep 23 16:48:15 bmach kernel: #9 0xffffffff8099b802 at pfil_run_hooks+0xd=
 2
 Sep 23 16:48:15 bmach kernel: #10 0xffffffff809f340f at ip_output+0x55f
 Sep 23 16:48:15 bmach kernel: #11 0xffffffff80a6c4c5 at udp_send+0x4d5
 Sep 23 16:48:15 bmach kernel: #12 0xffffffff80945a0e at sosend_dgram+0x21=
 e
 Sep 23 16:48:15 bmach kernel: #13 0xffffffff8094cd44 at kern_sendit+0x1b4=
 
 Sep 23 16:48:15 bmach kernel: #14 0xffffffff8094cfac at sendit+0xdc
 Sep 23 16:48:15 bmach kernel: #15 0xffffffff8094d037 at sys_sendmsg+0x87
 Sep 23 16:48:15 bmach kernel: #16 0xffffffff80bb979a at amd64_syscall+0x2=
 fa
 Sep 23 16:48:15 bmach kernel: #17 0xffffffff80ba4fc7 at Xfast_syscall+0xf=
 7
 Sep 23 16:48:15 bmach kernel: Uptime: 5h21m4s
 
 
 --wakNJ7tli9VR2rpMBX1smw7IHnHGSu6Wh
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSP8nuAAoJEKOfeD48G3g5/rYQALIEG1dQ7W4NLHE9OWSBEhFQ
 U+Rdj3rfukAfOzjB+qmP57OduI8N1V8i/wJCAIwGJh9U+TijpXbjIjcq+qWv/zLx
 WGaWm8RI6yD7qMbNWDy67lUbq//C7KyCg3T6ge+fOYklFlg7okp0IABEacfqX6jI
 uHXp2LpmKS2I2tkUgZbEpTPsY5M1shFZpUfXD9DX2HDUoTFqdaAL5D01fAa9JqK+
 a+R7LP0TJU7aWbLwOIVrs4VkNP5xqTR+Dmu5UIwQR8uw6GKaeipRW2gWc2K+uQrl
 /Tw7qCGvDVWQBQVmpGKa1L7FokoccFyH+aiVsvVyNrQwX8b/HmXDrlQ3iGphN5C8
 +5VbOWgnxvGGR++6TfU2OyginOoK+baMD+x5PNLnIjNtqX2v/HC87SJ3qOmzZ7CW
 WR7mXyslTnhf6h5zq22AWQZAhM6RYZMM4BbRgQWA5N+alYHejbF6iDbB6JaDl+pz
 bVo9eo86q7T9r4g2R4PFf2cm+kBC9Vnem1dM1fXf84Ly4MgZorjJwiP/d/U05tX/
 mHHP34rH6A4AmW5hJAwBKW6b4yGD4mE9Paz7a+UN6xxB2vMb79Q9ElnT2MZ+jWRz
 9l0B2GmytXp5JKhf6ffpX8McRuu5UTfLNpd7g/UyfDmiHBLgcBih1xtNdcv8Aec5
 YRcJvh5Li2in8qcrMa4c
 =Q+zs
 -----END PGP SIGNATURE-----
 
 --wakNJ7tli9VR2rpMBX1smw7IHnHGSu6Wh--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Mon, 23 Sep 2013 20:45:54 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --l3rXXvr6HQu6DJElI05GCSU5QQw5mvELF
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Getting late, forget about that last link, it's not similar.
 
 
 --l3rXXvr6HQu6DJElI05GCSU5QQw5mvELF
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSP//CAAoJEKOfeD48G3g5JPoQALE9Ac7QcRgVkWLSu54OHXn8
 BQ8zMmn2PWTSqJBc6H01sCV6tzauu6MbHtq2q7u0ENk87v2JCn3UMt+j0McVjKlq
 ndlAiOkANWRiE03kis/3WgOP1jwKaoaUP9HWM0t4x8Urf4mvh9ogxIKNkeZyWy0g
 yQ5z0yPajpJqv7FZZm1HHhJNXZ64zaKxdY0uCJ65hovaRvReTZoN10KgrZioo5tE
 nA5yuuFTqXMcdmyuHMR7JMnHqb5/i2BE16ai6ZnCEErZI2t88Fp9dvTf+lj8EEXB
 IIqKs9vt7LTtKR0hPAbdO0Rn0Od29EYwEn2p1q7j0kbQUYCgmG827ctryjcwZC3o
 c+incKMcV3xUbZxRtp6BLngwTzCRCFofTsXnGWnMr4ckQq7bH0kjQxU8FYBlMTDq
 N9KKs7MK88VmHxe0bQRDS8lclaPRa+3jgpD90Is+5p5Le+mM3IlsfVYDciK9nljE
 k98LArf0Y0hZbjVj1BoIN1H7Y9lq/IjWcfmOVkq/i5MryfrBoEG8LQlOFN96ow/2
 d+ZpdCKNOwexVG1k9F+LFdWTqr9PJGVY+wqp6fzMWy/nT0pX2tsFHcBshvRY7a9o
 f8ncmwkUoKmMBHtGiWbKvXJJQqjbRfcBxfL/Vj02IhJ6OCIjiZtUV55aXlBh/kuo
 he0WAGGA1DMw3g29Drsk
 =JCgw
 -----END PGP SIGNATURE-----
 
 --l3rXXvr6HQu6DJElI05GCSU5QQw5mvELF--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Mon, 23 Sep 2013 20:43:44 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --gm5WmnmhdEGNgKi9ccPs5CIsg1LAnmAo0
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Also my experience is very similar to this:
 http://marc.info/?l=3Dfreebsd-stable&m=3D135639435500691&w=3D2
 
 
 --gm5WmnmhdEGNgKi9ccPs5CIsg1LAnmAo0
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSP/9AAAoJEKOfeD48G3g5A2wP/3VIxOmWfn7l0/cRJsXv7CmO
 1w3xLfzCjlKwYmhyWBhR75zvi/qoPeWx8wWGqLty+dZv+nGuYGFlMw+ysnoi/BB0
 ZBIflxpgO6zoVR3kM/fjskL13uQiH5w0Zf1v/zsOxMFWC+NwP5HrYLZydJLDndIj
 eyr7OhO9F87+sq7lusR+2ZY22l4E98qFD/Uj3P2AZkK0Qkeobm6tJG9MvX/q35Ya
 vyf+U9rg6OuSTrZ0/iLLQkJTDkb53IU1VYhEgRmgXE0je0E0aETk7qN1clbNDdyV
 ByVeCBfu481JDKjNMrzb7Un4X3QJe6f8Ta+7i3i61SVvLNYyKQyQ/6079UivZORf
 7DegkTkGSKEcecWpdWQfzA/Kw/LAUjRVa8VpyxAGdEb+Io5L5fmto7HKcGhCJ7FG
 KrAkBGHgraqfUJE6XNwn4zQArTgrcmMAy06vU6DA4E/2wgGQf0Aa9GG47BB2Uife
 L81qoRznaQsCiuRIE85D6YZUNLQCsoZQFQISMFSeGgKeNED5al126JaECM37ij10
 7IT0c7PyWP/yQDCmG4QtG8aU1iTNSTUl1j0sxmgn0NI93o/zKHZc8LNx7VEYZxrP
 NQa5xBVU6NqLXfBFgcmASNlE6SCsFAnE7pfouyokrQt5muhp1iIL1Si/d6Tb5NSk
 yIFzp3q12Fgky+KsdsLy
 =xYNB
 -----END PGP SIGNATURE-----
 
 --gm5WmnmhdEGNgKi9ccPs5CIsg1LAnmAo0--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Mon, 23 Sep 2013 20:40:14 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --baIvWTb5LflbBqeqc1HcfGcSIm6dJlc40
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Similar stack dump in email thread here:
 http://lists.freebsd.org/pipermail/freebsd-pf/2012-September/006798.html
 
 
 --baIvWTb5LflbBqeqc1HcfGcSIm6dJlc40
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSP/5vAAoJEKOfeD48G3g5/8EQAKwsmMckSrMuRpUfLNs4uIAC
 oE7KmaQpHm1yu61PGNxKv3ayZvcUgdmUqJemugw5SnnYvvyV5VNGTMuoVn08y89G
 T7YfUhwLGSNrWSVjAyceC7PfQpiLYGYG77cgJbRC4SBsoPvYYJSi8UO/tsoA9gB2
 zpN77DdnnvaDUKFzPV64qHBSTC14oj9xM0IMRl9JDEg6zvRj+4PI23aL4r8OxOIG
 KEp+M7GNxkvSXfAaemZGzcNE16/Do+ZeDdOfmUTIvpU5pwwf7BHSGOzImfhar1IR
 Xie/LishmvcRQlPg4L9xlR29zC9e+51aFGzgeDmHrAoG/NdHKLGtZEzmLGfP1Wqc
 MGWpQCef3Wb0tWrNALbZ4z0yofAgKaO+d2qF3GFH2v7iWoBK7sgtV/NuzazxhGLh
 PRdBFt9JeYXbtwePJ2oinU9/4Wqb61rA5ycx8K3vpadSfuXfzQKafkpdYVO4E0kF
 uwty+VvoMZQ3qRNw1V5WbzPePVqXIroQ/RsaFTDgxr6VlAZYyhy+uKjO4lbS8SrQ
 cjmkbpWyPEU7L7LfVeoP9Mp9lXSCduKM655W1yRvI3JICWS34MWgb+dqJkNcd/2u
 kh6VKPnNnLr4oS16mkzFMffJXcWc4Adgek/3vBxNJxte1xK1E0RLm+r1btOlI3+f
 NUuEHxbBeLopuwQfAHcA
 =s5QY
 -----END PGP SIGNATURE-----
 
 --baIvWTb5LflbBqeqc1HcfGcSIm6dJlc40--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Tue, 24 Sep 2013 19:08:43 +1200

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --MsXgDscvu9D8o3vma3VVP3A6PxtDjJuFj
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Slightly different stack trace with kernel without ALTQ, and without INET=
 6.
 
 Sep 24 18:58:58 bmach kernel: Fatal trap 12: page fault while in kernel m=
 ode
 Sep 24 18:58:58 bmach kernel: cpuid =3D 1; apic id =3D 01
 Sep 24 18:58:58 bmach kernel: fault virtual address	=3D 0x10
 Sep 24 18:58:58 bmach kernel: fault code		=3D supervisor read data, page
 not present
 Sep 24 18:58:58 bmach kernel: instruction pointer	=3D 0x20:0xffffffff816f=
 4dee
 Sep 24 18:58:58 bmach kernel: stack pointer	        =3D
 0x28:0xffffff811945d390
 Sep 24 18:58:58 bmach kernel: frame pointer	        =3D
 0x28:0xffffff811945d540
 Sep 24 18:58:58 bmach kernel: code segment		=3D base 0x0, limit 0xfffff,
 type 0x1b
 Sep 24 18:58:58 bmach kernel: =3D DPL 0, pres 1, long 1, def32 0, gran 1
 Sep 24 18:58:58 bmach kernel: processor eflags	=3D interrupt enabled,
 resume, IOPL =3D 0
 Sep 24 18:58:58 bmach kernel: current process		=3D 1739 (named)
 Sep 24 18:58:58 bmach kernel: trap number		=3D 12
 Sep 24 18:58:58 bmach kernel: panic: page fault
 Sep 24 18:58:58 bmach kernel: cpuid =3D 1
 Sep 24 18:58:58 bmach kernel: KDB: stack backtrace:
 Sep 24 18:58:58 bmach kernel: #0 0xffffffff8091a936 at kdb_backtrace+0x66=
 
 Sep 24 18:58:58 bmach kernel: #1 0xffffffff808e4a8d at panic+0x1cd
 Sep 24 18:58:58 bmach kernel: #2 0xffffffff80b9cd90 at trap_fatal+0x290
 Sep 24 18:58:58 bmach kernel: #3 0xffffffff80b9d0cd at trap_pfault+0x1ed
 Sep 24 18:58:58 bmach kernel: #4 0xffffffff80b9d6ee at trap+0x3ce
 Sep 24 18:58:58 bmach kernel: #5 0xffffffff80b87cff at calltrap+0x8
 Sep 24 18:58:58 bmach kernel: #6 0xffffffff816f6906 at pf_test+0xc26
 Sep 24 18:58:58 bmach kernel: #7 0xffffffff816fd891 at pf_check_out+0x41
 Sep 24 18:58:58 bmach kernel: #8 0xffffffff809a81ce at pfil_run_hooks+0x9=
 e
 Sep 24 18:58:58 bmach kernel: #9 0xffffffff80a04227 at ip_output+0x407
 Sep 24 18:58:58 bmach kernel: #10 0xffffffff80a79b19 at udp_send+0x509
 Sep 24 18:58:58 bmach kernel: #11 0xffffffff8094f527 at sosend_dgram+0x2d=
 7
 Sep 24 18:58:58 bmach kernel: #12 0xffffffff80953813 at kern_sendit+0x1a3=
 
 Sep 24 18:58:58 bmach kernel: #13 0xffffffff80953aac at sendit+0xdc
 Sep 24 18:58:58 bmach kernel: #14 0xffffffff80953b37 at sys_sendmsg+0x87
 Sep 24 18:58:58 bmach kernel: #15 0xffffffff80b9c636 at amd64_syscall+0x5=
 46
 Sep 24 18:58:58 bmach kernel: #16 0xffffffff80b87fe7 at Xfast_syscall+0xf=
 7
 
 
 --MsXgDscvu9D8o3vma3VVP3A6PxtDjJuFj
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSQTp7AAoJEKOfeD48G3g5CtIP/0SnchLS7abeieq0tDqfzuUN
 4SxAH92PDJCEnkcAnqke8svlDG7bRMWPoC8OYQsteDvMqPjTIWSutGbY6ndqPlok
 gQKsbI6DpUh02sJocrVgpqfgscgpBFXrulQBIaOqL8DutoNHnFSO2/HEmaQrPS42
 UuaDTjm+6JJAuUG+qGAU9jY56OIK5soC2Z1Kolq8EstpKFfcSCY4cJmlrMk50YyL
 aMHRvLB1sjVhdVvVDitRg6Lpls8zN6Z60njTGZgZAVLPhJcWfbz6oa2oW+63jk13
 36M+X1yfPTHth8PPziD0qu4TRBl/VmXOU36FfCgr+4Ivqod537ro75WMlZUAfT6h
 lEzAftEizg9ixdesaRqYTGWgTXoULxp8Pm4Te6DUbd8x6xk40CASp6caKktOAWJx
 I/b6FqX+lY9hJWq/6cQxZB5PJy9+8lLWnrvVpVZPkf3zmzFhdOKrC4SrThhkUX3h
 BZxh3jPGlKv1VRTE+N/fj46/bFVCCpDBFUCkdQHJrpinYtYaHpFPJhY6FrhWcZso
 ZCksHZWW1dm5mRXwntOfHYhq8sS42ECDvEohvKEEoev11Q4BcTcHJbZBhcBPxsUQ
 wOS36b1VkLGpx3vNAp45E4BsDJrwHIWnUJmBhtE6EsRT9YO7R5lC4XJfJrMLgu/T
 B0Q+hoElHG5qLG88igbD
 =Xu+s
 -----END PGP SIGNATURE-----
 
 --MsXgDscvu9D8o3vma3VVP3A6PxtDjJuFj--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Tue, 01 Oct 2013 20:11:55 +1300

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --DIFikbPeFdck6a5mTLB6apdf76qmSs6R8
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 I went down to an extremely simple pf config, even that reboots (but not
 as often, can stay up for almost 2 days):
 
 nat on egress proto udp from { lan:network, $vpn_if:network,
 192.168.233.21, 192.168.233.25 } port $voip_ports to any -> (egress)
 static-port
 nat on egress from any to any -> (egress) round-robin sticky-address
 
 block log
 pass on lan
 pass on public_wifi
 pass on $vpn_if
 set skip on lo0
 pass out on egress
 
 
 --DIFikbPeFdck6a5mTLB6apdf76qmSs6R8
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSSnXLAAoJEKOfeD48G3g5xE8QAKkf27t4uqbtP8Y6p+iyl3e3
 2f9bQz1h8W63qZEmg9Oo878+ms+bjth7Luspf3PP7E7ZJqo2BSxQQNvoWivafMOs
 0Lk1jCIwiB5EwG7k9ivZroPe2INSQqIffj/tefIPaXiscWcGdr9myq+Ijl4eqTQN
 +YotZCQOiYGB+gu8/1kLD+rSu3suRObutIf8lnC6fr73nY6gD9mNGkpaDP/hqiTr
 je/TxQYukzrqFwGcCuTlp6pt5QESzV22ijSY+JaitYi06jNUcn4buP23rJOgChHO
 QfwfkwRGLk1OL7VLCAEX4mzn04xBa1SQKgx9Gk8imFdc7t6af8w+iCDjxe3jTeUx
 I2xSp7v9QYu/ICDWOioRiKAVtKwInZkuJ7WPN1NtDq7+muE1xJJnApzruAqXmb7z
 EG4+Av3Ik0pNj5QexkZkEqA7zoVukqFxa1A+tmApLUes0+e3gD6QnK52IzcrFD6f
 1jzgXqJVBMekKZlr+TIn0wO7MQ0RZwQU4Jl1n5/mt/IQY+hnWPdUYbLeKRUQJxFg
 qE8v7q/X6StfSLsM+fVTonwPL3BIGnEA/CpBduRGxuwrOM9IPc854mApAqbHGCVB
 oQ5ZeXONL8EfoN4YZ/1+6Dfqly9opV+Hj9WPIaKRaUpGseG5IVDeIqvdySo1JIkP
 yrV7e1Uw3ZgSbxxu7uyv
 =jDqM
 -----END PGP SIGNATURE-----
 
 --DIFikbPeFdck6a5mTLB6apdf76qmSs6R8--

From: Berend de Boer <berend@pobox.com>
To: bug-followup@FreeBSD.org, berend@pobox.com
Cc:  
Subject: Re: misc/182141: Very frequent (at most 3 hours) kernel trap 12
Date: Wed, 16 Oct 2013 17:25:12 +1300

 This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
 --MWGbNFtLhPVqLTrrC7cb8fqWcrsJ4FNfW
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 I have now narrowed the reboots down to this line:
 
   nat on egress from any to any -> (egress) sticky-address
 
 or
 
 nat pass on egress from any to any -> (egress) sticky-address
 
 (pass doesn't matter). It's the sticky-address that causes the crashes.
 Absent that, it stays up.
 
 
 --MWGbNFtLhPVqLTrrC7cb8fqWcrsJ4FNfW
 Content-Type: application/pgp-signature; name="signature.asc"
 Content-Description: OpenPGP digital signature
 Content-Disposition: attachment; filename="signature.asc"
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
 
 iQIcBAEBCAAGBQJSXhUyAAoJEKOfeD48G3g5PuMP/3LspUj7V8DnyqNS9d0Ng95W
 gNZrAPdCYlv/dkQuuBqWD7DF/SUnPTCmAQ/ibNV2pmR2zbwdb/ZKP/lBCvi2x6OE
 zPRErSAib+OGcklcA9DiVFDgqGYwEsekZEukY/jA7rNuPFdnioXY16xLbpFk1vDq
 s6QLD5NQGH/zE4NLf6M3LBWa7pl15zbbOifG/LszE2w7CMOY7dPUD7R9ovSaBsdq
 9YEvrsHN1/lzfEIr5QTtAHTHgjV1BwZ4uLrkse/0Lg4h6nzV9iMoDAwnHPMqhdMG
 yV6dPEgn6XAq4XqVDv3xvXwiYTFQw62quuBMCo8jcTmCsNEiqzySX2vxAOCoebKM
 dWkPaAutx5VvXZowOt81E8xpWJ3LHY+mHfrLPgdyO2BTN191oAYM84fxJNKSGFix
 zTy8z1jrAlveZweWOWQpyuZjD3T3VQWUjwQDlffH2lLE8SnEhgo3PJjxyT/KNCly
 HZpOpC572d7/pOL8s7F3wyvkAFoojEJjQU+UxVWK6bsliIN+nwEox2V+ojHxaZG3
 WBr/HvTmkDNIV2sB5su3oS25yG6LXlbQCwUL409hZP3Di3CVf+pQTkkjxsaP/Pq4
 VFDHt6HkxFMowtMRJs7cAIE/nq+KtySZ7/T+C/eMnqvSoR58nAOLHRWW/egHtPl9
 FIiP0FC1okLkvmE2Wjuk
 =Pjgg
 -----END PGP SIGNATURE-----
 
 --MWGbNFtLhPVqLTrrC7cb8fqWcrsJ4FNfW--
State-Changed-From-To: open->feedback 
State-Changed-By: glebius 
State-Changed-When: Wed Jan 1 21:53:02 UTC 2014 
State-Changed-Why:  
Patch sent to submitter for testing. 


Responsible-Changed-From-To: freebsd-bugs->glebius 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Wed Jan 1 21:53:02 UTC 2014 
Responsible-Changed-Why:  
Patch sent to submitter for testing. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=182141 
State-Changed-From-To: feedback->patched 
State-Changed-By: glebius 
State-Changed-When: Mon Jan 6 19:05:16 UTC 2014 
State-Changed-Why:  
Fixed in head in r260377. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=182141 
State-Changed-From-To: patched->closed 
State-Changed-By: glebius 
State-Changed-When: Wed Jan 22 10:44:37 UTC 2014 
State-Changed-Why:  
Fix merged to stable/10. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=182141 
>Unformatted:
