From nobody@FreeBSD.org  Fri May 28 13:06:13 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 05581106567B
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 28 May 2010 13:06:13 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id E8FD78FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 28 May 2010 13:06:12 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o4SD6B1t073249
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 28 May 2010 13:06:11 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o4SD6BvN073233;
	Fri, 28 May 2010 13:06:11 GMT
	(envelope-from nobody)
Message-Id: <201005281306.o4SD6BvN073233@www.freebsd.org>
Date: Fri, 28 May 2010 13:06:11 GMT
From: Tom Dewaele <tom.dewaele@abvv.be>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Page Fault / Kernel panic when jail starts on boot
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         147162
>Category:       kern
>Synopsis:       [jail] [panic] Page Fault / Kernel panic when jail starts on boot
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-jail
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri May 28 13:10:00 UTC 2010
>Closed-Date:    Mon Sep 06 07:18:37 UTC 2010
>Last-Modified:  Mon Sep 06 07:18:37 UTC 2010
>Originator:     Tom Dewaele
>Release:        FreeBSD 8.0p3 AMD64
>Organization:
ABVV
>Environment:
FreeBSD T00FVS01.fed.diva.net 8.0-RELEASE-p3 FreeBSD 8.0-RELEASE-p3 #0: Tue May 25 20:54:11 UTC 2010     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
A short system explanation:
Hardware: HP DL 360G6 2x E5504 / 8GB
I'am running 3 jails on this server with ezjail (latest port)

I've updated from 8.0p2 to 8.0p3 yesterday.
After reboot the server did not come back.
It showed a page fault crash on the console after ezjail started the first jail. The page fault happen with the 'ping' proces. The jail that started and crashed the system is a network monitoring server with several ping scripts.
I have "security.jail.allow_raw_sockets=1" active on the system to allow this.

I had to reboot in single user mode and disable the autostart of my jails.

I've updated all jails to 8.0p3 (with ezjail - update binary via FTP) and rebuilt all ports to make sure that this was not the issue.

That did not solve my problem.
However, once the system is booted, I can start the jails via ezjail with crashing. The crash only happens when the jails start on boot.

I think this is caused by the security fixes in p3 for the jail system.
I've never had this problem before this.
>How-To-Repeat:
It happens everytime I boot with ezjail_enable="YES".
After the first jail start, the system goes down with kernel panic / page fault.
>Fix:
I does not happen when I start the jails manually using ezjail-admin onestart [jail]

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-amd64->freebsd-jail 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 31 05:01:35 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=147162 

From: Andriy Gapon <avg@icyb.net.ua>
To: bug-followup@FreeBSD.org, tom.dewaele@abvv.be
Cc:  
Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail
 starts on boot
Date: Tue, 01 Jun 2010 18:41:26 +0300

 Are you able to get a crash dump?
 Or at least a screen shot/capture of a panic message _with_ backtrace.
 
 Otherwise, chances of getting this resolved are lower.
 
 -- 
 Andriy Gapon

From: <Tom.DEWAELE@abvv.be>
To: <bug-followup@FreeBSD.org>
Cc:  
Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot
Date: Thu, 3 Jun 2010 15:32:03 +0200

 This is a multi-part message in MIME format.
 
 ------_=_NextPart_001_01CB0321.2763EF58
 Content-Type: text/plain;
 	charset="us-ascii"
 Content-Transfer-Encoding: quoted-printable
 
 I've obtained some more information.
 
 =20
 
 I had a hunch this was caused by a PF rule.
 
 I disabled PF and the jails did not cause a panic.
 
 Then I disabled the suspicious rule and again no panic.
 
 Re-enabled the rule -> panic.
 
 =20
 
 This is the rule that is either directly or indirectly involved in the
 kernel panic:
 
 pass out quick route-to (lagg0 10.200.0.1) from 10.200.0.0/24 to !
 10.200.0.0/24
 
 =20
 
 I have this rule because my jails are on another vlan so they need to
 use another interface for outgoing packets.
 
 The problem must be linked to this rerouting.
 
 =20
 
 Today I've built a debugging kernel with the options you suggested.
 
 I was able to get a backtrace with DDB.
 
 =20
 
 You can find the panic here:
 
 http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg
 <http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg>=20
 
 and the backtrace I did here:
 
 http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg
 <http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg>=20
 
 =20
 
 The backtrace has 2 pf_route calls so that confirms again my hunch.
 
 =20
 
 This must be caused by kernel code changes made from 8.0p2 to 8.0p3
 
 8.0p2 did not cause the kernel panic.
 
 =20
 
 
 ------_=_NextPart_001_01CB0321.2763EF58
 Content-Type: text/html;
 	charset="us-ascii"
 Content-Transfer-Encoding: quoted-printable
 
 <html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
 xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
 xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
 xmlns:x=3D"urn:schemas-microsoft-com:office:excel" =
 xmlns:p=3D"urn:schemas-microsoft-com:office:powerpoint" =
 xmlns:a=3D"urn:schemas-microsoft-com:office:access" =
 xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" =
 xmlns:s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" =
 xmlns:rs=3D"urn:schemas-microsoft-com:rowset" xmlns:z=3D"#RowsetSchema" =
 xmlns:b=3D"urn:schemas-microsoft-com:office:publisher" =
 xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadsheet" =
 xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" =
 xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" =
 xmlns:oa=3D"urn:schemas-microsoft-com:office:activation" =
 xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
 xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" =
 xmlns:rtc=3D"http://microsoft.com/officenet/conferencing" =
 xmlns:D=3D"DAV:" xmlns:Repl=3D"http://schemas.microsoft.com/repl/" =
 xmlns:mt=3D"http://schemas.microsoft.com/sharepoint/soap/meetings/" =
 xmlns:x2=3D"http://schemas.microsoft.com/office/excel/2003/xml" =
 xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" =
 xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/" =
 xmlns:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/" =
 xmlns:ds=3D"http://www.w3.org/2000/09/xmldsig#" =
 xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint/dsp" =
 xmlns:udc=3D"http://schemas.microsoft.com/data/udc" =
 xmlns:xsd=3D"http://www.w3.org/2001/XMLSchema" =
 xmlns:sub=3D"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/"=
  xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#" =
 xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" =
 xmlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/" =
 xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance" =
 xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/soap" =
 xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" =
 xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttopart" =
 xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/" =
 xmlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup" =
 xmlns:dssi=3D"http://schemas.microsoft.com/office/2006/digsig" =
 xmlns:mdssi=3D"http://schemas.openxmlformats.org/package/2006/digital-sig=
 nature" =
 xmlns:mver=3D"http://schemas.openxmlformats.org/markup-compatibility/2006=
 " xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
 xmlns:mrels=3D"http://schemas.openxmlformats.org/package/2006/relationshi=
 ps" xmlns:spwp=3D"http://microsoft.com/sharepoint/webpartpages" =
 xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/2006/types"=
  =
 xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/2006/messag=
 es" =
 xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/=
 " =
 xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortalServer/Pub=
 lishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" =
 xmlns:st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40">
 
 <head>
 <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
 charset=3Dus-ascii">
 <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
 <style>
 <!--
  /* Font Definitions */
  @font-face
 	{font-family:"Cambria Math";
 	panose-1:2 4 5 3 5 4 6 3 2 4;}
 @font-face
 	{font-family:Calibri;
 	panose-1:2 15 5 2 2 2 4 3 2 4;}
 @font-face
 	{font-family:Tahoma;
 	panose-1:2 11 6 4 3 5 4 4 2 4;}
 @font-face
 	{font-family:Consolas;
 	panose-1:2 11 6 9 2 2 4 3 2 4;}
  /* Style Definitions */
  p.MsoNormal, li.MsoNormal, div.MsoNormal
 	{margin:0cm;
 	margin-bottom:.0001pt;
 	font-size:11.0pt;
 	font-family:"Calibri","sans-serif";}
 a:link, span.MsoHyperlink
 	{mso-style-priority:99;
 	color:blue;
 	text-decoration:underline;}
 a:visited, span.MsoHyperlinkFollowed
 	{mso-style-priority:99;
 	color:purple;
 	text-decoration:underline;}
 p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
 	{mso-style-priority:99;
 	mso-style-link:"Plain Text Char";
 	margin:0cm;
 	margin-bottom:.0001pt;
 	font-size:10.5pt;
 	font-family:Consolas;}
 p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
 	{mso-style-priority:99;
 	mso-style-link:"Balloon Text Char";
 	margin:0cm;
 	margin-bottom:.0001pt;
 	font-size:8.0pt;
 	font-family:"Tahoma","sans-serif";}
 span.EmailStyle17
 	{mso-style-type:personal-compose;
 	font-family:"Calibri","sans-serif";
 	color:windowtext;}
 span.BalloonTextChar
 	{mso-style-name:"Balloon Text Char";
 	mso-style-priority:99;
 	mso-style-link:"Balloon Text";
 	font-family:"Tahoma","sans-serif";}
 span.PlainTextChar
 	{mso-style-name:"Plain Text Char";
 	mso-style-priority:99;
 	mso-style-link:"Plain Text";
 	font-family:Consolas;}
 .MsoChpDefault
 	{mso-style-type:export-only;}
 @page Section1
 	{size:612.0pt 792.0pt;
 	margin:70.85pt 70.85pt 70.85pt 70.85pt;}
 div.Section1
 	{page:Section1;}
 -->
 </style>
 <!--[if gte mso 9]><xml>
  <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
 </xml><![endif]--><!--[if gte mso 9]><xml>
  <o:shapelayout v:ext=3D"edit">
   <o:idmap v:ext=3D"edit" data=3D"1" />
  </o:shapelayout></xml><![endif]-->
 </head>
 
 <body lang=3DNL-BE link=3Dblue vlink=3Dpurple>
 
 <div class=3DSection1>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>I've obtained some more =
 information.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>I had a hunch this was caused =
 by a PF
 rule.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>I disabled PF and the jails =
 did not
 cause a panic.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>Then I disabled the =
 suspicious rule and
 again no panic.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>Re-enabled the rule -&gt; =
 panic.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>This is the rule that is =
 either directly
 or indirectly involved in the kernel panic:<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>pass out quick route-to =
 (lagg0
 10.200.0.1) from 10.200.0.0/24 to ! 10.200.0.0/24<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>I have this rule because my =
 jails are on
 another vlan so they need to use another interface for outgoing =
 packets.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>The problem must be linked to =
 this
 rerouting.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>Today I've built a debugging =
 kernel with
 the options you suggested.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>I was able to get a backtrace =
 with DDB.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>You can find the panic =
 here:<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><a
 href=3D"http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg"><spa=
 n
 lang=3DEN-US>http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg<=
 /span></a><span
 lang=3DEN-US><o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>and the backtrace I did =
 here:<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><a
 href=3D"http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg"><spa=
 n
 lang=3DEN-US>http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg<=
 /span></a><span
 lang=3DEN-US><o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>The backtrace has 2 pf_route =
 calls so
 that confirms again my hunch.<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>This must be caused by kernel =
 code
 changes made from 8.0p2 to 8.0p3<o:p></o:p></span></p>
 
 <p class=3DMsoPlainText><span lang=3DEN-US>8.0p2 did not cause the =
 kernel panic.<o:p></o:p></span></p>
 
 <p class=3DMsoNormal><span lang=3DEN-US><o:p>&nbsp;</o:p></span></p>
 
 </div>
 
 </body>
 
 </html>
 
 ------_=_NextPart_001_01CB0321.2763EF58--

From: <Tom.DEWAELE@abvv.be>
To: <bug-followup@FreeBSD.org>,
	<Tom.DEWAELE@abvv.be>
Cc:  
Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot
Date: Thu, 3 Jun 2010 16:26:25 +0200

 I've obtained some more information.
 
 I had a hunch this was caused by a PF rule.
 I disabled PF and the jails did not cause a panic.
 Then I disabled the suspicious rule and again no panic.
 Re-enabled the rule -> panic.
 
 This is the rule that is either directly or indirectly involved in the
 kernel panic:
 pass out quick route-to (lagg0 10.200.0.1) from 10.200.0.0/24 to !
 10.200.0.0/24
 
 I have this rule because my jails are on another vlan so they need to
 use another interface for outgoing packets.
 The problem must be linked to this rerouting.
 
 Today I've built a debugging kernel with the options you suggested.
 I was able to get a backtrace with DDB.
 
 You can find the panic here:
 http://tweakers.net/ext/f/iirS2Omm8Nfmbpb0BLKSRSzY/full.jpg
 and the backtrace I did here:
 http://tweakers.net/ext/f/doP5G3aux6tSZdZt6hTtaWTR/full.jpg
 
 The backtrace has 2 pf_route calls so that confirms again my hunch.
 
 This must be caused by kernel code changes made from 8.0p2 to 8.0p3
 8.0p2 did not cause the kernel panic.

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, tom.dewaele@abvv.be
Cc:  
Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail
 starts on boot
Date: Thu, 3 Jun 2010 14:39:59 +0000 (UTC)

 Hi,
 
 the only kernel changes from 8.0p2 to 8.0p3 are related to NFS imho.
 You are not also using NFS with this machine?
 
 If you are not, this is likely due to some changed timing or similar
 and you are just hitting  generel problem that most likely is entirely
 unrelated to jails and would either be a pf@ or a net@ kernel issue.
 
 /bz

From: <Tom.DEWAELE@abvv.be>
To: <bz@FreeBSD.org>,
	<bug-followup@FreeBSD.org>
Cc:  
Subject: RE: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot
Date: Thu, 3 Jun 2010 21:08:44 +0200

 Hi Bjoern,
 
 I am not using NFS.
 
 I received an security announcement concerning jails
 (FreeBSD-SA-10:04.jail) which was fixed in 8.0p3.
 That's why I assumed that there were kernel changes for the jail system.
 Then can be a wrongful assumption of my part.
 
 You can be entirely right that this has nothing the to do with the jail
 system or updating to 8.0p3.
 But this is the first time I ran into this kernel panic.
 It started after my first reboot with the 8.0p3 kernel.
 It panics every time my first jail boots and always due to the ping
 proces.
 
 I did not change my system between 8.0p2 and 8.0p3.
 No hardware changes/firmware updates or changes to pf.conf or rc.conf.
 I only kept my ports up to date.
 
 I feel it is linked to 8.0p3 but it might indeed be a net/pf issue.
 
 Kind regards,
 Tom
 

From: pred@telenet.be
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail
 starts on boot
Date: Wed, 1 Sep 2010 09:55:47 +0200 (CEST)

 ------=_Part_235589_241905550.1283327747791
 Content-Type: text/plain; charset=utf-8
 Content-Transfer-Encoding: quoted-printable
 
 Just a quick note that I've upgraded this server to 8.1-RELEASE and re-enab=
 led the PF routing rule that was causing the kernel panic.=20
 
 The server did a clean reboot so it seems this problem is no longer present=
  in 8.1-RELEASE, which is ofcourse=C2=A0good news.=20
 
 ------=_Part_235589_241905550.1283327747791--
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Mon Sep 6 07:18:03 UTC 2010 
State-Changed-Why:  
Submitter notes that this seems to be fixed on 8.1. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=147162 
>Unformatted:
