From nobody@FreeBSD.org  Thu Oct 28 03:33:19 2004
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 415A616A4CF
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 28 Oct 2004 03:33:19 +0000 (GMT)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id C4B5643D1F
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 28 Oct 2004 03:33:18 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.12.11/8.12.11) with ESMTP id i9S3XIOg029175
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 28 Oct 2004 03:33:18 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.11/8.12.11/Submit) id i9S3XIuO029170;
	Thu, 28 Oct 2004 03:33:18 GMT
	(envelope-from nobody)
Message-Id: <200410280333.i9S3XIuO029170@www.freebsd.org>
Date: Thu, 28 Oct 2004 03:33:18 GMT
From: Paul <paul79@optonline.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [PANIC] Reproducable Page Fault
X-Send-Pr-Version: www-2.3

>Number:         73225
>Category:       kern
>Synopsis:       [sched_ule] [panic] reproducable Page Fault
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    jeff
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Oct 28 03:40:30 GMT 2004
>Closed-Date:    Tue Jun 12 04:57:16 GMT 2007
>Last-Modified:  Tue Jun 12 04:57:16 GMT 2007
>Originator:     Paul
>Release:        5.3-RELEASE
>Organization:
DataPipe
>Environment:
Lucky# uname -a
FreeBSD Lucky 5.3-RELEASE FreeBSD 5.3-RELEASE #1: Mon Oct 25 22:17:04 EDT 2004     root@Lucky:/usr/obj/usr/src/sys/LUCKY  i386
>Description:
Fatal double fault
eip = 0x05e1ed0
esp = 0xd400e000
ebp = 0xd400e004
Panic: double fault

The above fault happens when net.inet.ip.fw.one_pass is set to 0 *on boot* of the machine.

Again, *on boot*.

Settings on my firewall here that may be of importance to reproduce the crash:
#########################
#########################
Lucky# fgrep inet /etc/sysctl.conf 
net.inet.ip.fw.one_pass=0
#########################
#########################
* Please note that gen_goodlist.php is a simple script that generates lists of ip addresses to block in the form of 206.191.24.64/27 *
Lucky# cat /root/etc/rc.firewall 
#!/bin/sh

for i in `/root/bin/gen_goodlist.php`;
do
        /sbin/ipfw -q table 0 add $i;
done

/sbin/ipfw -q pipe 1 config bw 1024Kbit/s
/sbin/ipfw -q pipe 2 config bw 50Kbit/s
/sbin/ipfw -q queue 1 config pipe 1 weight 7
/sbin/ipfw -q queue 2 config pipe 1 weight 5
/sbin/ipfw -q queue 3 config pipe 2 weight 7
/sbin/ipfw -q add 00900 deny ip from table\(0\) to me in via sis0
/sbin/ipfw -q add 00901 deny ip from me to table\(0\) out via sis0
/sbin/ipfw -q add 00902 deny ip from any to me frag in via sis0
/sbin/ipfw -q add 01000 queue 1 tcp from xx.xxx.xxx.xx to me \
dst-port 22,1021,1023,1234,2049 in via sis0 setup keep-state
/sbin/ipfw -q add 01100 queue 1 tcp from me to any \
dst-port 22,25,110,80,443 out via sis0 setup keep-state
/sbin/ipfw -q add 01200 queue 1 udp from me to any out keep-state
/sbin/ipfw -q add 01300 queue 1 icmp from me to any out keep-state
/sbin/ipfw -q add 01400 queue 3 tcp from me to any out via sis0 setup keep-state
/sbin/ipfw -q add 65534 deny log ip from any to any

###########################
###########################
Lucky# ipfw table 0 list | wc -l
   16668

###########################
###########################
Lucky# dmesg
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 5.3-RELEASE #1: Mon Oct 25 22:17:04 EDT 2004
    root@Lucky:/usr/obj/usr/src/sys/LUCKY
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) processor (1334.97-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x644  Stepping = 4
  Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
  AMD Features=0xc0440000<RSVD,AMIE,DSP,3DNow!>
real memory  = 536805376 (511 MB)
avail memory = 519823360 (495 MB)
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <AMD-761 host to PCI bridge> pcibus 0 on motherboard
pir0: <PCI Interrupt Routing Table: 10 Entries> on motherboard
pci0: <PCI bus> on pcib0
agp0: <AMD 761 host to AGP bridge> port 0xd000-0xd003 mem 0xfb004000-0xfb004fff,0xf0000000-0xf3ffffff at device 0.0 on pci0
pcib1: <PCIBIOS PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pci1: <display, VGA> at device 5.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <VIA 82C686B UDMA100 controller> port 0xd400-0xd40f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 7.1 on pci0
ata0: channel #0 on atapci0
ata1: channel #1 on atapci0
uhci0: <VIA 83C572 USB controller> port 0xd800-0xd81f irq 5 at device 7.2 on pci0
uhci0: [GIANT-LOCKED]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xdc00-0xdc1f irq 5 at device 7.3 on pci0
uhci1: [GIANT-LOCKED]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
pci0: <serial bus, SMBus> at device 7.4 (no driver attached)
fwohci0: <Texas Instruments TSB43AB23> mem 0xfb000000-0xfb003fff,0xfb006000-0xfb0067ff irq 5 at device 11.0 on pci0
fwohci0: OHCI version 1.10 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:30:95:00:a0:02:52:94
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
sis0: <NatSemi DP8381[56] 10/100BaseTX> port 0xe000-0xe0ff mem 0xfb005000-0xfb005fff irq 10 at device 13.0 on pci0
sis0: Silicon Revision: DP83815D
miibus0: <MII bus> on sis0
ukphy0: <Generic IEEE 802.3u media interface> on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis0: Ethernet address: 00:09:5b:07:8a:e5
pci0: <multimedia, audio> at device 15.0 (no driver attached)
sis1: <NatSemi DP8381[56] 10/100BaseTX> port 0xe800-0xe8ff mem 0xfb007000-0xfb007fff irq 11 at device 17.0 on pci0
sis1: Silicon Revision: DP83815D
miibus1: <MII bus> on sis1
ukphy1: <Generic IEEE 802.3u media interface> on miibus1
ukphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
sis1: Ethernet address: 00:09:5b:1a:3a:f3
cpu0 on motherboard
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5 irq 6 drq 2 on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <PNP0f13> can't assign resources (irq)
unknown: <PNP0700> can't assign resources (port)
unknown: <PNP0400> can't assign resources (port)
Timecounter "TSC" frequency 1334966604 Hz quality 800
Timecounters tick every 10.000 msec
ipfw2 initialized, divert enabled, rule-based forwarding disabled, default to accept, logging unlimited
ad0: 29312MB <Maxtor 33073H3/YAH814Y0> [59556/16/63] at ata0-master UDMA100
ad1: 95396MB <WDC WD1000BB-00CAA1/17.07W17> [193821/16/63] at ata0-slave UDMA100
acd0: CDRW <SONY CD-RW CRX168B/1.0a> at ata1-master UDMA33
Mounting root from ufs:/dev/ad0s1a
WARNING: /mnt/disk2 was not properly dismounted
sis0: Applying short cable fix (reg=e6)
sis0: Applying short cable fix (reg=e8)
KLD vmmon_up.ko: depends on linux - not available
KLD vmnet.ko: depends on linux - not available

##################################
##################################
*Kernel Configuration with comments stripped out*
machine         i386
cpu             I686_CPU
ident           LUCKY
options         SCHED_ULE
options         INET
options         NFSCLIENT
options         NFSSERVER
options         FFS
options         SOFTUPDATES
options         UFS_DIRHASH
options         MD_ROOT
options         MSDOSFS
options         CD9660
options         PROCFS
options         PSEUDOFS
options         COMPAT_43
options         COMPAT_FREEBSD4
options         SCSI_DELAY=15000
options         KTRACE
options         SYSVSHM
options         SYSVMSG
options         SYSVSEM
options         _KPOSIX_PRIORITY_SCHEDULING
options         KBD_INSTALL_CDEV
options         AHC_REG_PRETTY_PRINT    
options         AHD_REG_PRETTY_PRINT    
device          isa
device          eisa
device          pci
device          fdc
device          ata
device          atadisk                 
device          atapicd                 
device          atapifd                 
options         ATA_STATIC_ID           
device          scbus           
device          da
device          pass            
device          atkbdc          
device          atkbd           
device          psm             
device          vga             
device          splash          
device          sc
options         XSERVER                 
device          agp             
device          npx
device          pmtimer
device          ppc
device          ppbus           
device          lpt             
device          ppi 
device          miibus
device          sis
device          random
device          loop
device          ether
device          pty 
device          tun
device          md 
device          bpf
device          uhci                    
device          ohci                    
device          usb
device          ugen
device          uhid            
device          ukbd
device          ulpt            
device          umass           
device          ums             
device          uscanner        
device          firewire        
device          sbp             
device          apic
device          mem
device          io              
options         CPU_ATHLON_SSE_HACK
options IPFIREWALL
options DUMMYNET
options IPDIVERT
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
>How-To-Repeat:
Given the settings above, my machine always crashes during boot.  If I comment out the loading of the firewall, it boots normally.  After which I can load the firewall by hand with no problems.  It's actually a mistake that net.inet.ip.fw.one_pass=0 was set, as it was initally set to test other things.  That's not what I wanted, however it still creates a reproducable fault with it set to 0.

Once again, I can only produce this on boot of the machine.
>Fix:
Unknown.  I haven't been able *yet* to get a kernel dump, but in the event that I do, I will provide that information here as well.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-i386->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sat Oct 30 05:47:03 GMT 2004 
Responsible-Changed-Why:  
This does not sound i386-specific. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73225 

From: Gleb Smirnoff <glebius@cell.sick.ru>
To: Paul <paul79@optonline.net>
Cc: freebsd-gnats-submit@freebsd.org
Subject: kern/73225: [PANIC] Reproducable Page Fault
Date: Sun, 12 Dec 2004 15:44:51 +0300

   Paul,
 
   I have several questions, to narrow scope of problem search.
 
 1) Is there active traffic flow when machine reboots?
 
 2) Is the problem reproducible if you remove all 'queue' rules
    from rc.firewall?
  If the answer is 'no', can you pls add these rules one by one
  and try to reproduce the problem. I mean first try only rule 1000,
  then 1000 and 1100, etc..
 
 3) Can you try to reproduce the problem with this patch. This is
 not a fix, but it will give some idea on what's going on.
 
 Patch (TABS are lost, apply it by hand):
 
 Index: ip_fw2.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/netinet/ip_fw2.c,v
 retrieving revision 1.85
 diff -u -r1.85 ip_fw2.c
 --- ip_fw2.c    10 Dec 2004 02:17:18 -0000      1.85
 +++ ip_fw2.c    12 Dec 2004 12:38:34 -0000
 @@ -1909,7 +1909,11 @@
  
                 f = args->rule->next_rule;
                 if (f == NULL)
 +#if 0
                         f = lookup_next_rule(args->rule);
 +#else
 +                       print("ipfw: no next rule\n");
 +#endif
         } else {
                 /*
                  * Find the starting rule. It can be either the first
   
 
 -- 
 Totus tuus, Glebius.
 GLEBIUS-RIPN GLEB-RIPE

From: Paul <paul79@optonline.net>
To: Gleb Smirnoff <glebius@cell.sick.ru>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: kern/73225: [PANIC] Reproducable Page Fault
Date: Mon, 13 Dec 2004 02:02:07 -0500

 Hi,
 
 I think I may have isolated the problem.  This is a known issue regarding
 SCHED_ULE being broken.  I'm just one of the lucky ones that build everything
 prior to any warning of it being broken.  I believe this was causing the
 problem, because at this time, I am unable to reproduce the problem.  I have
 reverted back to SCHED_4BSD.
 
 For clarity sake, I will go through each of your questions with whatever
 information I have in case this becomes a problem again, once SCHED_ULE is
 fixed.
 
 1) When rebooting the machine, it doesn't matter whether there's traffic
 going through the interfaces while I issue the 'reboot' command, OR if I 
 boot the machine from a cold start.  This happends irregardless.
 
 2) I didn't get to this part to your question, becaues I was unable to
 *currently* reproduce the problem.  Again, I believe this was do to a
 buggy scheduler.
 
 3) I did have a chance to apply your patch, and I have achieve the following
 result with one_pass set to true and *no* queue lines removed from my ruleset:
 
 kernel: ipfw: no next rule
 kernel: ipfw: ouch!, skip past end of rules, denying packet
 
 *Note: this occur over and over and over.
 
 Thanks a Bunch,
 Paul
 
 On Sun, Dec 12, 2004 at 03:44:51PM +0300, Gleb Smirnoff wrote:
 >   Paul,
 > 
 >   I have several questions, to narrow scope of problem search.
 > 
 > 1) Is there active traffic flow when machine reboots?
 > 
 > 2) Is the problem reproducible if you remove all 'queue' rules
 >    from rc.firewall?
 >  If the answer is 'no', can you pls add these rules one by one
 >  and try to reproduce the problem. I mean first try only rule 1000,
 >  then 1000 and 1100, etc..
 > 
 > 3) Can you try to reproduce the problem with this patch. This is
 > not a fix, but it will give some idea on what's going on.
 > 
 > Patch (TABS are lost, apply it by hand):
 > 
 > Index: ip_fw2.c
 > ===================================================================
 > RCS file: /home/ncvs/src/sys/netinet/ip_fw2.c,v
 > retrieving revision 1.85
 > diff -u -r1.85 ip_fw2.c
 > --- ip_fw2.c    10 Dec 2004 02:17:18 -0000      1.85
 > +++ ip_fw2.c    12 Dec 2004 12:38:34 -0000
 > @@ -1909,7 +1909,11 @@
 >  
 >                 f = args->rule->next_rule;
 >                 if (f == NULL)
 > +#if 0
 >                         f = lookup_next_rule(args->rule);
 > +#else
 > +                       print("ipfw: no next rule\n");
 > +#endif
 >         } else {
 >                 /*
 >                  * Find the starting rule. It can be either the first
 >   
 > 
 > -- 
 > Totus tuus, Glebius.
 > GLEBIUS-RIPN GLEB-RIPE
 > 
Responsible-Changed-From-To: freebsd-bugs->jeff 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Oct 24 09:41:59 GMT 2005 
Responsible-Changed-Why:  
Over to maintainer. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73225 

From: Maxim Konovalov <maxim@macomnet.ru>
To: Paul <paul79@optonline.net>
Cc: bug-followup@freebsd.org
Subject: kern/73225
Date: Tue, 4 Jul 2006 18:45:37 +0400 (MSD)

 Hi,
 
 There were some fixes committed for ULE since 5.3-RELEASE.  Is it
 possible to check it's still an issue with 6.1-RELEASE or RELENG_6?
 
 -- 
 Maxim Konovalov
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Wed Mar 14 22:24:10 UTC 2007 
State-Changed-Why:  
The ULE scheduler has been extensively upgraded in -CURRENT.  Does this 
problem still occur there? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73225 
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Tue Jun 12 04:56:48 UTC 2007 
State-Changed-Why:  
Feedback timeout (2 months). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73225 
>Unformatted:
