From thomas@cuivre.fr.eu.org  Thu Aug  9 15:34:12 2001
Return-Path: <thomas@cuivre.fr.eu.org>
Received: from melchior.cuivre.fr.eu.org (melchior.enst.fr [137.194.161.6])
	by hub.freebsd.org (Postfix) with ESMTP id 0861037B403
	for <FreeBSD-gnats-submit@freebsd.org>; Thu,  9 Aug 2001 15:34:12 -0700 (PDT)
	(envelope-from thomas@cuivre.fr.eu.org)
Received: from melusine.cuivre.fr.eu.org (melusine.enst.fr [137.194.160.34])
	by melchior.cuivre.fr.eu.org (Postfix) with ESMTP id 7D958826F
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 10 Aug 2001 00:34:09 +0200 (CEST)
Received: by melusine.cuivre.fr.eu.org (Postfix, from userid 1000)
	id BD95424D46; Fri, 10 Aug 2001 00:34:11 +0200 (CEST)
Message-Id: <20010809223411.BD95424D46@melusine.cuivre.fr.eu.org>
Date: Fri, 10 Aug 2001 00:34:11 +0200 (CEST)
From: Thomas Quinot <thomas@cuivre.fr.eu.org>
Reply-To: Thomas Quinot <thomas@cuivre.fr.eu.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: 4.4-PREREL/ipf 3.4.20: 'to' rule with tun causes crash
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         29583
>Category:       kern
>Synopsis:       4.4-PREREL/ipf 3.4.20: 'to' rule with tun causes crash
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    darrenr
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Aug 09 15:40:01 PDT 2001
>Closed-Date:    Thu May 02 15:39:34 PDT 2002
>Last-Modified:  Thu May 02 15:39:34 PDT 2002
>Originator:     Thomas Quinot
>Release:        FreeBSD 4.4-PRERELEASE i386
>Organization:
>Environment:
System: FreeBSD melusine.cuivre.fr.eu.org 4.4-PRERELEASE FreeBSD 4.4-PRERELEASE #9: Thu Aug 9 17:33:53 CEST 2001 thomas@melusine.cuivre.fr.eu.org:/usr/obj/usr/src/sys/MELUSINE i386


	
>Description:

I have an ipf rule that performs routing based on source address
(for VPN purposes, using a pipsec tunnel) :

block out log quick on tun0 to tun1:10.3.0.1
  from VPN.IP.ADDR.ESS/32 to any group 11

(group 11 is the outbound group. When an outbound packet has the
VPN tunnel interface address as its source, route it back through
the VPN tunnel (tun1) instead of the default route (tun0)).

This rule used to work as expected with ipfilter 3.4.16 under
FreeBSD 4.3-STABLE. With 4.4-PRERELEASE (ipfilter 3.4.20), it
freezes the machine. On one of my attempts, I obtained a
kernel crash dump. One possible hypothesis is that ipfilter
has corrupted an mbuf while moving the packet from one
interface to another:

Script started on Fri Aug 10 00:00:56 2001
$ gdb -k /usr/obj/usr/src/sys/MELUSINE/kernel.debug vmcore.5
GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-unknown-freebsd"...

IdlePTD 4108288
initial pcb at 344dc0
panicstr: page fault
panic messages:
---
Fatal trap 12: page fault while in kernel mode
fault virtual address	= 0x6c2f6c71
fault code		= supervisor read, page not present
instruction pointer	= 0x8:0xc018b2af
stack pointer	        = 0x10:0xc8f4ad88
frame pointer	        = 0x10:0xc8f4ad98
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 518 (pipsecd)
interrupt mask		= net 
trap number		= 12
panic: page fault

syncing disks... 19 
done
Uptime: 26m53s

dumping to dev #ad/0x20009, offset 270360
dump ata0: resetting devices .. done
127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 44 43 [CTRL-C to abort] 42 [CTRL-C to abort] 41 [CTRL-C to abort] 40 [CTRL-C to abort] 39 [CTRL-C to abort] 38 [CTRL-C to abort] 37 [CTRL-C to abort] 36 [CTRL-C to abort] 35 34 [CTRL-C to abort] 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:472
472		if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:472
#1  0xc016fbfd in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:312
#2  0xc016ffe5 in panic (fmt=0xc02f928c "%s")
    at /usr/src/sys/kern/kern_shutdown.c:580
#3  0xc02ae91b in trap_fatal (frame=0xc8f4ad48, eva=1815047281)
    at /usr/src/sys/i386/i386/trap.c:951
#4  0xc02ae5d5 in trap_pfault (frame=0xc8f4ad48, usermode=0, eva=1815047281)
    at /usr/src/sys/i386/i386/trap.c:844
#5  0xc02ae17f in trap (frame={tf_fs = 134479888, tf_es = -1065877488, 
      tf_ds = -923533296, tf_edi = 6685184, tf_esi = 1815047265, 
      tf_ebp = -923488872, tf_isp = -923488908, tf_ebx = -1065820160, 
      tf_edx = 6685184, tf_ecx = -923488560, tf_eax = 6685184, tf_trapno = 12, 
      tf_err = 0, tf_eip = -1072123217, tf_cs = 8, tf_eflags = 66054, 
      tf_esp = -1065820160, tf_ss = -1065820160})
    at /usr/src/sys/i386/i386/trap.c:443
#6  0xc018b2af in m_freem (m=0x6c2f6c61) at /usr/src/sys/kern/uipc_mbuf.c:618
#7  0xc018b2cd in m_freem (m=0xc0759700) at /usr/src/sys/kern/uipc_mbuf.c:618
#8  0xc01b5c0a in tunread (dev=0xc0cc1e80, uio=0xc8f4aed0, flag=8323072)
    at /usr/src/sys/net/if_tun.c:584
#9  0xc01a7e27 in spec_read (ap=0xc8f4ae5c)
    at /usr/src/sys/miscfs/specfs/spec_vnops.c:253
#10 0xc0242888 in ufsspec_read (ap=0xc8f4ae5c)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:1834
---Type <return> to continue, or q <return> to quit--- 
#11 0xc0242e7d in ufs_vnoperatespec (ap=0xc8f4ae5c)
    at /usr/src/sys/ufs/ufs/ufs_vnops.c:2391
#12 0xc01a3daf in vn_read (fp=0xc0d67f80, uio=0xc8f4aed0, cred=0xc0731880, 
    flags=0, p=0xc8edba40) at vnode_if.h:334
#13 0xc017e149 in dofileread (p=0xc8edba40, fp=0xc0d67f80, fd=9, 
    buf=0x804e660, nbyte=4096, offset=-1, flags=0)
    at /usr/src/sys/sys/file.h:146
#14 0xc017e00a in read (p=0xc8edba40, uap=0xc8f4af80)
    at /usr/src/sys/kern/sys_generic.c:117
#15 0xc02aebba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, 
      tf_edi = 134538848, tf_esi = 0, tf_ebp = -1077936800, 
      tf_isp = -923488300, tf_ebx = 9, tf_edx = 134538496, tf_ecx = 134538532, 
      tf_eax = 3, tf_trapno = 7, tf_err = 2, tf_eip = 672790868, tf_cs = 31, 
      tf_eflags = 518, tf_esp = -1077937148, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1150
#16 0xc029fdd5 in Xint0x80_syscall ()
#17 0x80490ef in ?? ()
(kgdb) fr 8
#8  0xc01b5c0a in tunread (dev=0xc0cc1e80, uio=0xc8f4aed0, flag=8323072)
    at /usr/src/sys/net/if_tun.c:584
584			m_freem(m0);
(kgdb) list
579			m0 = m;
580		}
581	
582		if (m0) {
583			TUNDEBUG("%s%d: Dropping mbuf\n", ifp->if_name, ifp->if_unit);
584			m_freem(m0);
585		}
586		return error;
587	}
588	
(kgdb) print ifp
$1 = (struct ifnet *) 0xc0d5c308
(kgdb) print *ifp
$2 = {if_softc = 0xc0d5c300, if_name = 0xc02d5d20 "tun", if_link = {
    tqe_next = 0x0, tqe_prev = 0xc0cfef10}, if_addrhead = {
    tqh_first = 0xc0d5c200, tqh_last = 0xc0d69c60}, if_pcount = 0, 
  if_bpf = 0x0, if_index = 5, if_unit = 1, if_timer = 0, if_flags = -32687, 
  if_ipending = 0, if_linkmib = 0x0, if_linkmiblen = 0, if_data = {
    ifi_type = 23 '\027', ifi_physical = 0 '\000', ifi_addrlen = 0 '\000', 
    ifi_hdrlen = 0 '\000', ifi_recvquota = 0 '\000', ifi_xmitquota = 0 '\000', 
    ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 0, ifi_ipackets = 999, 
    ifi_ierrors = 0, ifi_opackets = 1127, ifi_oerrors = 0, ifi_collisions = 0, 
    ifi_ibytes = 139268, ifi_obytes = 1176565, ifi_imcasts = 0, 
    ifi_omcasts = 0, ifi_iqdrops = 0, ifi_noproto = 0, ifi_hwassist = 0, 
    ifi_unused = 0, ifi_lastchange = {tv_sec = 997392138, tv_usec = 502288}}, 
  if_multiaddrs = {lh_first = 0xc0d54c20}, if_amcount = 0, 
  if_output = 0xc01b53f8 <tunoutput>, if_start = 0, if_done = 0, 
  if_ioctl = 0xc01b52a0 <tunifioctl>, if_watchdog = 0, if_poll_recv = 0, 
  if_poll_xmit = 0, if_poll_intren = 0, if_poll_slowinput = 0, if_init = 0, 
  if_resolvemulti = 0, if_snd = {ifq_head = 0x0, ifq_tail = 0x0, ifq_len = 0, 
    ifq_maxlen = 50, ifq_drops = 0}, if_poll_slowq = 0x0, if_prefixhead = {
    tqh_first = 0x0, tqh_last = 0xc0d5c3d8}}
(kgdb) fr 6
#6  0xc018b2af in m_freem (m=0x6c2f6c61) at /usr/src/sys/kern/uipc_mbuf.c:618
618			MFREE(m, n);

Note: This was also reported as FreeBSD PR kern/

>How-To-Repeat:

        Create a rule set with a 'to' rule diverting packets from one tun
        interface to another tun interface [it is unknown whether this
        problem occurs with non-tun interfaces].

        Trigger the rule by sending out a matching packet.

>Fix:

None known.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->darrenr 
Responsible-Changed-By: kris 
Responsible-Changed-When: Thu Aug 9 17:07:05 PDT 2001 
Responsible-Changed-Why:  
Darren is the ipf maintainer 

http://www.FreeBSD.org/cgi/query-pr.cgi?pr=29583 

From: Thomas Quinot <thomas@cuivre.fr.eu.org>
To: darrenr@pobox.com, freebsd-gnats-submit@freebsd.org
Cc:  
Subject: kern/29583: 'to' rule with tun causes crash
Date: Fri, 7 Dec 2001 01:15:36 +0100

 Darren,
 
 I have just re-tested the problem described in FreeBSD PR kern/29583
 ('to' redirection based on source IP address causes crash). On a
 4.4-STABLE cvsupped today, with your 3.4.22 IPFilter release, I still
 obtained an instantaneous reboot as soon as a packet matching the
 rule went out of the machine.
 
 Thomas.
 
 -- 
     Thomas.Quinot@Cuivre.FR.EU.ORG

From: Thomas Quinot <thomas@cuivre.fr.eu.org>
To: Darren Reed <darrenr@reed.wattle.id.au>, darrenr@pobox.com
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: kern/29583: 'to' rule with tun causes crash
Date: Tue, 15 Jan 2002 23:11:41 +0100

 In a previous email, I wrote:
 
 > I have just re-tested the problem described in FreeBSD PR kern/29583
 > ('to' redirection based on source IP address causes crash). On a
 > 4.4-STABLE cvsupped today, with your 3.4.22 IPFilter release, I still
 > obtained an instantaneous reboot as soon as a packet matching the
 > rule went out of the machine.
 
 Tried 3.4.23 on 4.5-RC. There is definitely an improvement: the system
 does not crash anymore -- it hangs instead, stuck in a tight loop in
 the kernel. Fortunately, I was able to drop into DDB and make it panic,
 so we now have a clean crash dump, and an identification of the point
 where it is hanging. See attached typescript...
 
 Please let me know if there is anything I can do to help tracking this
 problem down.
 
 Script started on Tue Jan 15 22:59:18 2002
 # gdb -k /usr/obj/usr/src/sys/MELUSINE/kernel.debug vmcore.0
 GNU gdb 4.18
 Copyright 1998 Free Software Foundation, Inc.
 GDB is free software, covered by the GNU General Public License, and you are
 welcome to change it and/or distribute copies of it under certain conditions.
 Type "show copying" to see the conditions.
 There is absolutely no warranty for GDB.  Type "show warranty" for details.
 This GDB was configured as "i386-unknown-freebsd"...
 IdlePTD 4194304
 initial pcb at 3555e0
 panicstr: from debugger
 panic messages:
 ---
 panic: from debugger
 
 syncing disks... 16 
 done
 Uptime: 5m45s
 
 dumping to dev #ad/0x20009, offset 262168
 dump ata0: resetting devices .. done
 [...]
 ---
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:473
 473		if (dumping++) {
 (kgdb) bt
 #0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:473
 #1  0xc0174a81 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:313
 #2  0xc0174e71 in panic (fmt=0xc02d9ae4 "from debugger")
     at /usr/src/sys/kern/kern_shutdown.c:581
 #3  0xc014aa31 in db_panic (addr=-1070927120, have_addr=0, count=-1, 
     modif=0xd9c5bb78 "") at /usr/src/sys/ddb/db_command.c:435
 #4  0xc014a9d0 in db_command (last_cmdp=0xc0315c24, cmd_table=0xc0315a64, 
     aux_cmd_tablep=0xc034e058) at /usr/src/sys/ddb/db_command.c:333
 #5  0xc014aa96 in db_command_loop () at /usr/src/sys/ddb/db_command.c:457
 #6  0xc014cbc3 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:71
 #7  0xc02af09a in kdb_trap (type=3, code=0, regs=0xd9c5bc98)
     at /usr/src/sys/i386/i386/db_interface.c:158
 
 #8  0xc02bbd94 in trap (frame={tf_fs = -641400816, tf_es = 16, tf_ds = 16, 
       tf_edi = -1070175424, tf_esi = 0, tf_ebp = -641352480, 
       tf_isp = -641352508, tf_ebx = 134, tf_edx = 4325376, tf_ecx = 32, 
       tf_eax = 38, tf_trapno = 3, tf_err = 0, tf_eip = -1070927120, tf_cs = 8, 
       tf_eflags = 582, tf_esp = -1070567809, tf_ss = -1070581751})
     at /usr/src/sys/i386/i386/trap.c:574
 #9  0xc02af2f0 in Debugger (msg=0xc0303809 "manual escape to debugger")
     at machine/cpufunc.h:67
 #10 0xc02aabb6 in scgetc (sc=0xc036c540, flags=2)
     at /usr/src/sys/dev/syscons/syscons.c:3148
 #11 0xc02a73c5 in sckbdevent (thiskbd=0xc0365240, event=0, arg=0xc036c540)
     at /usr/src/sys/dev/syscons/syscons.c:616
 ---Type <return> to continue, or q <return> to quit---
 #12 0xc029ef9e in atkbd_intr (kbd=0xc0365240, arg=0x0)
     at /usr/src/sys/dev/kbd/atkbd.c:462
 #13 0xc02c9140 in atkbd_isa_intr (arg=0xc0365240)
     at /usr/src/sys/isa/atkbd_isa.c:140
 #14 0xc0172e8e in add_interrupt_randomness (vsc=0xc036bdcc)
     at /usr/src/sys/kern/kern_random.c:245
 #15 0xc01952b4 in sbappendaddr (sb=0xd7ccdf0c, asa=0xc031e020, m0=0xc1826500, 
     control=0xc1826500) at /usr/src/sys/kern/uipc_socket2.c:657
 #16 0xc019837e in uipc_send (so=0xd7ccd740, flags=0, m=0xc1826500, nam=0x0, 
     control=0x0, p=0xd6ddac20) at /usr/src/sys/kern/uipc_usrreq.c:300
 #17 0xc0192ce7 in sosend (so=0xd7ccd740, addr=0x0, uio=0xd9c5bec4, 
     top=0xc1826500, control=0x0, flags=0, p=0xd6ddac20)
     at /usr/src/sys/kern/uipc_socket.c:611
 #18 0xc019662d in sendit (p=0xd6ddac20, s=3, mp=0xd9c5bf04, flags=0)
     at /usr/src/sys/kern/uipc_syscalls.c:583
 #19 0xc0196732 in sendto (p=0xd6ddac20, uap=0xd9c5bf80)
     at /usr/src/sys/kern/uipc_syscalls.c:636
 #20 0xc02bc6ba in syscall2 (frame={tf_fs = 47, tf_es = 47, tf_ds = 47, 
       tf_edi = -1077943328, tf_esi = 672154340, tf_ebp = -1077943412, 
       tf_isp = -641351724, tf_ebx = 672333636, tf_edx = 97, 
       tf_ecx = -1077942260, tf_eax = 133, tf_trapno = 22, tf_err = 2, 
       tf_eip = 672036456, tf_cs = 31, tf_eflags = 647, tf_esp = -1077943472, 
       tf_ss = 47}) at /usr/src/sys/i386/i386/trap.c:1157
 #21 0xc02afe55 in Xint0x80_syscall ()
 ---Type <return> to continue, or q <return> to quit---
 #22 0x281046d7 in ?? ()
 #23 0x28104351 in ?? ()
 #24 0x80685da in ?? ()
 #25 0x80665cb in ?? ()
 #26 0x80668c9 in ?? ()
 #27 0x8065eb9 in ?? ()
 #28 0x8065986 in ?? ()
 #29 0x806117f in ?? ()
 #30 0x80515c9 in ?? ()
 #31 0x805cf4f in ?? ()
 #32 0x805cc28 in ?? ()
 #33 0x8049a57 in ?? ()
 (kgdb) fr 15
 #15 0xc01952b4 in sbappendaddr (sb=0xd7ccdf0c, asa=0xc031e020, m0=0xc1826500, 
     control=0xc1826500) at /usr/src/sys/kern/uipc_socket2.c:657
 657			sballoc(sb, n);
 (kgdb) list
 652			n->m_next = m0;		/* concatenate data to control */
 653		else
 654			control = m0;
 655		m->m_next = control;
 656		for (n = m; n; n = n->m_next)
 657			sballoc(sb, n);
 658		n = sb->sb_mb;
 659		if (n) {
 660			while (n->m_nextpkt)
 661				n = n->m_nextpkt;
 (kgdb) print n
 $1 = (struct mbuf *) 0xc1826500
 (kgdb) print n->m_hdr.mh_next
 $3 = (struct mbuf *) 0xc1826500
 (kgdb) print *sb
 $5 = {sb_cc = 1983817059, sb_hiwat = 4096, sb_mbcnt = 1676293120, 
   sb_mbmax = 32768, sb_lowat = 1, sb_mb = 0xc1853f00, sb_sel = {si_pid = 0, 
     si_note = {slh_first = 0x0}, si_flags = 0}, sb_flags = 0, sb_timeo = 0}
 (kgdb) quit
 
 -- 
     Thomas.Quinot@Cuivre.FR.EU.ORG

From: Thomas Quinot <thomas@cuivre.fr.eu.org>
To: bug-followup@freebsd.org
Cc:  
Subject: kern/29583: Routing based on source address problem
Date: Tue, 22 Jan 2002 23:43:51 +0100

 Guido van Rooij provided the following explanation for the cause
 of the crashes:
 
 ----- Forwarded message from Guido van Rooij <guido@gvr.org> -----
 
 Date: Wed, 16 Jan 2002 16:18:14 +0100
 From: Guido van Rooij <guido@gvr.org>
 Cc: ipfilter@coombs.anu.edu.au
 Subject: Re: Routing based on source address problem (with request to Darren)
 
 On Wed, Jan 16, 2002 at 03:40:35PM +0100, Thomas Quinot wrote:
 > All the details are in PR kern/29583, with the exact ipf rules involved
 > and a few debug traces:
 >   http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/29583
 > 
 
 The most important part of PR is the following rule:
 
 block out log quick on tun0 to tun1:10.3.0.1
   from VPN.IP.ADDR.ESS/32 to any group 11
 
 Darren: I think IPfilter should forbid blocking of a packet using
 the 'to' directive.
 
 Explanation:
 
 "to" will ensure that the packet gets fastrouted. This means the mbuf will
 be queued at the specific interface queue to be sent out.
 If you block the packet, it will be freed (not sure if that is
 in ipfilter or in ip_input or whatever).
 This means we have a race: the freeing versus the use in the tun1
 interface queue.
 
 I guess that is the problem here.
 
 [...]
 
 ----- End forwarded message -----
 
 A work-around was to rewrite the aforementioned rule as:
   pass out log quick on tun0 to tun1:10.3.0.1
     from VPN.IP.ADDR.ESS/32 to any group 11
 which, contrary to what is written in the IPF-HOWTO,
 won't panic the machine.
 
 Thomas.
 
 -- 
     Thomas.Quinot@Cuivre.FR.EU.ORG
State-Changed-From-To: open->feedback 
State-Changed-By: darrenr 
State-Changed-When: Sat Apr 27 13:05:30 PDT 2002 
State-Changed-Why:  
please test this with -stabel or -current as of today. many changes 
have been made which more than likely stop these panics (I hope!) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=29583 

From: Thomas Quinot <thomas@cuivre.fr.eu.org>
To: darrenr@FreeBSD.org
Cc: bug-followup@freebsd.org
Subject: Re: kern/29583: 4.4-PREREL/ipf 3.4.20: 'to' rule with tun causes crash
Date: Fri, 3 May 2002 00:00:20 +0200

 Le 2002-04-27, darrenr@FreeBSD.org crivait :
 
 > please test this with -stabel or -current as of today. many changes
 > have been made which more than likely stop these panics (I hope!)
 
 Confirmed. IPF 3.4.27, MFC'd in FreeBSD 4-STABLE, does not exhibit
 this crash anymore. Thanks!
 
 -- 
     Thomas.Quinot@Cuivre.FR.EU.ORG
State-Changed-From-To: feedback->closed 
State-Changed-By: darrenr 
State-Changed-When: Thu May 2 15:38:27 PDT 2002 
State-Changed-Why:  
fixed in both -stable and -current 

http://www.freebsd.org/cgi/query-pr.cgi?pr=29583 
>Unformatted:
