From dmitry@bis.ru  Wed Oct 27 16:52:32 2004
Return-Path: <dmitry@bis.ru>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 74CE016A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 27 Oct 2004 16:52:32 +0000 (GMT)
Received: from bis.ru (ns.citlab.ru [195.128.76.226])
	by mx1.FreeBSD.org (Postfix) with SMTP id 2B6C643D1D
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 27 Oct 2004 16:52:31 +0000 (GMT)
	(envelope-from dmitry@bis.ru)
Received: (qmail 94210 invoked by uid 1010); 27 Oct 2004 16:52:29 -0000
Message-Id: <20041027165229.94209.qmail@bis.ru>
Date: 27 Oct 2004 16:52:29 -0000
From: Dmitry Miloserdov <dmitry@bis.ru>
Reply-To: Dmitry Miloserdov <dmitry@bis.ru>
To: <FreeBSD-gnats-submit@freebsd.org>
Subject: panic by duplicating UDP NFS v2 packets
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         73208
>Category:       kern
>Synopsis:       [5.3-RC1] panic by duplicating UDP NFS v2 packets
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    rwatson
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 27 17:00:46 GMT 2004
>Closed-Date:    Sat Apr 09 22:20:12 GMT 2005
>Last-Modified:  Sat Apr 09 22:20:12 GMT 2005
>Originator:     Dmitry Miloserdov
>Release:        FreeBSD 5.3-RC1 i386
>Organization:
>Environment:
System: FreeBSD dhcp.bis.local 5.3-RC1 FreeBSD 5.3-RC1 #0: Wed Oct 27 15:48:02 MSD 2004 dmitry@dhcp.bis.local:/usr/obj/u/src5/sys/DHCP i386

>Description:
	System creshes when NFS server receive two same packets
	in a short period of time and command in them must be rejected
	by access control. In my opinion access control itself is not
	the reason of crash - it just helps exploit a race somethere.
	// Feel free to ignore my opinion
	BTW sending duplicate for most control NFS packets is default
	behavior of UnixWare NFS client.

	/etc/exports:
	/u -alldirs -mapall=www
	---
	ls -ld /u/db
	drwxr-xr-x  2 www  www  3072 25 Oct 21:54 /u/db
	---
	On client trying create file /u/db/fil (which is allowed)
	and then client's creat() syscall trying to change group
	of /u/db/fil to primary group of client's user (which is denied).
	tethereal -td:
	1 0.021505  192.168.1.4 -> 10.1.1.1 NFS V2 LOOKUP Call, DH:0x3273bcaa/db
	2 0.000016  192.168.1.4 -> 10.1.1.1 NFS [RPC retransmission of #1]V2 LOOKUP Call, DH:0x3273bcaa/db
	3 0.010767  192.168.1.4 -> 10.1.1.1 NFS V2 LOOKUP Call, DH:0x9d5440aa/fil
	4 0.000015  192.168.1.4 -> 10.1.1.1 NFS [RPC retransmission of #3]V2 LOOKUP Call, DH:0x9d5440aa/fil
	5 0.011850  192.168.1.4 -> 10.1.1.1 NFS V2 CREATE Call, DH:0x9d5440aa/fil
	6 0.000016  192.168.1.4 -> 10.1.1.1 NFS [RPC retransmission of #5]V2 CREATE Call, DH:0x9d5440aa/fil
	7 0.000534  192.168.1.4 -> 10.1.1.1 NFS V2 SETATTR Call, FH:0x7233c0b2
	8 0.000012  192.168.1.4 -> 10.1.1.1 NFS [RPC retransmission of #7]V2 SETATTR Call, FH:0x7233c0b2
	9 0.863791  192.168.1.4 -> 10.1.1.1 NFS [RPC retransmission of #7]V2 SETATTR Call, FH:0x7233c0b2
	---
	On 8th packet system creshes.
	Problem not in content of packet but in packet's frequency as
	blocking half of nfs packets with ipfw allowes system fullfill
	request without panic.

	---
	kernel trap 12 with interrupts disabled
	Fatal trap 12: page fault while in kernel mode
	cpuid = 0; apic id = 00
	fault virtual address   = 0x24
	fault code              = supervisor read, page not present
	instruction pointer     = 0x8:0xc0511337
	stack pointer           = 0x10:0xe4b45b50
	frame pointer           = 0x10:0xe4b45b64
	code segment            = base 0x0, limit 0xfffff, type 0x1b
	                        = DPL 0, pres 1, def32 1, gran 1
	processor eflags        = resume, IOPL = 0
	current process         = 86 (swi1: net)
	trap number             = 12
	panic: page fault
	cpuid = 0
	boot() called on cpu#0
	Uptime: 6m50s
	---
	(kgdb) bt
	#0  doadump () at pcpu.h:159
	#1  0xc04f2293 in boot (howto=260) at /u/src5/sys/kern/kern_shutdown.c:397
	#2  0xc04f25b9 in panic (fmt=0xc064bd2f "%s")
	    at /u/src5/sys/kern/kern_shutdown.c:553
	#3  0xc0629690 in trap_fatal (frame=0xe4b45b10, eva=36)
	    at /u/src5/sys/i386/i386/trap.c:809
	#4  0xc0628e4d in trap (frame=
	      {tf_fs = -65512, tf_es = -457965552, tf_ds = -1068498928, tf_edi = -1041038560, tf_esi = -1066696864, tf_ebp = -457942172, tf_isp = -457942212, tf_ebx = -1041117680, tf_edx = -1041463000, tf_ecx = -1041462912, tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1068428489, tf_cs = 8, tf_eflags = 65683, tf_esp = 40, tf_ss = 0}) at /u/src5/sys/i386/i386/trap.c:247
	#5  0xc0617c3a in calltrap () at /u/src5/sys/i386/i386/exception.s:140
	#6  0xffff0018 in ?? ()
	#7  0xe4b40010 in ?? ()
	#8  0xc0500010 in osethostid (td=0xc1f1ce10, uap=0x0)
	    at /u/src5/sys/kern/kern_xxx.c:145
	#9  0xc0511af1 in turnstile_wait (ts=0xc1ec8980, lock=0xc06b7f60,
	    owner=0xc1f30320) at /u/src5/sys/kern/subr_turnstile.c:556
	#10 0xc04e9899 in _mtx_lock_sleep (m=0xc06b7f60, td=0xc1f1ce10, opts=0,
	    file=0x0, line=0) at /u/src5/sys/kern/kern_mutex.c:560
	#11 0xc05b05ae in nfsrv_rcv (so=0xc234b144, arg=0xc22aa280, waitflag=1)
	    at /u/src5/sys/nfsserver/nfs_srvsock.c:443
	#12 0xc052ba6d in sowakeup (so=0xc234b144, sb=0xc234b194)
	    at /u/src5/sys/kern/uipc_socket2.c:413
	#13 0xc0580e90 in udp_append (last=0xc234b144, ip=0xc27b4810, n=0xc278b300,
	    off=28) at /u/src5/sys/netinet/udp_usrreq.c:509
	#14 0xc0580c93 in udp_input (m=0xc278b300, off=20)
	    at /u/src5/sys/netinet/udp_usrreq.c:402
	#15 0xc056fd1d in ip_input (m=0xc278b300) at /u/src5/sys/netinet/ip_input.c:739
	#16 0xc055c38b in netisr_processqueue (ni=0xc06b03d8)
	    at /u/src5/sys/net/netisr.c:233
	#17 0xc055c7b6 in swi_net (dummy=0x0) at /u/src5/sys/net/netisr.c:346
	#18 0xc04de181 in ithread_loop (arg=0xc1f34200)
	    at /u/src5/sys/kern/kern_intr.c:547
	#19 0xc04dd231 in fork_exit (callout=0xc04de028 <ithread_loop>,
	    arg=0xc1f34200, frame=0xe4b45d48) at /u/src5/sys/kern/kern_fork.c:811
	#20 0xc0617c9c in fork_trampoline () at /u/src5/sys/i386/i386/exception.s:209
	---
	As GENERIC kernel panics too, kernel config skipped.

>How-To-Repeat:
	May be `ipfw tee natd` can emulate my situation but I didn't tried
	myself.

>Fix:
	Disable UDP transport on NFS. But problem seems to be deeper.
>Release-Note:
>Audit-Trail:

From: "Dmitry Miloserdov" <dmitry@bis.ru>
To: <freebsd-gnats-submit@FreeBSD.org>
Cc:  
Subject: kern/73208: [5.3-RC1] panic by duplicating UDP NFS v2 packets
Date: Thu, 11 Nov 2004 15:12:02 +0300

 Problem still exist on 5.3-RELEASE
 Here is panic with WITNESS & INVARIANTS enabled and debug.mpsafenet=0;
 
 Panicstring: mutex nfsd_mtx not owned at
 /u/src5/sys/nfsserver/nfs_serv.c:4493
 (kgdb) bt
 #0  doadump () at pcpu.h:159
 #1  0xc04eb7f8 in boot (howto=260) at /u/src5/sys/kern/kern_shutdown.c:397
 #2  0xc04ebada in panic (fmt=0xc0641d22 "mutex %s not owned at %s:%d")
     at /u/src5/sys/kern/kern_shutdown.c:553
 #3  0xc04e4230 in _mtx_assert (m=0xc06cd160, what=0,
     file=0xc064fdf1 "/u/src5/sys/nfsserver/nfs_serv.c", line=4493)
     at /u/src5/sys/kern/kern_mutex.c:746
 #4  0xc059cde0 in nfsrv_access (vp=0xc272c420, flags=128, cred=0xc26c4580,
     rdonly=0, td=0xc22d07d0, override=0)
     at /u/src5/sys/nfsserver/nfs_serv.c:4493
 #5  0xc05944f6 in nfsrv_create (nfsd=0xc26c4500, slp=0xc2281500,
     td=0xc22d07d0, mrq=0xe6f07ca8) at /u/src5/sys/nfsserver/nfs_serv.c:1913
 #6  0xc05a0439 in nfssvc_nfsd (td=0x0)
     at /u/src5/sys/nfsserver/nfs_syscalls.c:471
 #7  0xc059fd88 in nfssvc (td=0xc22d07d0, uap=0xe6f07d14)
     at /u/src5/sys/nfsserver/nfs_syscalls.c:181
 #8  0xc061009b in syscall (frame=
       {tf_fs = 47, tf_es = 47, tf_ds = 47, tf_edi = -1077940540, tf_esi = 4,
 tf_ebp = -1077941448, tf_isp = -420446860, tf_ebx = 0, tf_edx = 1209258552,
 tf_ecx = 25, tf_eax = 155, tf_trapno = 12, tf_err = 2, tf_eip = 1208756727,
 tf_cs = 31, tf_eflags = 662, tf_esp = -1077941476, tf_ss = 47})
     at /u/src5/sys/i386/i386/trap.c:1001
 #9  0xc05feedf in Xint0x80_syscall () at
 /u/src5/sys/i386/i386/exception.s:201
 
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Thu Nov 11 18:19:56 GMT 2004 
Responsible-Changed-Why:  
I've been working with NFS server locking, can grab ownership of this. 
Thanks for the update with INVARIANTS; should help sort it out much 
faster. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=73208 

From: Robert Watson <rwatson@freebsd.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:  
Subject: Re: kern/73208: [5.3-RC1] panic by duplicating UDP NFS v2 packets
Date: Thu, 11 Nov 2004 20:11:17 +0000 (GMT)

 On Thu, 11 Nov 2004, Robert Watson wrote:
 
 > I've been working with NFS server locking, can grab ownership of this. 
 > Thanks for the update with INVARIANTS; should help sort it out much
 > faster. 
 
 Could you please try the attached patch?  A complete description of the
 patch is present in kern/73208, which I believe to be the same problem.
 The upshot is that nfsrv_acess() was being incorrectly invoked with the
 wrong locks held; this patch cleans up the locking around nfsrv_acacess()
 and I believe corrects the bug.
 
 Index: nfs_serv.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/nfsserver/nfs_serv.c,v
 retrieving revision 1.150
 diff -u -r1.150 nfs_serv.c
 --- nfs_serv.c	22 Oct 2004 08:47:20 -0000	1.150
 +++ nfs_serv.c	11 Nov 2004 18:38:36 -0000
 @@ -137,6 +137,9 @@
  SYSCTL_STRUCT(_vfs_nfsrv, NFS_NFSRVSTATS, nfsrvstats, CTLFLAG_RD,
  	&nfsrvstats, nfsrvstats, "S,nfsrvstats");
  
 +static int	nfsrv_access_withgiant(struct vnode *vp, int flags,
 +		    struct ucred *cred, int rdonly, struct thread *td,
 +		    int override);
  static int	nfsrv_access(struct vnode *, int, struct ucred *, int,
  		    struct thread *, int);
  static void	nfsrvw_coalesce(struct nfsrv_descript *,
 @@ -195,8 +198,10 @@
  		goto nfsmout;
  	}
  	nfsmode = fxdr_unsigned(u_int32_t, *tl);
 +	NFSD_UNLOCK();
 +	mtx_lock(&Giant);	/* VFS */
  	if ((nfsmode & NFSV3ACCESS_READ) &&
 -		nfsrv_access(vp, VREAD, cred, rdonly, td, 0))
 +		nfsrv_access_withgiant(vp, VREAD, cred, rdonly, td, 0))
  		nfsmode &= ~NFSV3ACCESS_READ;
  	if (vp->v_type == VDIR)
  		testmode = (NFSV3ACCESS_MODIFY | NFSV3ACCESS_EXTEND |
 @@ -204,17 +209,15 @@
  	else
  		testmode = (NFSV3ACCESS_MODIFY | NFSV3ACCESS_EXTEND);
  	if ((nfsmode & testmode) &&
 -		nfsrv_access(vp, VWRITE, cred, rdonly, td, 0))
 +		nfsrv_access_withgiant(vp, VWRITE, cred, rdonly, td, 0))
  		nfsmode &= ~testmode;
  	if (vp->v_type == VDIR)
  		testmode = NFSV3ACCESS_LOOKUP;
  	else
  		testmode = NFSV3ACCESS_EXECUTE;
  	if ((nfsmode & testmode) &&
 -		nfsrv_access(vp, VEXEC, cred, rdonly, td, 0))
 +		nfsrv_access_withgiant(vp, VEXEC, cred, rdonly, td, 0))
  		nfsmode &= ~testmode;
 -	NFSD_UNLOCK();
 -	mtx_lock(&Giant);	/* VFS */
  	getret = VOP_GETATTR(vp, vap, cred, td);
  	vput(vp);
  	mtx_unlock(&Giant);	/* VFS */
 @@ -857,12 +860,14 @@
  		else
  			error = (vp->v_type == VDIR) ? EISDIR : EACCES;
  	}
 -	if (!error) {
 -		if ((error = nfsrv_access(vp, VREAD, cred, rdonly, td, 1)) != 0)
 -			error = nfsrv_access(vp, VEXEC, cred, rdonly, td, 1);
 -	}
  	NFSD_UNLOCK();
  	mtx_lock(&Giant);	/* VFS */
 +	if (!error) {
 +		if ((error = nfsrv_access_withgiant(vp, VREAD, cred, rdonly,
 +		    td, 1)) != 0)
 +			error = nfsrv_access_withgiant(vp, VEXEC, cred,
 +			    rdonly, td, 1);
 +	}
  	getret = VOP_GETATTR(vp, vap, cred, td);
  	if (!error)
  		error = getret;
 @@ -1497,9 +1502,11 @@
  		} else {
  		    vp = NULL;
  		}
 -		if (!error)
 -		    error = nfsrv_access(vp, VWRITE, cred, rdonly, td, 1);
  		NFSD_UNLOCK();
 +		mtx_lock(&Giant);	/* VFS */
 +		if (!error)
 +		    error = nfsrv_access_withgiant(vp, VWRITE, cred, rdonly,
 +			td, 1);
  		if (nfsd->nd_stable == NFSV3WRITE_UNSTABLE)
  		    ioflags = IO_NODELOCKED;
  		else if (nfsd->nd_stable == NFSV3WRITE_DATASYNC)
 @@ -1511,7 +1518,6 @@
  		uiop->uio_td = NULL;
  		uiop->uio_offset = nfsd->nd_off;
  		uiop->uio_resid = nfsd->nd_eoff - nfsd->nd_off;
 -		mtx_lock(&Giant);	/* VFS */
  		if (uiop->uio_resid > 0) {
  		    mp = mrep;
  		    i = 0;
 @@ -1910,8 +1916,8 @@
  		}
  	} else {
  		if (vap->va_size != -1) {
 -			error = nfsrv_access(nd.ni_vp, VWRITE, cred,
 -			    (nd.ni_cnd.cn_flags & RDONLY), td, 0);
 +			error = nfsrv_access_withgiant(nd.ni_vp, VWRITE,
 +			    cred, (nd.ni_cnd.cn_flags & RDONLY), td, 0);
  			if (!error) {
  				tempsize = vap->va_size;
  				VATTR_NULL(vap);
 @@ -3370,13 +3376,8 @@
  			error = NFSERR_BAD_COOKIE;
  #endif
  	}
 -	if (!error) {
 -		mtx_unlock(&Giant);	/* VFS */
 -		NFSD_LOCK();
 -		error = nfsrv_access(vp, VEXEC, cred, rdonly, td, 0);
 -		NFSD_UNLOCK();
 -		mtx_lock(&Giant);	/* VFS */
 -	}
 +	if (!error)
 +		error = nfsrv_access_withgiant(vp, VEXEC, cred, rdonly, td, 0);
  	if (error) {
  		vput(vp);
  		mtx_unlock(&Giant);	/* VFS */
 @@ -3685,13 +3686,8 @@
  	if (!error && toff && verf && verf != at.va_filerev)
  		error = NFSERR_BAD_COOKIE;
  #endif
 -	if (!error) {
 -		mtx_unlock(&Giant);	/* VFS */
 -		NFSD_LOCK();
 -		error = nfsrv_access(vp, VEXEC, cred, rdonly, td, 0);
 -		NFSD_UNLOCK();
 -		mtx_lock(&Giant);	/* VFS */
 -	}
 +	if (!error)
 +		error = nfsrv_access_withgiant(vp, VEXEC, cred, rdonly, td, 0);
  	if (error) {
  		vput(vp);
  		mtx_unlock(&Giant);	/* VFS */
 @@ -4473,7 +4469,8 @@
   * Perform access checking for vnodes obtained from file handles that would
   * refer to files already opened by a Unix client. You cannot just use
   * vn_writechk() and VOP_ACCESS() for two reasons.
 - * 1 - You must check for exported rdonly as well as MNT_RDONLY for the write case
 + * 1 - You must check for exported rdonly as well as MNT_RDONLY for the write
 + *     case.
   * 2 - The owner is to be given access irrespective of mode bits for some
   *     operations, so that processes that chmod after opening a file don't
   *     break. I don't like this because it opens a security hole, but since
 @@ -4482,17 +4479,23 @@
   *
   * The exception to rule 2 is EPERM. If a file is IMMUTABLE, VOP_ACCESS()
   * will return EPERM instead of EACCESS. EPERM is always an error.
 + *
 + * There are two versions: one to be called while holding Giant (which is
 + * needed due to use of VFS), and the other called with the NFS server lock
 + * (which will be dropped and reacquired).  This is necessary because
 + * nfsrv_access checks are required from both classes of contexts.
   */
  static int
 -nfsrv_access(struct vnode *vp, int flags, struct ucred *cred, int rdonly,
 -    struct thread *td, int override)
 +nfsrv_access_withgiant(struct vnode *vp, int flags, struct ucred *cred,
 +    int rdonly, struct thread *td, int override)
  {
  	struct vattr vattr;
  	int error;
  
 -	NFSD_LOCK_ASSERT();
 +	GIANT_REQUIRED;
  
  	nfsdbprintf(("%s %d\n", __FILE__, __LINE__));
 +
  	if (flags & VWRITE) {
  		/* Just vn_writechk() changed to check rdonly */
  		/*
 @@ -4517,11 +4520,10 @@
  		if (vp->v_vflag & VV_TEXT)
  			return (ETXTBSY);
  	}
 -	NFSD_UNLOCK();
 -	mtx_lock(&Giant);	/* VFS */
 +
  	error = VOP_GETATTR(vp, &vattr, cred, td);
  	if (error)
 -		goto out;
 +		return (error);
  	error = VOP_ACCESS(vp, flags, cred, td);
  	/*
  	 * Allow certain operations for the owner (reads and writes
 @@ -4529,9 +4531,21 @@
  	 */
  	if (override && error == EACCES && cred->cr_uid == vattr.va_uid)
  		error = 0;
 -out:
 -	NFSD_UNLOCK_ASSERT();
 +	return (error);
 +}
 +
 +static int
 +nfsrv_access(struct vnode *vp, int flags, struct ucred *cred, int rdonly,
 +    struct thread *td, int override)
 +{
 +	int error;
 +
 +	NFSD_LOCK_ASSERT();
 +
 +	NFSD_UNLOCK();
 +	mtx_lock(&Giant);	/* VFS */
 +	error = nfsrv_access_withgiant(vp, flags, cred, rdonly, td, override);
  	mtx_unlock(&Giant);	/* VFS */
  	NFSD_LOCK();
 -	return error;
 +	return (error);
  }
 
 

From: "Dmitry Miloserdov" <dmitry@bis.ru>
To: <freebsd-gnats-submit@FreeBSD.org>
Cc:  
Subject: kern/73208: [5.3-RC1] panic by duplicating UDP NFS v2 packets
Date: Fri, 12 Nov 2004 19:41:24 +0300

 Yes. Patch solves the problem.

From: Robert Watson <rwatson@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/73208: [5.3-RC1] panic by duplicating UDP NFS v2 packets
Date: Sat, 13 Nov 2004 12:39:18 +0000 (GMT)

 On Fri, 12 Nov 2004, Dmitry Miloserdov wrote:
 
 > The following reply was made to PR kern/73208; it has been noted by GNATS.
 > 
 > From: "Dmitry Miloserdov" <dmitry@bis.ru>
 > To: <freebsd-gnats-submit@FreeBSD.org>
 > Cc:  
 > Subject: kern/73208: [5.3-RC1] panic by duplicating UDP NFS v2 packets
 > Date: Fri, 12 Nov 2004 19:41:24 +0300
 > 
 >  Yes. Patch solves the problem.
 
 I have committed this change to CVS HEAD (6-CURRENT) as nfs_serv.c:1.151. 
 I plan to merge this to RELENG_5 after one week, and will request a merge
 to the RELENG_5_3 errata branch after additional testing/shakeout. 
 
 Thanks!
 
 Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
 robert@fledge.watson.org      Principal Research Scientist, McAfee Research
 
 
State-Changed-From-To: open->patched 
State-Changed-By: rwatson 
State-Changed-When: Fri Nov 19 12:06:41 GMT 2004 
State-Changed-Why:  
This patch has been committed to HEAD as nfs_serv.c:1.151, and now merged 
to RELENG_5 as 1.147.2.3.  I will merge to RELENG_5_3 as an errata branch 
fix after additional testing. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73208 

From: Matteo Riondato <rionda@gufi.org>
To: Gnats PR Database <freebsd-gnats-submit@freebsd.org>
Cc: rwatson@freebsd.org
Subject: Re: kern/73208: [5.3-RC1] panic by duplicating UDP NFS v2 packets
Date: Fri, 8 Apr 2005 22:34:18 +0200

 --FZIkiClxIZ9JeWSb
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 This was fixed and MFCed both to RELENG_5 and RELENG_5_3, so this PR
 can be closed.
 Thank you.
 Best Regards
 --=20
 Rionda aka Matteo Riondato
 Disinformato per default
 G.U.F.I. Staff Member (http://www.gufi.org)
 FreeSBIE Developer (http://www.freesbie.org)
 
 --FZIkiClxIZ9JeWSb
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.0 (FreeBSD)
 
 iD4DBQFCVurK2Mp4pR7Fa+wRAjjiAJd1Fr72P+z3j3UbxeUs8xbVtHbJAKCsm9vB
 0c0oQxPF8GFTk1qTCuwukA==
 =hC0D
 -----END PGP SIGNATURE-----
 
 --FZIkiClxIZ9JeWSb--
State-Changed-From-To: patched->closed 
State-Changed-By: rwatson 
State-Changed-When: Sat Apr 9 22:19:46 GMT 2005 
State-Changed-Why:  
Closed as this problem is believed to be fixed and merged.  Thanks for the 
report (and reminder!). 


http://www.freebsd.org/cgi/query-pr.cgi?pr=73208 
>Unformatted:
