From nobody@FreeBSD.org  Wed Feb 11 10:11:34 2004
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 4A4EF16A4CE
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 11 Feb 2004 10:11:34 -0800 (PST)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 4682243D31
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 11 Feb 2004 10:11:34 -0800 (PST)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.12.10/8.12.10) with ESMTP id i1BIBY72080043
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 11 Feb 2004 10:11:34 -0800 (PST)
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.10/8.12.10/Submit) id i1BIBYMH080042;
	Wed, 11 Feb 2004 10:11:34 -0800 (PST)
	(envelope-from nobody)
Message-Id: <200402111811.i1BIBYMH080042@www.freebsd.org>
Date: Wed, 11 Feb 2004 10:11:34 -0800 (PST)
From: Kevin Martin <sigma@pair.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: fstat randomly crashes with "out of memory"
X-Send-Pr-Version: www-2.0

>Number:         62699
>Category:       bin
>Synopsis:       fstat randomly crashes with "out of memory"
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 11 10:20:14 PST 2004
>Closed-Date:    Sat Jan 29 11:38:21 GMT 2005
>Last-Modified:  Sun Jul  3 01:10:56 GMT 2005
>Originator:     Kevin Martin
>Release:        4.8-STABLE
>Organization:
pair Networks, Inc
>Environment:
FreeBSD koloda.pair.com 4.8-STABLE FreeBSD 4.8-STABLE #0: Fri Oct  3 11:10:15 EDT 2003     sigma@koloda.pair.com:/usr/src/sys/compile/PAIRqm  i386
>Description:
About 1 in every 1000 runs, "fstat" will die with "Out of memory".  The problem has been traced to line 349 of fstat.c 1.21.2.7, where filed.fd_lastfile occasionally gets a huge, incorrect value, and subsequently tries to malloc several gigabytes to hold the array of files.
>How-To-Repeat:
#!/bin/csh
while (1)
fstat >/dev/null
end

Watch for "Out of memory" instances.
>Fix:
A workaround is simply to insert a sanity check that fd_lastfile is less than, say, 100000.  A more elegant workaround would be a comparison to the real size of the file table.  A real fix would be to find out why garbage ends up in the filed structure, which is read via KVM_READ just a few lines earlier.

>Release-Note:
>Audit-Trail:

From: "Mark W. Krentel" <krentel@dreamscape.com>
To: sigma@pair.com, bug-followup@freebsd.org,
	freebsd-bugs@freebsd.org
Cc: kris@obsecurity.org, alc@cs.rice.edu
Subject: Re: i386/62699: fstat randomly crashes with "out of memory"
Date: Thu, 27 Jan 2005 20:39:48 -0500

 The analysis by the originator, Kevin Martin, is correct.  I looked
 into this and indeed, filed.fd_lastfile does occasionally have a way-
 out-of-bounds value (positive or negative), leading to an "out of
 memory" crash, or worse.
 
 This is the same bug reported by Kris Kennaway in the thread "fstat
 triggered INVARIANTS panic in memrw()" on -current in January 2005.
 Except that Kris was getting a panic because kernacc() in vm_glue.c
 was not checking for address wrap.  That has now been patched.
 
 This is a classic case of the transitory nature of the data obtained
 from kvm_read() in a running kernel.  Even if kvm_read() returns
 successfully, there is no guarantee that the data is meaningful or
 what you expect.  It may be garbage, and even if it's valid, it may be
 out of date by the time you see it.
 
 There is no exact "fix", the best you can do is a sanity check on
 filed.fd_lastfile, as you suggest.  One could argue over the best
 bound, I went with 0x01000000 in the patch below.  In my tests, all
 of the out-of-bounds values were either negative or greater than
 0x70000000, so I decided that 0x01000000 (16 million) was a reasonable
 compromise for the number of file descriptors that are simultaneously
 open.
 
 Note that fd_lastfile == -1 is valid, it just means that the process
 has no open files.  In this case, there's no reason to continue with
 the "open files" section of dofiles().
 
 P.S. This PR should be category "bin", not "i386".  The problem is in
 fstat(1), there's nothing x86-specific about it.
 
 P.P.S. Please limit lines to about 75-80 characters.
 
 --Mark
 
 Patch for the "out of memory" problem:
 
 Index: fstat.c
 ===================================================================
 RCS file: /data/ncvs/src/usr.bin/fstat/fstat.c,v
 retrieving revision 1.57
 diff -u -r1.57 fstat.c
 --- fstat.c	11 Jan 2005 18:52:12 -0000	1.57
 +++ fstat.c	27 Jan 2005 22:31:28 -0000
 @@ -358,6 +358,12 @@
  	 * open files
  	 */
  #define FPSIZE	(sizeof (struct file *))
 +#define	MAX_LASTFILE  (0x01000000)
 +
 +	/* Sanity check on filed.fd_lastfile */
 +	if (filed.fd_lastfile <= -1 || filed.fd_lastfile > MAX_LASTFILE)
 +		return;
 +
  	ALLOC_OFILES(filed.fd_lastfile+1);
  	if (!KVM_READ(filed.fd_ofiles, ofiles,
  	    (filed.fd_lastfile+1) * FPSIZE)) {
 
 
 Two other points, while I'm here.  First, in the macro definition of
 KVM_READ in fstat.h, the check for "(len) < SSIZE_MAX" is pretty much
 pointless.  What positive value for len is ever greater than SSIZE_MAX?
 It would make more sense to check for "(len) > 0" because if len == 0
 then kvm_read() would return 0 bytes and KVM_READ would return true
 but not return any useful data.
 
 Actually, since all but one use of KVM_READ has sizeof(...) for len,
 and the last use is in the patch above, I would skip the test
 altogether and write it as:
 
 #define	KVM_READ(kaddr, paddr, len) \
 	(kvm_read(kd, (u_long)(kaddr), (char *)(paddr), (len)) == (ssize_t)(len))
 
 
 Second, dofiles() should also include the jail root directory, if
 there is one.  I'm guessing that fstat(1) was written long before
 there were jails.
 
 Index: fstat.c
 ===================================================================
 RCS file: /data/ncvs/src/usr.bin/fstat/fstat.c,v
 retrieving revision 1.57
 diff -u -r1.57 fstat.c
 --- fstat.c	11 Jan 2005 18:52:12 -0000	1.57
 +++ fstat.c	28 Jan 2005 01:02:21 -0000
 @@ -106,6 +106,7 @@
  #define	RDIR	-3
  #define	TRACE	-4
  #define	MMAP	-5
 +#define	JDIR	-6
  
  DEVS *devs;
  
 @@ -309,6 +310,9 @@
  	case MMAP: \
  		printf(" mmap"); \
  		break; \
 +	case JDIR: \
 +		printf(" jail"); \
 +		break; \
  	default: \
  		printf(" %4d", i); \
  		break; \
 @@ -345,6 +349,11 @@
  	 */
  	vtrans(filed.fd_cdir, CDIR, FREAD);
  	/*
 +	 * Jail root directory vnode.
 +	 */
 +	if (filed.fd_jdir != NULL)
 +		vtrans(filed.fd_jdir, JDIR, FREAD);
 +	/*
  	 * ktrace vnode, if one
  	 */
  	if (kp->ki_tracep)
Responsible-Changed-From-To: freebsd-i386->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Jan 28 05:29:40 GMT 2005 
Responsible-Changed-Why:  
Patch submitted; submitter notes that this problem is not i386-specific. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=62699 
State-Changed-From-To: open->closed 
State-Changed-By: phk 
State-Changed-When: Sat Jan 29 11:38:12 GMT 2005 
State-Changed-Why:  
Fixed, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=62699 

From: "Mark W. Krentel" <krentel@dreamscape.com>
To: sigma@pair.com, bug-followup@freebsd.org,
	freebsd-bugs@freebsd.org
Cc: alc@cs.rice.edu
Subject: Re: i386/62699: fstat randomly crashes with "out of memory"
Date: Thu, 27 Jan 2005 20:39:48 -0500

 The analysis by the originator, Kevin Martin, is correct.  I looked
 into this and indeed, filed.fd_lastfile does occasionally have a way-
 out-of-bounds value (positive or negative), leading to an "out of
 memory" crash, or worse.
 
 This is the same bug reported by Kris Kennaway in the thread "fstat
 triggered INVARIANTS panic in memrw()" on -current in January 2005.
 Except that Kris was getting a panic because kernacc() in vm_glue.c
 was not checking for address wrap.  That has now been patched.
 
 This is a classic case of the transitory nature of the data obtained
 from kvm_read() in a running kernel.  Even if kvm_read() returns
 successfully, there is no guarantee that the data is meaningful or
 what you expect.  It may be garbage, and even if it's valid, it may be
 out of date by the time you see it.
 
 There is no exact "fix", the best you can do is a sanity check on
 filed.fd_lastfile, as you suggest.  One could argue over the best
 bound, I went with 0x01000000 in the patch below.  In my tests, all
 of the out-of-bounds values were either negative or greater than
 0x70000000, so I decided that 0x01000000 (16 million) was a reasonable
 compromise for the number of file descriptors that are simultaneously
 open.
 
 Note that fd_lastfile == -1 is valid, it just means that the process
 has no open files.  In this case, there's no reason to continue with
 the "open files" section of dofiles().
 
 P.S. This PR should be category "bin", not "i386".  The problem is in
 fstat(1), there's nothing x86-specific about it.
 
 P.P.S. Please limit lines to about 75-80 characters.
 
 --Mark
 
 Patch for the "out of memory" problem:
 
 Index: fstat.c
 ===================================================================
 RCS file: /data/ncvs/src/usr.bin/fstat/fstat.c,v
 retrieving revision 1.57
 diff -u -r1.57 fstat.c
 --- fstat.c	11 Jan 2005 18:52:12 -0000	1.57
 +++ fstat.c	27 Jan 2005 22:31:28 -0000
 @@ -358,6 +358,12 @@
  	 * open files
  	 */
  #define FPSIZE	(sizeof (struct file *))
 +#define	MAX_LASTFILE  (0x01000000)
 +
 +	/* Sanity check on filed.fd_lastfile */
 +	if (filed.fd_lastfile <= -1 || filed.fd_lastfile > MAX_LASTFILE)
 +		return;
 +
  	ALLOC_OFILES(filed.fd_lastfile+1);
  	if (!KVM_READ(filed.fd_ofiles, ofiles,
  	    (filed.fd_lastfile+1) * FPSIZE)) {
 
 
 Two other points, while I'm here.  First, in the macro definition of
 KVM_READ in fstat.h, the check for "(len) < SSIZE_MAX" is pretty much
 pointless.  What positive value for len is ever greater than SSIZE_MAX?
 It would make more sense to check for "(len) > 0" because if len == 0
 then kvm_read() would return 0 bytes and KVM_READ would return true
 but not return any useful data.
 
 Actually, since all but one use of KVM_READ has sizeof(...) for len,
 and the last use is in the patch above, I would skip the test
 altogether and write it as:
 
 #define	KVM_READ(kaddr, paddr, len) \
 	(kvm_read(kd, (u_long)(kaddr), (char *)(paddr), (len)) == (ssize_t)(len))
 
 
 Second, dofiles() should also include the jail root directory, if
 there is one.  I'm guessing that fstat(1) was written long before
 there were jails.
 
 Index: fstat.c
 ===================================================================
 RCS file: /data/ncvs/src/usr.bin/fstat/fstat.c,v
 retrieving revision 1.57
 diff -u -r1.57 fstat.c
 --- fstat.c	11 Jan 2005 18:52:12 -0000	1.57
 +++ fstat.c	28 Jan 2005 01:02:21 -0000
 @@ -106,6 +106,7 @@
  #define	RDIR	-3
  #define	TRACE	-4
  #define	MMAP	-5
 +#define	JDIR	-6
  
  DEVS *devs;
  
 @@ -309,6 +310,9 @@
  	case MMAP: \
  		printf(" mmap"); \
  		break; \
 +	case JDIR: \
 +		printf(" jail"); \
 +		break; \
  	default: \
  		printf(" %4d", i); \
  		break; \
 @@ -345,6 +349,11 @@
  	 */
  	vtrans(filed.fd_cdir, CDIR, FREAD);
  	/*
 +	 * Jail root directory vnode.
 +	 */
 +	if (filed.fd_jdir != NULL)
 +		vtrans(filed.fd_jdir, JDIR, FREAD);
 +	/*
  	 * ktrace vnode, if one
  	 */
  	if (kp->ki_tracep)
 _______________________________________________
 freebsd-bugs@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org"
 
>Unformatted:
