From sean-freebsd@farley.org  Thu Jul  6 01:52:44 2006
Return-Path: <sean-freebsd@farley.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id E21FF16A4DE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu,  6 Jul 2006 01:52:44 +0000 (UTC)
	(envelope-from sean-freebsd@farley.org)
Received: from mail.farley.org (farley.org [67.64.95.201])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 1717543D46
	for <FreeBSD-gnats-submit@freebsd.org>; Thu,  6 Jul 2006 01:52:43 +0000 (GMT)
	(envelope-from sean-freebsd@farley.org)
Received: from thor.farley.org (thor.farley.org [IPv6:2001:470:1f01:290:1::5])
	by mail.farley.org (8.13.4/8.13.1) with ESMTP id k661s0R2042509
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 5 Jul 2006 20:54:01 -0500 (CDT)
	(envelope-from sean-freebsd@gw.farley.org)
Received: from thor.farley.org (localhost [127.0.0.1])
	by thor.farley.org (8.13.6/8.13.6) with ESMTP id k661qedM052994
	for <FreeBSD-gnats-submit@freebsd.org>; Wed, 5 Jul 2006 20:52:40 -0500 (CDT)
	(envelope-from sean-freebsd@thor.farley.org)
Received: (from sean@localhost)
	by thor.farley.org (8.13.6/8.13.6/Submit) id k661qeq3052993;
	Wed, 5 Jul 2006 20:52:40 -0500 (CDT)
	(envelope-from sean-freebsd)
Message-Id: <200607060152.k661qeq3052993@thor.farley.org>
Date: Wed, 5 Jul 2006 20:52:40 -0500 (CDT)
From: Sean Farley <sean-freebsd@farley.org>
Reply-To: Sean Farley <sean-freebsd@farley.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: setenv()/unsetenv() leak memory
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         99826
>Category:       kern
>Synopsis:       [libc] [patch] setenv(3)/unsetenv(3) leak memory
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    scf
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jul 06 02:00:32 GMT 2006
>Closed-Date:    Thu Feb 21 22:28:12 CST 2008
>Last-Modified:  Thu Feb 21 22:28:12 CST 2008
>Originator:     Sean Farley
>Release:        FreeBSD 6.1-STABLE i386
>Organization:
>Environment:
System: FreeBSD thor.farley.org 6.1-STABLE FreeBSD 6.1-STABLE #0: Tue Jun 6 12:54:08 CDT 2006 root@thor.farley.org:/usr/obj/usr/src/sys/THOR i386

>Description:
When writing code to use a third-party library, I ran into a leak that
setenv() exhibits when overwriting a variable with a larger value.
Calling unsetenv() will just leak memory.  All of this is documented in
the man page (setenv(3)):

    Successive calls to setenv() or putenv() assigning a differently
    sized value to the same name will result in a memory leak.  The
    FreeBSD seman- tics for these functions (namely, that the contents
    of value are copied and that old values remain accessible
    indefinitely) make this bug unavoidable.  Future versions may
    eliminate one or both of these semantic guarantees in order to fix
    the bug.

Unfortunately, I must reset an environment variable quite a lot
resulting in a quick memory leak.

To prevent the leak, I have written a patch which copies the entire
environment into dynamic memory upon the first setenv().  This way any
time an existing variable has its value updated with a larger value the
old value is freed and replaced with a new copy.  unsetenv() will free
values if setenv() has been called first.  This was tested with dmalloc
and MALLOC_OPTIONS=AX.

A test program can be found here[1] and expanded here[2].

  1. http://www.farley.org/freebsd/tmp/setenv-3.tar.bz2
  2. http://www.farley.org/freebsd/tmp/setenv-3/


>How-To-Repeat:
Run hungry and watch it grow via top.  Run lean and watch it hold
steady.

>Fix:
It can be also found at
http://www.farley.org/freebsd/tmp/setenv-3/setenv.c.diff

-----Begin Fix-----
--- /usr/src/lib/libc/stdlib/setenv.c	Fri Mar 22 15:53:10 2002
+++ setenv.c	Wed Jul  5 20:47:35 2006
@@ -40,9 +40,12 @@
 #include <stddef.h>
 #include <stdlib.h>
 #include <string.h>
+#include <sys/types.h>
 
 char *__findenv(const char *, int *);
 
+static uint8_t dynamicEnv = 0;			/* Env vars were copied */
+
 /*
  * setenv --
  *	Set the value of the environmental variable "name" to be
@@ -59,6 +62,18 @@
 	char *c;
 	int l_value, offset;
 
+	/*
+	 * Make copies of environment variables to allow for
+	 * reallocation.  This stops a potential memory leak
+	 * when resetting a variable with larger values.
+	 */
+	if (!dynamicEnv) {
+		for (offset = 0; environ[offset]; offset++)
+			if ((environ[offset] = strdup(environ[offset])) == NULL)
+				return (-1);
+		dynamicEnv = 1;
+	}
+
 	if (*value == '=')			/* no `=' in value */
 		++value;
 	l_value = strlen(value);
@@ -74,27 +89,21 @@
 		char **p;
 
 		for (p = environ, cnt = 0; *p; ++p, ++cnt);
-		if (alloced == environ) {			/* just increase size */
-			p = (char **)realloc((char *)environ,
-			    (size_t)(sizeof(char *) * (cnt + 2)));
-			if (!p)
-				return (-1);
-			alloced = environ = p;
-		}
-		else {				/* get new space */
-						/* copy old entries into it */
-			p = malloc((size_t)(sizeof(char *) * (cnt + 2)));
-			if (!p)
-				return (-1);
+		p = (char **)realloc((char *)alloced,
+		    (size_t)(sizeof(char *) * (cnt + 2)));
+		if (!p)
+			return (-1);
+		if (alloced != environ) {		/* alloced environ */
+			/* copy old entries into it */
 			bcopy(environ, p, cnt * sizeof(char *));
-			alloced = environ = p;
 		}
+		alloced = environ = p;
 		environ[cnt + 1] = NULL;
 		offset = cnt;
 	}
 	for (c = (char *)name; *c && *c != '='; ++c);	/* no `=' in name */
 	if (!(environ[offset] =			/* name + `=' + value */
-	    malloc((size_t)((int)(c - name) + l_value + 2))))
+	    reallocf(environ[offset], (size_t)((int)(c - name) + l_value + 2))))
 		return (-1);
 	for (c = environ[offset]; (*c = *name++) && *c != '='; ++c);
 	for (*c++ = '='; (*c++ = *value++); );
@@ -113,8 +122,11 @@
 	char **p;
 	int offset;
 
-	while (__findenv(name, &offset))	/* if set multiple times */
+	while (__findenv(name, &offset)) {	/* if set multiple times */
+		if (dynamicEnv)
+			free(environ[offset]);
 		for (p = &environ[offset];; ++p)
 			if (!(*p = *(p + 1)))
 				break;
+	}
 }
-----End Fix-----
>Release-Note:
>Audit-Trail:

From: Sean Farley <sean-freebsd@farley.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/99826: [libc] [patch] setenv(3)/unsetenv(3) leak memory
Date: Fri, 1 Jun 2007 12:11:13 -0500 (CDT)

 I have completed a new version[1] of a replacement (kern/99826[1]) for
 getenv/setenv/putenv/unsetenv().  A patch against CURRENT that updates
 libc and a few base utilities can be found here[2].  I built it and ran
 it successfully.
 
 Notes:
 1. Fixes memory leak as noted in BUGS section for setenv(3).
     Example of the leak:
     setenv("TZ", "CDT", 1);
     setenv("TZ", "YEKST", 1);  // Leaks
     setenv("TZ", "CDT", 1);
     setenv("TZ", "YEKST", 1);  // Leaks
 2. Converts all calls to POSIX from historic BSD API.
     a. unsetenv returns an int.
     b. putenv takes a char * instead of const char *.
     c. putenv no longer makes a copy of the input string.
     d. errno is set appropriately for POSIX.  Exceptions involve bad
        environ variable and internal initialization code.  These both set
        errno to EFAULT.
 3. Several patches to base utilities to handle the POSIX changes are
     from Andrey Chernov's previous commit.  A few I re-wrote to use
     setenv() instead of putenv().  Yes, I dislike putenv().  :)
 4. A new regression module to test these functions was written.  It also
     can be used to test the performance.  I found performance to be
     on-par or better.  It is found in tools/regression/environment.
 5. Man page could use more work to match all the changes.
 6. make universe successfully completed.
 
 Thank you to Andrey and others for your help in proofreading several
 iterations of the code.
 
 
 Sean
    1. http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/99826
    2. http://www.farley.org/freebsd/tmp/setenv/setenv-9/
    3. http://www.farley.org/freebsd/tmp/setenv/setenv-9/setenv.diff
 -- 
 sean-freebsd@farley.org

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/99826: commit references a PR
Date: Wed,  4 Jul 2007 00:00:48 +0000 (UTC)

 scf         2007-07-04 00:00:41 UTC
 
   FreeBSD src repository
 
   Modified files:
     bin/df               df.c 
     bin/sh               var.c 
     include              stdlib.h 
     libexec/pppoed       pppoed.c 
     sys/sys              param.h 
     lib/libc/stdlib      Makefile.inc getenv.3 getenv.c 
     usr.bin/du           du.c 
     usr.bin/env          env.c 
     usr.bin/limits       limits.c 
     usr.bin/login        login.c 
     usr.bin/su           su.c 
     usr.sbin/pstat       pstat.c 
     usr.sbin/sade        main.c variable.c 
     usr.sbin/sysinstall  main.c variable.c 
   Added files:
     tools/regression/environ Makefile Makefile.envctl 
                              Makefile.retention Makefile.timings 
                              envctl.c envtest.t retention.c 
                              timings.c 
   Removed files:
     lib/libc/stdlib      putenv.c setenv.c 
   Log:
   Significantly reduce the memory leak as noted in BUGS section for
   setenv(3) by tracking the size of the memory allocated instead of using
   strlen() on the current value.
   
   Convert all calls to POSIX from historic BSD API:
    - unsetenv returns an int.
    - putenv takes a char * instead of const char *.
    - putenv no longer makes a copy of the input string.
    - errno is set appropriately for POSIX.  Exceptions involve bad environ
      variable and internal initialization code.  These both set errno to
      EFAULT.
   
   Several patches to base utilities to handle the POSIX changes from
   Andrey Chernov's previous commit.  A few I re-wrote to use setenv()
   instead of putenv().
   
   New regression module for tools/regression/environ to test these
   functions.  It also can be used to test the performance.
   
   Bump __FreeBSD_version to 700050 due to API change.
   
   PR:             kern/99826
   Approved by:    wes
   Approved by:    re (kensmith)
   
   Revision  Changes    Path
   1.71      +4 -4      src/bin/df/df.c
   1.36      +13 -3     src/bin/sh/var.c
   1.65      +2 -2      src/include/stdlib.h
   1.54      +4 -4      src/lib/libc/stdlib/Makefile.inc
   1.27      +80 -24    src/lib/libc/stdlib/getenv.3
   1.9       +539 -60   src/lib/libc/stdlib/getenv.c
   1.7       +0 -56     src/lib/libc/stdlib/putenv.c (dead)
   1.15      +0 -116    src/lib/libc/stdlib/setenv.c (dead)
   1.27      +4 -4      src/libexec/pppoed/pppoed.c
   1.305     +1 -1      src/sys/sys/param.h
   1.1       +14 -0     src/tools/regression/environ/Makefile (new)
   1.1       +16 -0     src/tools/regression/environ/Makefile.envctl (new)
   1.1       +16 -0     src/tools/regression/environ/Makefile.retention (new)
   1.1       +16 -0     src/tools/regression/environ/Makefile.timings (new)
   1.1       +154 -0    src/tools/regression/environ/envctl.c (new)
   1.1       +182 -0    src/tools/regression/environ/envtest.t (new)
   1.1       +109 -0    src/tools/regression/environ/retention.c (new)
   1.1       +195 -0    src/tools/regression/environ/timings.c (new)
   1.42      +3 -3      src/usr.bin/du/du.c
   1.19      +6 -1      src/usr.bin/env/env.c
   1.17      +8 -2      src/usr.bin/limits/limits.c
   1.106     +5 -2      src/usr.bin/login/login.c
   1.86      +6 -2      src/usr.bin/su/su.c
   1.102     +3 -3      src/usr.sbin/pstat/pstat.c
   1.80      +1 -1      src/usr.sbin/sade/main.c
   1.42      +4 -1      src/usr.sbin/sade/variable.c
   1.77      +1 -1      src/usr.sbin/sysinstall/main.c
   1.40      +4 -1      src/usr.sbin/sysinstall/variable.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
Responsible-Changed-From-To: freebsd-bugs->scf 
Responsible-Changed-By: scf 
Responsible-Changed-When: Sat Aug 4 13:52:29 CDT 2007 
Responsible-Changed-Why:  
Take ownership of this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=99826 
State-Changed-From-To: open->feedback 
State-Changed-By: scf 
State-Changed-When: Tue Aug 7 14:25:25 CDT 2007 
State-Changed-Why:  
Is the fix acceptable?  :) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=99826 

From: "Sean C. Farley" <sean-freebsd@farley.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/99826: [libc] [patch] setenv(3)/unsetenv(3) leak memory
Date: Tue, 7 Aug 2007 14:49:01 -0500 (CDT)

 The fix works for me.  :)
State-Changed-From-To: feedback->closed 
State-Changed-By: scf 
State-Changed-When: Thu Feb 21 22:15:13 CST 2008 
State-Changed-Why:  
With the replacement of getenv() and family within RELENG_7's libc and 
updates to several base programs to use the POSIX-style API, I am 
declaring this PR to be fixed. 

To verify that the changes worked, I had implemented a regression suite 
for environ that can be found in src/tools/regression/environ.  Using the 
new (POSIX) API, it shows that the environment is valid after a series of 
API calls in various combinations. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=99826 
>Unformatted:
