From tejblum@developer.yandex.ru  Sun Mar 20 10:06:54 2005
Return-Path: <tejblum@developer.yandex.ru>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id AC95C16A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 20 Mar 2005 10:06:54 +0000 (GMT)
Received: from developer.yandex.ru (developer.yandex.ru [213.180.193.15])
	by mx1.FreeBSD.org (Postfix) with ESMTP id E261143D49
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 20 Mar 2005 10:06:53 +0000 (GMT)
	(envelope-from tejblum@developer.yandex.ru)
Received: from developer.yandex.ru (localhost [127.0.0.1])
	by developer.yandex.ru (8.13.3/8.13.1) with ESMTP id j2KA6q9k047339
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 20 Mar 2005 13:06:52 +0300 (MSK)
	(envelope-from tejblum@developer.yandex.ru)
Received: (from tejblum@localhost)
	by developer.yandex.ru (8.13.3/8.13.1/Submit) id j2KA6q4b047338;
	Sun, 20 Mar 2005 13:06:52 +0300 (MSK)
	(envelope-from tejblum)
Message-Id: <200503201006.j2KA6q4b047338@developer.yandex.ru>
Date: Sun, 20 Mar 2005 13:06:52 +0300 (MSK)
From: Dmitrij Tejblum <tejblum@yandex-team.ru>
Reply-To: Dmitrij Tejblum <tejblum@yandex-team.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: realloc() copies data even when the size of allocated memory decrease
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         79048
>Category:       kern
>Synopsis:       [libc] realloc() copies data even when the size of allocated memory decrease
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 20 10:10:07 GMT 2005
>Closed-Date:    Sat Sep 15 23:54:47 UTC 2012
>Last-Modified:  Sat Sep 15 23:54:47 UTC 2012
>Originator:     Dmitrij Tejblum
>Release:        FreeBSD 5.4-PRERELEASE i386
>Organization:
>Environment:

>Description:

If a program malloc() a large piece of memory (e.g. p = malloc(1024000)) and
then realloc() it to a smaller size (e.g. realloc(p, 512000)) realloc() 
allocate new piece of memory and copy the old piece to new one. The unnecessary
reallocation also sometimes lead to allocation failures. But it is easy to
mark the tail of old memory as free.

>How-To-Repeat:

>Fix:

--- malloc.c	Sun Mar 20 03:11:19 2005
+++ malloc.c	Sun Mar 20 03:13:15 2005
@@ -286,6 +286,7 @@ static int extend_pgdir(u_long index);
 static void *imalloc(size_t size);
 static void ifree(void *ptr);
 static void *irealloc(void *ptr, size_t size);
+static __inline void free_pages(void *ptr, u_long index, struct pginfo const *info);
 
 static void
 wrtmessage(const char *p1, const char *p2, const char *p3, const char *p4)
@@ -787,10 +788,15 @@ irealloc(void *ptr, size_t size)
 
         if (!malloc_realloc && 			/* Unless we have to, */
 	  size <= osize && 			/* .. or are too small, */
-	  size > (osize - malloc_pagesize)) {	/* .. or can free a page, */
+	  size > malloc_maxsize) {		/* .. or want to allocate a chunk */
 	    if (malloc_junk)
 		memset((u_char *)ptr + size, SOME_JUNK, osize-size);
-	    return (ptr);			/* ..don't do anything else. */
+	    if (size <= (osize - malloc_pagesize)) {    /* have something to free */
+		size_t nsize = pageround(size);
+		void *fptr = (u_char *)ptr + nsize;
+		free_pages(fptr, ptr2index(fptr), MALLOC_FIRST);
+	    }
+	    return (ptr);
 	}
 
     } else if (*mp >= MALLOC_MAGIC) {		/* Chunk allocation */


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->phk 
Responsible-Changed-By: glebius 
Responsible-Changed-When: Wed Mar 23 12:37:45 GMT 2005 
Responsible-Changed-Why:  
Poul-Henning is the right person to handle this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79048 

From: "Poul-Henning Kamp" <phk@haven.freebsd.dk>
To: Gleb Smirnoff <glebius@FreeBSD.org>
Cc: freebsd-bugs@FreeBSD.org
Subject: Re: bin/79048: realloc() copies data even when the size of
	allocated memory decrease 
Date: Thu, 24 Mar 2005 20:20:27 +0100

 In message <200503231238.j2NCccug050911@freefall.freebsd.org>, Gleb Smirnoff wr
 ites:
 >Synopsis: realloc() copies data even when the size of allocated memory decrease
 >
 >Responsible-Changed-From-To: freebsd-bugs->phk
 >Responsible-Changed-By: glebius
 >Responsible-Changed-When: Wed Mar 23 12:37:45 GMT 2005
 >Responsible-Changed-Why: 
 >Poul-Henning is the right person to handle this.
 
 
 This is by design, read the paper.
 
 -- 
 Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
 phk@FreeBSD.ORG         | TCP/IP since RFC 956
 FreeBSD committer       | BSD since 4.3-tahoe    
 Never attribute to malice what can adequately be explained by incompetence.
 _______________________________________________
 freebsd-bugs@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/freebsd-bugs
 To unsubscribe, send any mail to "freebsd-bugs-unsubscribe@freebsd.org"
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Fri Oct 28 06:15:03 GMT 2005 
State-Changed-Why:  
phk responded long ago that this was by design and cited a paper that 
discusses it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79048 

From: Dmitrij Tejblum <tejblum@yandex-team.ru>
To: Mark Linimon <linimon@FreeBSD.org>, phk@FreeBSD.org,
        FreeBSD-gnats-submit@freebsd.org
Cc: frebsd-bugs@freebsd.org
Subject: Re: kern/79048: [libc] realloc() copies data even when the size of
 allocated memory decrease
Date: Fri, 28 Oct 2005 13:47:33 +0400

 Mark Linimon wrote:
 
 >Synopsis: [libc] realloc() copies data even when the size of allocated memory decrease
 >
 >State-Changed-From-To: open->closed
 >State-Changed-By: linimon
 >State-Changed-When: Fri Oct 28 06:15:03 GMT 2005
 >State-Changed-Why: 
 >phk responded long ago that this was by design and cited a paper that
 >discusses it.
 >
 >http://www.freebsd.org/cgi/query-pr.cgi?pr=79048
 >  
 >
 Ah, I did not seen the response, it was not CC'ed to me since GNATS does 
 not send Responsible-Changed notifications to the originator.
 
 Anyway, there is nothing in the paper that suggest that this is by 
 design. The only quotes concerning realloc(3) are:
 
 
  > A third kind of request, realloc (3), will resize a chunk,
  > trying to avoid copying the contents if possible.
  > It is seldom used, and has only had a significant
  > impact on performance in a few special situations.
  > The typical pattern of use is to malloc (3) a chunk of
  > the maximum size needed, read in the data and adjust
  > the size of the chunk to match the size of the data read
  > using realloc (3), or alternatively, to allocate with
  > malloc (3) a chunk which can handle a large fraction
  > of the requests, and if this proves insufficient, reallocating
  > with realloc (3), possibly several times, until the
  > necessary amount of memory has been obtained.
 
 Yes, and in several threads PHK says that the second way should be 
 avoided due to its inefficiency, but the first way should be used 
 instead. The problem is that the first way is also inefficient, since it 
 is often does not avoid copying even when it is possible.
 
  > [Talking about MALLOC_OPTIONS]
  > Realloc [option]
  > Always do a free and malloc when
  > realloc (3) is called. For programs doing garbage
  > collection using realloc (3), this make the heap collapse
  > faster since malloc will reallocate from the
  > lowest available address. The default is to leave
  > things alone if the size of the allocation is still in
  > the same size-bracket
 
 ... and the patch that I proposed does not break the behaviour of the 
 "always realloc" malloc option, it only extend the possibility to apply 
 the default behaviour. (BTW, always doing realloc can noticeable often 
 increase the heap instead of collapse it -- it is quite possible that 
 there is no room for new piece of memory in the allocated heap.)
 
 So, please reopen the PR, fix the issue, and only then close :-)
 
 
 
State-Changed-From-To: closed->suspended 
State-Changed-By: linimon 
State-Changed-When: Fri Oct 28 21:18:00 GMT 2005 
State-Changed-Why:  
Re-opened at submitter's insistence.  Since the author of the code in 
question disagrees with this fix, reassign it to freebsd-bugs.  Mark it 
suspended in case someone else someday wants to look into this. 


Responsible-Changed-From-To: phk->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Oct 28 21:18:00 GMT 2005 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=79048 
State-Changed-From-To: suspended->closed 
State-Changed-By: eadler 
State-Changed-When: Sat Sep 15 23:54:44 UTC 2012 
State-Changed-Why:  
we use jemalloc which may not behave the same way. in either case this 
was not originally a bug. close the pr 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79048 
>Unformatted:
