From nobody@FreeBSD.org  Mon Jan 18 01:18:53 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id ACF191065672
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 18 Jan 2010 01:18:53 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 9B9DE8FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 18 Jan 2010 01:18:53 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o0I1IrEU058947
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 18 Jan 2010 01:18:53 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o0I1IrN5058946;
	Mon, 18 Jan 2010 01:18:53 GMT
	(envelope-from nobody)
Message-Id: <201001180118.o0I1IrN5058946@www.freebsd.org>
Date: Mon, 18 Jan 2010 01:18:53 GMT
From: vermaden <vermaden@interia.pl>
To: freebsd-gnats-submit@FreeBSD.org
Subject: memory allocation/release problem?
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         142921
>Category:       kern
>Synopsis:       [kernel] memory allocation/release problem?
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    brucec
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jan 18 01:20:06 UTC 2010
>Closed-Date:    Mon Mar 01 20:23:07 UTC 2010
>Last-Modified:  Tue Mar 02 00:43:25 UTC 2010
>Originator:     vermaden
>Release:        8.0-RELEASE
>Organization:
>Environment:
FreeBSD savio 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:48:17 UTC 2009     root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
>Description:
I have a script [1] that generates links/bookmarks for me based upon the configuration file [2], it takes about 2 minutes on CentOS 5.4 LiveCD under VirtualBox with 256MB to complete the task, while it never finishes the work under FreeBSD because lack of free memory (3GB system). This problem is not sh(1) related, cause I tried to set zsh/bash/dash as executive shell (#! /usr/local/bin/zsh and so) to compare results, all of them scream about lack of free memory.

Below are results from CentOS 5.4 LiveCD and FreeBSD.

CENTOS
------------------------------------------------------------------------------
[centos@livecd ~]$ time ./generate_links_html.sh SITES > links.htm
real    1m43.700s
user    0m39.917s
sys     0m57.186s

[centos@livecd ~]$ ls -l links.htm
-rw-rw-r-- 1 centos centos 83786 Jan 18 01:32 links.htm

[centos@livecd ~]$ free -m
             total       used       free     shared    buffers     cached
Mem:           249        245          4          0         12        102
-/+ buffers/cache:        130        118
Swap:            0          0          0

[centos@livecd ~]$ uname -a
Linux livecd.localdomain 2.6.18-164.el5 #1 SMP Thu Sep 3 03:33:56 EDT 2009 i686 i686 i386 GNU/Linux
------------------------------------------------------------------------------

FREEBSD
------------------------------------------------------------------------------
vermaden % time generate_links_html.sh.bck SITES > links.htm
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
/home/vermaden/scripts/generate_links_html.sh: Out of space
^C
generate_links_html.sh SITES > links.htm  162.88s user 99.64s system 21% cpu 20:29.13 total

vermaden % top(1) memory output (after ^C):
Mem: 319M Active, 151M Inact, 722M Wired, 2008K Cache, 105M Buf, 1792M Free
Swap: 2048M Total, 2048M Free

vermaden % uname -a    
FreeBSD savio 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:48:17 UTC 2009     root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
------------------------------------------------------------------------------

[1] http://strony.toya.net.pl/~vermaden/tmp/generate_links_html.sh
[2] http://strony.toya.net.pl/~vermaden/tmp/SITES

Regards,
vermaden
>How-To-Repeat:
prompt% wget http://strony.toya.net.pl/~vermaden/tmp/generate_links_html.sh
prompt% wget http://strony.toya.net.pl/~vermaden/tmp/SITES
prompt% chmod +x generate_links_html.sh
prompt% ./generate_links_html.sh SITES > links.htm
prompt% ${BROWSER} links.htm
>Fix:
>Release-Note:
>Audit-Trail:

Date: 18 Jan 2010 08:21:35 +0100
From: vermaden <vermaden@interia.pl>
Sender: vermaden@interia.pl
Subject: =?UTF-8?q?Re:_kern/142921:_[kernel]_memory_allocation/release_problem=3F?=

 I left it for longer time, and the job completes after about 36 minutes (if=
  that helps anything):
 
 prompt% time generate_links_html.sh SITES > links.htm
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 /home/vermaden/scripts/generate_links_html.sh: Out of space
 generate_links_html.sh SITES > links.htm  303.40s user 196.26s system 23% c=
 pu 36:09.72 total
 
 prompt% ls -l links.htm
 -rw-r--r--  1 vermaden  vermaden  83786 Jan 18 03:01 links.htm
 
 The output links.htm has the same size/content as the CentOS generated vers=
 ion.
 
 Regards,
 vermaden
 
 ----------------------------------------------------------------------
 Rezerwując wczesniej kupisz bilety lotnicze taniej!
 Sprawdz >>> http://link.interia.pl/f256c
Responsible-Changed-From-To: freebsd-bugs->brucec  
Responsible-Changed-By: brucec 
Responsible-Changed-When: Sun Feb 28 21:42:51 UTC 2010 
Responsible-Changed-Why:  
Take. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=142921 

From: Bruce Cran <bruce@cran.org.uk>
To: bug-followup@freebsd.org,
 vermaden@interia.pl
Cc:  
Subject: Re: kern/142921: [kernel] memory allocation/release problem?
Date: Sun, 28 Feb 2010 21:39:58 +0000

 A simplification of the script which still produces the memory leak:
 
 #!/bin/sh
 
 __indent() {
  for I in $( jot ${1} )
   do
   done
 }
 
 __generate() {
  while (true)
  do
    __indent 0
  done
 }
 
 __generate
 
 
 
 -- 
 Bruce
State-Changed-From-To: open->closed  
State-Changed-By: brucec 
State-Changed-When: Mon Mar 1 20:20:28 UTC 2010 
State-Changed-Why:  
The problem is a difference in behaviour between seq and dot. 'seq 0' exits 
immediately, but 'jot 0' generates an infinite list of numbers. bash, sh etc. 
keep reallocating their buffers to accomodate the bigger and bigger list until  
they run out of memory. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=142921 

Date: 01 Mar 2010 21:43:21 +0100
From: vermaden <vermaden@interia.pl>
Sender: vermaden@interia.pl
To: bug-followup@FreeBSD.org
Subject: Re: kern/142921: [kernel] memory allocation/release problem?

 Thanks for help and information about seq/jot difference, script modified:
 
  __indent() {
 +  [ ${1} -eq 0 ] && return
    for I in $( jot ${1} 2> /dev/null )
    do
      echo -n "  "
    done
    }
 
 Regards,
 vermaden
 
 ----------------------------------------------------------------------
 Sprawdz warunki na drogach!
 Kliknij >>> http://link.interia.pl/f259f
>Unformatted:
