From nobody@FreeBSD.org  Sat Nov  3 17:11:23 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id F174843D
	for <freebsd-gnats-submit@FreeBSD.org>; Sat,  3 Nov 2012 17:11:23 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id BE2E08FC0A
	for <freebsd-gnats-submit@FreeBSD.org>; Sat,  3 Nov 2012 17:11:23 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id qA3HBNoI050475
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 3 Nov 2012 17:11:23 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id qA3HBNTY050474;
	Sat, 3 Nov 2012 17:11:23 GMT
	(envelope-from nobody)
Message-Id: <201211031711.qA3HBNTY050474@red.freebsd.org>
Date: Sat, 3 Nov 2012 17:11:23 GMT
From: Ian Lepore <freebsd@damnhippie.dyndns.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [patch] watchdogd wires over 10MB of memory after r239769
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         173332
>Category:       bin
>Synopsis:       [patch] watchdogd wires over 10MB of memory after r239769
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    ian
>State:          patched
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Nov 03 17:20:00 UTC 2012
>Closed-Date:    
>Last-Modified:  Sun Jan 27 22:42:48 UTC 2013
>Originator:     Ian Lepore
>Release:        FreeBSD 10.0-CURRENT @ r241077
>Organization:
>Environment:
FreeBSD tflex 10.0-CURRENT FreeBSD 10.0-CURRENT #22: Tue Oct 16 15:10:51 UTC 2012     root@revolution.hippie.lan:/usr/obj/arm.arm/usr/src/sys/TFLEX  arm

>Description:
watchdogd was changed in r239769 to use mlockall(2) to wire down the memory used by the app so that it can pat the watchdog without any delays caused by waiting for its pages to be swapped back in.  This has caused its resident size to increase from about 840KB to about 10MB, which is a serious problem on small-memory embedded systems.

Most of the memory increase seems to be related to jemalloc allocating vmspace in chunks of 8MB at a time.  Normally this is innocuous; even if the app only uses a few bytes of allocated memory there's no large downside to having over-allocated some vmspace.  However, when mlockall(MCL_FUTURE) is in effect, it is no longer a vmspace allocation, physical pages for the entire 8MB region are prefaulted and wired.

Another megabyte or so of the growth seems to be related to wiring in pages (whether needed or not) from libc and libm.


>How-To-Repeat:

>Fix:
The attached patch reduces the impact of the mlockall() change on small-memory embedded systems.  Such systems are often built with "option NO_SWAPPING" which eliminates most of the reason the mlockall() change was put in originally.

Note that this is not really a full fix for the problem, it just reduces the  impact on the most resource-limited systems.


Patch attached with submission follows:

diff -r 5f2e07d3ae42 usr.sbin/watchdogd/watchdogd.c
--- usr.sbin/watchdogd/watchdogd.c	Thu Oct 04 20:42:12 2012 -0600
+++ usr.sbin/watchdogd/watchdogd.c	Sat Nov 03 11:04:31 2012 -0600
@@ -35,6 +35,7 @@
 #include <sys/param.h>
 #include <sys/rtprio.h>
 #include <sys/stat.h>
+#include <sys/sysctl.h>
 #include <sys/time.h>
 #include <sys/watchdog.h>
 
@@ -78,6 +79,7 @@ main(int argc, char *argv[])
 	struct rtprio rtp;
 	struct pidfh *pfh;
 	pid_t otherpid;
+	int len, swap_enabled;
 
 	if (getuid() != 0)
 		errx(EX_SOFTWARE, "not super user");
@@ -118,8 +120,13 @@ main(int argc, char *argv[])
 		pidfile_write(pfh);
 		if (madvise(0, 0, MADV_PROTECT) != 0)
 			warn("madvise failed");
-		if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0)
-			warn("mlockall failed");
+		len = sizeof(swap_enabled);
+		if (sysctlbyname("vm.swap_enabled", &swap_enabled, &len, NULL,
+		    0) == -1)
+			swap_enabled = 1; /* Assume enabled */
+		if (swap_enabled)
+			if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0)
+				warn("mlockall failed");
 
 		watchdog_loop();
 


>Release-Note:
>Audit-Trail:

From: Ian Lepore <freebsd@damnhippie.dyndns.org>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc:  
Subject: Re: bin/173332: [patch] watchdogd wires over 10MB of memory after
 r239769
Date: Sat, 03 Nov 2012 11:36:18 -0600

 On Sat, 2012-11-03 at 17:20 +0000, FreeBSD-gnats-submit@FreeBSD.org
 wrote:
 > Thank you very much for your problem report.
 > It has the internal identification `bin/173332'.
 > The individual assigned to look at your
 > report is: freebsd-bugs. 
 > 
 > You can access the state of your problem report at any time
 > via this link:
 > 
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=173332
 > 
 > >Category:       bin
 > >Responsible:    freebsd-bugs
 > >Synopsis:       [patch] watchdogd wires over 10MB of memory after r239769
 > >Arrival-Date:   Sat Nov 03 17:20:00 UTC 2012
 
 Comparison of ps -u and procstat -v output with and without the
 mlockall(2) call in watchdogd...
 
 With mlockall:
 
 USER  PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
 root  1299  0.0 15.5 10236 10164  -  Ss    2:57PM   0:00.19 /usr/sbin/watchdogd -t 100
 
 tflex# procstat -v `pgrep watchdogd`
   PID      START        END PRT  RES PRES REF SHD   FL TP PATH
  1299     0x8000     0xa000 r-x    2    0   1   0 CN-- vn /usr/sbin/watchdogd
  1299    0x11000    0x12000 rw-    1    0   1   0 C--- df 
  1299 0x20011000 0x20027000 r-x   22    0  53  26 CN-- vn /libexec/ld-elf.so.1
  1299 0x2002e000 0x2002f000 rw-    1    0   1   0 C--- vn /libexec/ld-elf.so.1
  1299 0x2002f000 0x20038000 rw-    9    0   1   0 C--- df 
  1299 0x20038000 0x20060000 r-x   40    0  20  10 CN-- vn /lib/libm.so.5
  1299 0x20060000 0x20067000 ---    0    0   1   0 CN-- df 
  1299 0x20067000 0x20068000 rw-    1    0   1   0 C--- vn /lib/libm.so.5
  1299 0x20068000 0x20078000 r-x   16    0  19   9 CN-- vn /lib/libutil.so.9
  1299 0x20078000 0x2007f000 ---    0    0   1   0 CN-- df 
  1299 0x2007f000 0x20080000 rw-    1    0   1   0 C--- vn /lib/libutil.so.9
  1299 0x20080000 0x20082000 rw-    2    0   1   0 ---- df 
  1299 0x20082000 0x201a8000 r-x  294    0  53  26 CN-- vn /lib/libc.so.7
  1299 0x201a8000 0x201af000 ---    0    0   1   0 CN-- df 
  1299 0x201af000 0x201b6000 rw-    7    0   1   0 C--- vn /lib/libc.so.7
  1299 0x201b6000 0x201f4000 rw-   62    0   1   0 C--- df 
  1299 0x20400000 0x20c00000 rw- 2048    0   1   0 C--- df 
  1299 0xbffe0000 0xc0000000 rwx   32    0   1   0 C--D df 
 
 
 Without mlockall:
 
 USER  PID %CPU %MEM   VSZ   RSS TT  STAT STARTED    TIME COMMAND
 root  1276  0.0  1.3 10236   840  -  Ss    2:56PM   0:00.02 /usr/sbin/watchdogd -t 100
 
 tflex# procstat -v `pgrep watchdogd`
   PID      START        END PRT  RES PRES REF SHD   FL TP PATH
  1276     0x8000     0xa000 r-x    2    0   1   0 CN-- vn /usr/sbin/watchdogd
  1276    0x11000    0x12000 rw-    1    0   1   0 C--- df 
  1276 0x20011000 0x20027000 r-x   22    0  53  26 CN-- vn /libexec/ld-elf.so.1
  1276 0x2002e000 0x2002f000 rw-    0    0   1   0 CN-- vn /libexec/ld-elf.so.1
  1276 0x2002f000 0x20038000 rw-    8    0   1   0 C--- df 
  1276 0x20038000 0x20060000 r-x    5    0  20  10 CN-- vn /lib/libm.so.5
  1276 0x20060000 0x20067000 ---    0    0   1   0 CN-- df 
  1276 0x20067000 0x20068000 rw-    0    0   1   0 CN-- vn /lib/libm.so.5
  1276 0x20068000 0x20078000 r-x   16    0  19   9 CN-- vn /lib/libutil.so.9
  1276 0x20078000 0x2007f000 ---    0    0   1   0 CN-- df 
  1276 0x2007f000 0x20080000 rw-    1    0   1   0 C--- vn /lib/libutil.so.9
  1276 0x20080000 0x20082000 rw-    0    0   0   0 ---- -- 
  1276 0x20082000 0x201a8000 r-x  135    0  53  26 CN-- vn /lib/libc.so.7
  1276 0x201a8000 0x201af000 ---    0    0   1   0 CN-- df 
  1276 0x201af000 0x201b6000 rw-    7    0   1   0 C--- vn /lib/libc.so.7
  1276 0x201b6000 0x201f4000 rw-    5    0   1   0 C--- df 
  1276 0x20400000 0x20c00000 rw-    2    0   1   0 CN-- df 
  1276 0xbffe0000 0xc0000000 rwx    3    0   1   0 C--D df 
 
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: bin/173332: commit references a PR
Date: Sat, 26 Jan 2013 21:29:54 +0000 (UTC)

 Author: ian
 Date: Sat Jan 26 21:29:45 2013
 New Revision: 245949
 URL: http://svnweb.freebsd.org/changeset/base/245949
 
 Log:
   Reduce watchdogd's memory footprint when running daemonized.
   
   This uses the recently-added jemalloc(3) feature of setting the lg_chunk
   tuning option to zero to request that memory be allocated in the smallest
   chunks possible.  Without this option, the default is to initally map 8MB,
   and then the mlockall() call wires that entire allocation even though the
   program only uses a few Kbytes of it at runtime.
   
   PR:		bin/173332
   Approved by:	cognet (mentor)
 
 Modified:
   head/usr.sbin/watchdogd/watchdogd.c
 
 Modified: head/usr.sbin/watchdogd/watchdogd.c
 ==============================================================================
 --- head/usr.sbin/watchdogd/watchdogd.c	Sat Jan 26 20:16:58 2013	(r245948)
 +++ head/usr.sbin/watchdogd/watchdogd.c	Sat Jan 26 21:29:45 2013	(r245949)
 @@ -71,6 +71,14 @@ static int nap = 1;
  static char *test_cmd = NULL;
  
  /*
 + * Ask malloc() to map minimum-sized chunks of virtual address space at a time,
 + * so that mlockall() won't needlessly wire megabytes of unused memory into the
 + * process.  This must be done using the malloc_conf string so that it gets set
 + * up before the first allocation, which happens before entry to main().
 + */
 +const char * malloc_conf = "lg_chunk:0";
 +
 +/*
   * Periodically pat the watchdog, preventing it from firing.
   */
  int
 @@ -188,7 +196,7 @@ watchdog_loop(void)
  			if (watchdog_onoff(0) == 0) {
  				end_program = 2;
  			} else {
 -				warnx("Could not stop the watchdog, not exiting");
 +				warnx("Could not stop the watchdog, not exitting");
  				end_program = 0;
  			}
  		}
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: ian 
State-Changed-When: Sun Jan 27 18:36:49 UTC 2013 
State-Changed-Why:  
Fixed in -current; MFC after the corresponding jemalloc changes are MFC'd. 



Responsible-Changed-From-To: freebsd-bugs->ian 
Responsible-Changed-By: ian 
Responsible-Changed-When: Sun Jan 27 18:36:49 UTC 2013 
Responsible-Changed-Why:  
Fixed in -current; MFC after corresponding jemalloc changes are MFC'd. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=173332 
>Unformatted:
