From sa@gw2.svzserv.kemerovo.su  Fri Dec 27 00:55:15 2002
Return-Path: <sa@gw2.svzserv.kemerovo.su>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 89C3D37B401
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 27 Dec 2002 00:55:15 -0800 (PST)
Received: from gw2.svzserv.kemerovo.su (gw2.svzserv.kemerovo.su [213.184.65.69])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 074CC43EB2
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 27 Dec 2002 00:55:14 -0800 (PST)
	(envelope-from sa@gw2.svzserv.kemerovo.su)
Received: from gw2.svzserv.kemerovo.su (localhost [127.0.0.1])
	by gw2.svzserv.kemerovo.su (8.12.6/8.12.6) with ESMTP id gBR8tAh9075104
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 27 Dec 2002 15:55:10 +0700 (KRAT)
	(envelope-from sa@gw2.svzserv.kemerovo.su)
Received: (from root@localhost)
	by gw2.svzserv.kemerovo.su (8.12.6/8.12.6/Submit) id gBR8t8Ii075103;
	Fri, 27 Dec 2002 15:55:08 +0700 (KRAT)
Message-Id: <200212270855.gBR8t8Ii075103@gw2.svzserv.kemerovo.su>
Date: Fri, 27 Dec 2002 15:55:08 +0700 (KRAT)
From: Eugene Grosbein <eugen@grosbein.pp.ru>
Reply-To: Eugene Grosbein <eugen@grosbein.pp.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: ipfw pipe show fails with lots of queues
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         46557
>Category:       kern
>Synopsis:       ipfw pipe show fails with lots of queues
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    pjd
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Dec 27 01:00:08 PST 2002
>Closed-Date:    Sat Sep 16 11:50:32 GMT 2006
>Last-Modified:  Sat Sep 16 11:50:32 GMT 2006
>Originator:     Eugene Grosbein
>Release:        FreeBSD 4.7-STABLE i386
>Organization:
Svyaz Service JSC
>Environment:
System: FreeBSD gw2.svzserv.kemerovo.su 4.7-STABLE FreeBSD 4.7-STABLE #7: Thu Dec 26 14:00:53 KRAT 2002 sa@gw2.svzserv.kemerovo.su:/usr/obj/usr/src/sys/GW2 i386
	It is a router with 32Mb of RAM, 4x100Mbit NICs, ipfw1 and dummynet.
	
>Description:
	ipfw pipe show says:
	ipfw: getsockopt(IP_DUMMYNET_GET): No buffer space available

	I use dummynet pipes and queues for per-flow queueing, now I have
	192 queues (102 are active currently). Each queue is connected
	to a pipe, each pipe has 'mask all' attribute.

>How-To-Repeat:
	Build a kernel with options DUMMYNET.
	Get ipfw rules from PR kern/44045, make high load of traffic
	wait an hour, try to run ipfw pipe show.

>Fix:

	Unknown for me.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->ipfw 
Responsible-Changed-By: johan 
Responsible-Changed-When: Mon Feb 3 13:07:27 PST 2003 
Responsible-Changed-Why:  
Over to ipfw maintainer group. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=46557 

From: Pawel Malachowski <pawmal-posting@freebsd.lublin.pl>
To: freebsd-gnats-submit@FreeBSD.org, eugen@grosbein.pp.ru
Cc: freebsd-bugs@freebsd.org
Subject: Re: kern/46557: ipfw pipe show fails with lots of queues
Date: Thu, 12 Aug 2004 16:10:20 +0200

 Hello,
 
 	I've just hit this problem on my 4.10-STABLE dummynet shaper.
 Very bad, since `ipfw pipe show N' + net.inet.ip.dummynet.expire=0 is
 quite useful for grabbing some per-user statistics. :/
 
 
 Here are more details what is going on. I hope someone will look at this and
 explain why malloc() can fail here and what can be done to prevent this.
 
 
 % ipfw pipe show
 ipfw: getsockopt(IP_DUMMYNET_GET): No buffer space available
 
 This command invokes getsockopt() trying to fetch all pipes data from kernel
 to userland.
 
 Kernel part of this is being done in dummynet_get(), which tries to allocate
 buf big (previously computed size) enough for all data:
 
     buf = malloc(size, M_TEMP, M_NOWAIT);
     if (buf == 0) {
 	splx(s);
 	return ENOBUFS ;
     }
 
 This malloc() call fails sometimes on loaded system (10k-70k of dynamic
 pipes + make buildworld;)) causing `ipfw pipe show' command failure.
 
 I've registered temporary MALLOC_DEFINE M_YOYO and changed this malloc/free
 from M_TEMP to M_YOYO, ;) so I can easily track this in `vmstat -m' output:
 
 1. Quite early (no problems):
         Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
   IpFw/IpAcct   113    14K     14K 42107K      113    0     0  64,128,256
          yoyo     0     0K    120K 42107K        1    0     0  128K
      dummynet  1612   311K    313K 42107K   139366    0     0  16,128,256,4K
 Memory Totals:  In Use       Free    Requests
                  3921K        32K      173824
 
 2. After some time malloc() in dummynet_get() fails and:
         Type  InUse MemUse HighUse  Limit Requests Limit Limit Size(s)
   IpFw/IpAcct   113    14K     14K 42107K      113    0     0  64,128,256
          yoyo     0     0K   2828K 42107K    12913    0     0  128K,512K
      dummynet 21432  2790K   3354K 42107K  9804136    0     0  16,128,256,512,1K,4K
 Memory Totals:  In Use       Free    Requests
                  9932K      3029K    14216860
 
 In /sbin/ipfw2, list():
 	(do_cmd(ocmd, data, (uintptr_t)&nbytes) < 0)
 nbytes were 2188288  // In loop, I +1024 numbytes instead of *2+200, ignore this
 
 In ip_dummynet.c, dummynet_get():
 	buf = malloc(size, M_YOYO, M_NOWAIT); // note: M_TEMP => M_YOYO
 size was 2390416 and malloc() failed.
 
 
 -- 
 Pawe Maachowski

From: Pawel Malachowski <pawmal-posting@freebsd.lublin.pl>
To: Pawel Malachowski <pawmal-posting@freebsd.lublin.pl>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: kern/46557: ipfw pipe show fails with lots of queues
Date: Fri, 13 Aug 2004 10:13:35 +0200

 On Thu, Aug 12, 2004 at 07:30:25PM +0000, Pawel Malachowski wrote:
 
 >  Here are more details what is going on. I hope someone will look at this and
 >  explain why malloc() can fail here and what can be done to prevent this.
 
 Well, I will explain myself. First, a patch:
 
 *** ip_dummynet.c.orig  Fri Aug 13 09:51:54 2004
 --- ip_dummynet.c       Fri Aug 13 09:52:23 2004
 ***************
 *** 1833,1839 ****
       for (set = all_flow_sets ; set ; set = set->next )
         size += sizeof ( *set ) +
             set->rq_elements * sizeof(struct dn_flow_queue);
 !     buf = malloc(size, M_TEMP, M_NOWAIT);
       if (buf == 0) {
         splx(s);
         return ENOBUFS ;
 --- 1833,1839 ----
       for (set = all_flow_sets ; set ; set = set->next )
         size += sizeof ( *set ) +
             set->rq_elements * sizeof(struct dn_flow_queue);
 !     buf = malloc(size, M_TEMP, M_WAITOK);
       if (buf == 0) {
         splx(s);
         return ENOBUFS ;
 
 Since buf can be very big (on my system, up to 3MB), malloc typically
 consumes only 128k-512k buckets (as checked in vmstat -m). On heavy
 loaded system with huge number of pipes, there is a risk that first-fit
 strategy won't find memory because of fragmentation.
 We will change M_NOWAIT to M_WAITOK flag so malloc() will always succeed;
 dummynet_get() is called only when superuser runs `ipfw pipe show', so
 WAITOK is not a problem for us.
 
 
 -- 
 Pawe Maachowski
State-Changed-From-To: open->feedback 
State-Changed-By: pjd 
State-Changed-When: Mon Aug 23 21:50:27 GMT 2004 
State-Changed-Why:  


http://www.freebsd.org/cgi/query-pr.cgi?pr=46557 
Responsible-Changed-From-To: ipfw->pjd 
Responsible-Changed-By: pjd 
Responsible-Changed-When: Mon Aug 23 21:53:44 GMT 2004 
Responsible-Changed-Why:  
I'll take this one. 

Here is prosposed patch against HEAD: 

Index: ip_dummynet.c 
=================================================================== 
RCS file: /private/FreeBSD/src/sys/netinet/ip_dummynet.c,v 
retrieving revision 1.82 
diff -u -p -r1.82 ip_dummynet.c 
--- ip_dummynet.c	15 Jul 2004 08:26:07 -0000	1.82 
+++ ip_dummynet.c	23 Aug 2004 21:41:06 -0000 
@@ -1895,17 +1895,14 @@ dn_copy_set(struct dn_flow_set *set, cha 
return (char *)qp ; 
} 

-static int 
-dummynet_get(struct sockopt *sopt) 
+static size_t 
+dn_calc_size(void) 
{ 
-    char *buf, *bp ; /* bp is the "copy-pointer" */ 
-    size_t size ; 
struct dn_flow_set *set ; 
struct dn_pipe *p ; 
-    int error=0 ; 
+    size_t size ; 

-    /* XXX lock held too long */ 
-    DUMMYNET_LOCK(); 
+    DUMMYNET_LOCK_ASSERT(); 
/* 
* compute size of data structures: list of pipes and flow_sets. 
*/ 
@@ -1915,8 +1912,35 @@ dummynet_get(struct sockopt *sopt) 
for (set = all_flow_sets ; set ; set = set->next ) 
size += sizeof ( *set ) + 
set->rq_elements * sizeof(struct dn_flow_queue); 
-    buf = malloc(size, M_TEMP, M_NOWAIT); 
-    if (buf == 0) { 
+    return size ; 
+} 
+ 
+static int 
+dummynet_get(struct sockopt *sopt) 
+{ 
+    char *buf, *bp ; /* bp is the "copy-pointer" */ 
+    size_t size ; 
+    struct dn_flow_set *set ; 
+    struct dn_pipe *p ; 
+    int error=0, i ; 
+ 
+    /* XXX lock held too long */ 
+    DUMMYNET_LOCK(); 
+    /* 
+     * XXX: Ugly, but we need to allocate memory with M_WAITOK flag and we 
+     *      cannot use this flag while holding a mutex. 
+     */ 
+    for (i = 0; i < 10; i++) { 
+	size = dn_calc_size(); 
+	DUMMYNET_UNLOCK(); 
+	buf = malloc(size, M_TEMP, M_WAITOK); 
+	DUMMYNET_LOCK(); 
+	if (size == dn_calc_size()) 
+		break; 
+	free(buf, M_TEMP); 
+	buf = NULL; 
+    } 
+    if (buf == NULL) { 
DUMMYNET_UNLOCK(); 
return ENOBUFS ; 
} 

http://www.freebsd.org/cgi/query-pr.cgi?pr=46557 
State-Changed-From-To: feedback->patched 
State-Changed-By: pjd 
State-Changed-When: Wed Aug 25 09:22:07 GMT 2004 
State-Changed-Why:  
Fix committed to HEAD, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=46557 

From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/46557: ipfw pipe show fails with lots of queues
Date: Sat, 7 Jan 2006 16:52:16 +0700

 Hi!
 
 The fix is present in both of RELENG_5 and RELENG_6 branches by now.
 If no one is going to prepare and commit a fix for RELENG_4,
 this PR should be closed.
 
 Eugene Grosbein
State-Changed-From-To: patched->closed 
State-Changed-By: pjd 
State-Changed-When: Sat Sep 16 11:49:43 UTC 2006 
State-Changed-Why:  
The fix isn't going to be MFC to RELENG_4. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=46557 
>Unformatted:
