From antoine@massena-4-82-67-196-50.fbx.proxad.net  Thu Dec 30 23:43:27 2004
Return-Path: <antoine@massena-4-82-67-196-50.fbx.proxad.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id E3C1116A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 30 Dec 2004 23:43:27 +0000 (GMT)
Received: from barton.dreadbsd.org (massena-4-82-67-196-50.fbx.proxad.net [82.67.196.50])
	by mx1.FreeBSD.org (Postfix) with ESMTP id D012543D41
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 30 Dec 2004 23:43:26 +0000 (GMT)
	(envelope-from antoine@massena-4-82-67-196-50.fbx.proxad.net)
Received: from barton.dreadbsd.org (localhost [127.0.0.1])
	by barton.dreadbsd.org (8.13.1/8.13.1) with ESMTP id iBUNhOpD001308
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 31 Dec 2004 00:43:25 +0100 (CET)
	(envelope-from antoine@massena-4-82-67-196-50.fbx.proxad.net)
Received: (from antoine@localhost)
	by barton.dreadbsd.org (8.13.1/8.13.1/Submit) id iBUNhOAD001307;
	Fri, 31 Dec 2004 00:43:24 +0100 (CET)
	(envelope-from antoine)
Message-Id: <200412302343.iBUNhOAD001307@barton.dreadbsd.org>
Date: Fri, 31 Dec 2004 00:43:24 +0100 (CET)
From: Antoine Brodin <antoine.brodin@laposte.net>
Reply-To: Antoine Brodin <antoine.brodin@laposte.net>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: [patch] sk(4): plug leaks and fix acquiring duplicate lock of same type: "network driver"
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         75677
>Category:       kern
>Synopsis:       [patch] sk(4): plug leaks and fix acquiring duplicate lock of same type: "network driver"
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    bz
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Dec 30 23:50:06 GMT 2004
>Closed-Date:    Thu Mar 31 09:53:54 GMT 2005
>Last-Modified:  Thu Mar 31 09:53:54 GMT 2005
>Originator:     Antoine Brodin
>Release:        FreeBSD 6.0-CURRENT i386
>Organization:
none
>Environment:
System: FreeBSD barton.dreadbsd.org 6.0-CURRENT FreeBSD 6.0-CURRENT #1: Thu Dec 30 19:02:55 CET 2004 antoine@barton.dreadbsd.org:/usr/obj/usr/src/sys/BARTON i386
>Description:
* Witness reports this on a -current box :

acquiring duplicate lock of same type: "network driver"
 1st nv0 @ /usr/ports/net/nvnet/work/nvnet/module/../src/if_nv.c:1511
 2nd skc0 @ /usr/src/sys/modules/sk/../../pci/if_sk.c:1112
KDB: stack backtrace:
witness_checkorder(c2855c44,9,c07316ba,458) at witness_checkorder+0x500
_mtx_lock_flags(c2855c44,0,c07316ba,458,c2ac4e00) at _mtx_lock_flags+0x40
sk_jfree(e5a44340,c28e3000,0,c2857c00,e3c79c80) at sk_jfree+0x33
mb_free_ext(c2ac4e00) at mb_free_ext+0x36
m_freem(c2ac4e00,c2857c00,e5901000,e56fb0e0,c2857e60) at m_freem+0x21
nv_ospackettx(c2857c00,e56fb0e0,1,e5901000,0) at nv_ospackettx+0x73
UpdateTransmitDescRingData() at UpdateTransmitDescRingData+0xd3

I think this can cause a deadlock an a box with 2 sk(4) interfaces if
the 2 interfaces free a buffer allocated by the other at the same time.
A patch is attached (use an sk_jlist_mtx).

* When compiling if_sk as a module and unloading it, there seems to be
a few leaks.
A patch that fixes some of them is attached, but there is probably more
leaks.
Especially mii_phy_probe() and miibus_probe() allocate things during
a probe routine without freeing them and I suspect it's bad.

>How-To-Repeat:
* Enable witness on a box that acts as a gateway.

* Do kldload/kldunload if_sk and look at vmstat -m|grep devbuf.

>Fix:

--- sk.patch begins here ---
Index: if_sk.c
===================================================================
RCS file: /home/ncvs/src/sys/pci/if_sk.c,v
retrieving revision 1.94
diff -u -r1.94 if_sk.c
--- if_sk.c	24 Dec 2004 14:13:38 -0000	1.94
+++ if_sk.c	30 Dec 2004 21:43:06 -0000
@@ -1041,6 +1041,7 @@
 
 	SLIST_INIT(&sc_if->sk_jfree_listhead);
 	SLIST_INIT(&sc_if->sk_jinuse_listhead);
+	mtx_init(&sc_if->sk_jlist_mtx, "sk_jlist_mtx", NULL, MTX_DEF);
 
 	/*
 	 * Now divide it up into 9K pieces and save the addresses
@@ -1053,8 +1054,15 @@
 		entry = malloc(sizeof(struct sk_jpool_entry), 
 		    M_DEVBUF, M_NOWAIT);
 		if (entry == NULL) {
-			free(sc_if->sk_cdata.sk_jumbo_buf, M_DEVBUF);
+			while (!SLIST_EMPTY(&sc_if->sk_jfree_listhead)) {
+				entry = SLIST_FIRST(&sc_if->sk_jfree_listhead);
+				SLIST_REMOVE_HEAD(&sc_if->sk_jfree_listhead, jpool_entries);
+				free(entry, M_DEVBUF);
+			}
+			contigfree(sc_if->sk_cdata.sk_jumbo_buf, SK_JMEM,
+			    M_DEVBUF);
 			sc_if->sk_cdata.sk_jumbo_buf = NULL;
+			mtx_destroy(&sc_if->sk_jlist_mtx);
 			printf("sk%d: no memory for jumbo "
 			    "buffer queue!\n", sc_if->sk_unit);
 			return(ENOBUFS);
@@ -1076,7 +1084,7 @@
 {
 	struct sk_jpool_entry   *entry;
 
-	SK_IF_LOCK_ASSERT(sc_if);
+	SK_JLIST_LOCK(sc_if);
 
 	entry = SLIST_FIRST(&sc_if->sk_jfree_listhead);
 
@@ -1089,6 +1097,9 @@
 
 	SLIST_REMOVE_HEAD(&sc_if->sk_jfree_listhead, jpool_entries);
 	SLIST_INSERT_HEAD(&sc_if->sk_jinuse_listhead, entry, jpool_entries);
+
+	SK_JLIST_UNLOCK(sc_if);
+
 	return(sc_if->sk_cdata.sk_jslots[entry->slot]);
 }
 
@@ -1109,7 +1120,7 @@
 	if (sc_if == NULL)
 		panic("sk_jfree: didn't get softc pointer!");
 
-	SK_IF_LOCK(sc_if);
+	SK_JLIST_LOCK(sc_if);
 
 	/* calculate the slot this buffer belongs to */
 	i = ((vm_offset_t)buf
@@ -1125,7 +1136,7 @@
 	SLIST_REMOVE_HEAD(&sc_if->sk_jinuse_listhead, jpool_entries);
 	SLIST_INSERT_HEAD(&sc_if->sk_jfree_listhead, entry, jpool_entries);
 
-	SK_IF_UNLOCK(sc_if);
+	SK_JLIST_UNLOCK(sc_if);
 	return;
 }
 
@@ -1737,8 +1748,21 @@
 		device_delete_child(dev, sc_if->sk_miibus);
 	*/
 	bus_generic_detach(dev);
-	if (sc_if->sk_cdata.sk_jumbo_buf != NULL)
+	if (sc_if->sk_cdata.sk_jumbo_buf != NULL) {
+		struct sk_jpool_entry	*entry;
+		while (!SLIST_EMPTY(&sc_if->sk_jfree_listhead)) {
+			entry = SLIST_FIRST(&sc_if->sk_jfree_listhead);
+			SLIST_REMOVE_HEAD(&sc_if->sk_jfree_listhead, jpool_entries);
+			free(entry, M_DEVBUF);
+		}
+		while (!SLIST_EMPTY(&sc_if->sk_jinuse_listhead)) {
+			entry = SLIST_FIRST(&sc_if->sk_jinuse_listhead);
+			SLIST_REMOVE_HEAD(&sc_if->sk_jinuse_listhead, jpool_entries);
+			free(entry, M_DEVBUF);
+		}
 		contigfree(sc_if->sk_cdata.sk_jumbo_buf, SK_JMEM, M_DEVBUF);
+		mtx_destroy(&sc_if->sk_jlist_mtx);
+	}
 	if (sc_if->sk_rdata != NULL) {
 		contigfree(sc_if->sk_rdata, sizeof(struct sk_ring_data),
 		    M_DEVBUF);
Index: if_skreg.h
===================================================================
RCS file: /home/ncvs/src/sys/pci/if_skreg.h,v
retrieving revision 1.24
diff -u -r1.24 if_skreg.h
--- if_skreg.h	15 Nov 2004 19:37:21 -0000	1.24
+++ if_skreg.h	30 Dec 2004 17:42:22 -0000
@@ -1468,8 +1468,12 @@
 	int			sk_if_flags;
 	SLIST_HEAD(__sk_jfreehead, sk_jpool_entry)	sk_jfree_listhead;
 	SLIST_HEAD(__sk_jinusehead, sk_jpool_entry)	sk_jinuse_listhead;
+	struct mtx		sk_jlist_mtx;
 };
 
+#define	SK_JLIST_LOCK(_sc)	mtx_lock(&(_sc)->sk_jlist_mtx)
+#define	SK_JLIST_UNLOCK(_sc)	mtx_unlock(&(_sc)->sk_jlist_mtx)
+
 #define SK_MAXUNIT	256
 #define SK_TIMEOUT	1000
 #define ETHER_ALIGN	2
--- sk.patch ends here ---


>Release-Note:
>Audit-Trail:

From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>
To: Antoine Brodin <antoine.brodin@laposte.net>
Cc: FreeBSD-gnats-submit@FreeBSD.org
Subject: Re: kern/75677: [patch] sk(4): plug leaks and fix acquiring duplicate
 lock of same type: "network driver"
Date: Fri, 31 Dec 2004 09:12:45 +0000 (UTC)

 On Fri, 31 Dec 2004, Antoine Brodin wrote:
 
 > A patch that fixes some of them is attached, but there is probably more
 > leaks.
 > Especially mii_phy_probe() and miibus_probe() allocate things during
 > a probe routine without freeing them and I suspect it's bad.
 
 a patch[1] for mii already exists for the parts that are not specific
 to sk(4) (but this is outside of this PR).
 
 [1] http://sources.zabbadoz.net/freebsd/patchset/mii-memleaks.diff
Responsible-Changed-From-To: freebsd-bugs->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Sun Feb 27 14:59:27 GMT 2005 
Responsible-Changed-Why:  
I'll handle this. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=75677 

From: Antoine Brodin <antoine.brodin@laposte.net>
To: freebsd-gnats-submit@FreeBSD.org, antoine.brodin@laposte.net
Cc:  
Subject: Re: kern/75677: [patch] sk(4): plug leaks and fix acquiring
 duplicate lock of same type: "network driver"
Date: Sun, 27 Feb 2005 17:06:13 +0100

 I'm not happy about the following lines in the patch:
 in sk_detach():
 +		while (!SLIST_EMPTY(&sc_if->sk_jinuse_listhead)) {
 +			entry = SLIST_FIRST(&sc_if->sk_jinuse_listhead);
 +			SLIST_REMOVE_HEAD(&sc_if->sk_jinuse_listhead, jpool_entries);
 +			free(entry, M_DEVBUF);
 +		}
 
 Something like this would be probably better:
 
 KASSERT(SLIST_EMPTY(&sc_if->sk_jinuse_listhead),
     ("sk_detach: Jumbo buffers still in use"));

State-Changed-From-To: open->closed 
State-Changed-By: bz 
State-Changed-When: Thu Mar 31 09:50:32 GMT 2005 
State-Changed-Why:  
Changes committed to HEAD and MFCed for 5.4R. 
Note as stated in commit message: 
rev. 1.102 does not solve problems if buffers are still in 
use when unloading if_sk.ko. There is ongoing work which 
will address jumbogram allocations in a more general way. 

miibus fixes got posted to net@ for testing. 
(also needed for other drivers). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=75677 
>Unformatted:
