From peterjeremy@optushome.com.au  Mon Dec 31 23:19:40 2007
Return-Path: <peterjeremy@optushome.com.au>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 3C2F916A419
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Dec 2007 23:19:40 +0000 (UTC)
	(envelope-from peterjeremy@optushome.com.au)
Received: from turion.vk2pj.dyndns.org (c220-239-20-82.belrs4.nsw.optusnet.com.au [220.239.20.82])
	by mx1.freebsd.org (Postfix) with ESMTP id 7291813C45A
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Dec 2007 23:19:39 +0000 (UTC)
	(envelope-from peterjeremy@optushome.com.au)
Received: from turion.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1])
	by turion.vk2pj.dyndns.org (8.14.2/8.14.2) with ESMTP id lBVNJbZY062121;
	Tue, 1 Jan 2008 10:19:37 +1100 (EST)
	(envelope-from peter@turion.vk2pj.dyndns.org)
Received: (from peter@localhost)
	by turion.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id lBVNJbDw062120;
	Tue, 1 Jan 2008 10:19:37 +1100 (EST)
	(envelope-from peter)
Message-Id: <200712312319.lBVNJbDw062120@turion.vk2pj.dyndns.org>
Date: Tue, 1 Jan 2008 10:19:37 +1100 (EST)
From: Peter Jeremy <peterj@freebsd.org>
Reply-To: Peter Jeremy <peterj@freebsd.org>
To: FreeBSD-gnats-submit@freebsd.org
Subject: [patch] Collect various stats regarding dc(4) interrupts
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         119205
>Category:       kern
>Synopsis:       [dc] [patch] Collect various stats regarding dc(4) interrupts
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    peterj
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Mon Dec 31 23:20:01 UTC 2007
>Closed-Date:    
>Last-Modified:  Fri Oct 19 15:40:01 UTC 2012
>Originator:     Peter Jeremy
>Release:        FreeBSD 6.3-PRERELEASE amd64
>Organization:
n/a
>Environment:
System: FreeBSD turion.vk2pj.dyndns.org 6.3-PRERELEASE FreeBSD 6.3-PRERELEASE #31: Wed Dec 26 09:23:54 EST 2007 root@turion.vk2pj.dyndns.org:/usr/obj/usr/src/sys/turion amd64

>Description:
	I wrote the attached patch some time ago to try and identify
	the breakdown of dc(4) interrupts as part of some investigations
	into the performance of a dc NIC that I was using.  It adds a new
	"hw.dc" sysctl tree and reports total interrupts, interrupt
	categories (abnormal, bus_err, normal, rx_nobuf, rx_ok,
	rx_watchdogtimeout, tx_idle, tx_nobuf, tx_ok, tx_underrun) and
	receiver resync events.  Note that this patch accumulates totals
	for all dc(4) devices, not per-device.

	The new sysctl nodes are:
hw.dc.abnormal	Abnormal interrupts
hw.dc.bus_err	"Bus error" interrupts
hw.dc.ints	Total interrupts
hw.dc.normal	"normal" interrupts
hw.dc.resync1	Resync needed following normal receive interrupt
hw.dc.resync2	Resync needed following watchdog timeout or no buffers
hw.dc.rx_nobuf	No receive buffers
hw.dc.rx_ok	Normal receive interrupt
hw.dc.rx_watdogtimeo  Watchdog timeout on receive
hw.dc.tx_idle	Transmitter idle
hw.dc.tx_nobuf	No transmit buffers
hw.dc.tx_ok	Normal transmit interrupt.
hw.dc.tx_underrun Transmit underrun

	It's possible they may be of use to someone else.  

>How-To-Repeat:
	code inspection
>Fix:
	See attached patches for RELENG_6 and -current/RELENG_7.  Note that
	the latter patch compiles cleanly but has not been tested.

--- dc.releng6.patch begins here ---
Index: sys/pci/if_dc.c
===================================================================
RCS file: /usr/ncvs/src/sys/pci/Attic/if_dc.c,v
retrieving revision 1.160.2.13
diff -u -r1.160.2.13 if_dc.c
--- sys/pci/if_dc.c	15 Sep 2007 10:00:51 -0000	1.160.2.13
+++ sys/pci/if_dc.c	25 Dec 2007 20:42:18 -0000
@@ -342,6 +342,47 @@
     "do not m_devget() in dc driver");
 #endif
 
+SYSCTL_NODE(_hw, OID_AUTO, dc, CTLFLAG_RD, 0, "if_dc parameters");
+
+static int dc_ints;
+SYSCTL_INT(_hw_dc, OID_AUTO, ints, CTLFLAG_RW, &dc_ints, 0, "");
+
+static int dc_isr_abnormal;
+SYSCTL_INT(_hw_dc, OID_AUTO, abnormal, CTLFLAG_RW, &dc_isr_abnormal, 0, "");
+
+static int dc_isr_bus_err;
+SYSCTL_INT(_hw_dc, OID_AUTO, bus_err, CTLFLAG_RW, &dc_isr_bus_err, 0, "");
+
+static int dc_isr_normal;
+SYSCTL_INT(_hw_dc, OID_AUTO, normal, CTLFLAG_RW, &dc_isr_normal, 0, "");
+
+static int dc_isr_rx_nobuf;
+SYSCTL_INT(_hw_dc, OID_AUTO, rx_nobuf, CTLFLAG_RW, &dc_isr_rx_nobuf, 0, "");
+
+static int dc_isr_rx_ok;
+SYSCTL_INT(_hw_dc, OID_AUTO, rx_ok, CTLFLAG_RW, &dc_isr_rx_ok, 0, "");
+
+static int dc_isr_rx_watdogtimeo;
+SYSCTL_INT(_hw_dc, OID_AUTO, rx_watdogtimeo, CTLFLAG_RW, &dc_isr_rx_watdogtimeo, 0, "");
+
+static int dc_isr_tx_idle;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_idle, CTLFLAG_RW, &dc_isr_tx_idle, 0, "");
+
+static int dc_isr_tx_nobuf;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_nobuf, CTLFLAG_RW, &dc_isr_tx_nobuf, 0, "");
+
+static int dc_isr_tx_ok;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_ok, CTLFLAG_RW, &dc_isr_tx_ok, 0, "");
+
+static int dc_isr_tx_underrun;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_underrun, CTLFLAG_RW, &dc_isr_tx_underrun, 0, "");
+
+static int dc_rx_resync1;
+SYSCTL_INT(_hw_dc, OID_AUTO, resync1, CTLFLAG_RW, &dc_rx_resync1, 0, "");
+
+static int dc_rx_resync2;
+SYSCTL_INT(_hw_dc, OID_AUTO, resync2, CTLFLAG_RW, &dc_rx_resync2, 0, "");
+
 DRIVER_MODULE(dc, cardbus, dc_driver, dc_devclass, 0, 0);
 DRIVER_MODULE(dc, pci, dc_driver, dc_devclass, 0, 0);
 DRIVER_MODULE(miibus, dc, miibus_driver, miibus_devclass, 0, 0);
@@ -3097,26 +3138,43 @@
 	/* Disable interrupts. */
 	CSR_WRITE_4(sc, DC_IMR, 0x00000000);
 
+	dc_ints++;
+
 	while (((status = CSR_READ_4(sc, DC_ISR)) & DC_INTRS) &&
 	    status != 0xFFFFFFFF &&
 	    (ifp->if_drv_flags & IFF_DRV_RUNNING)) {
 
 		CSR_WRITE_4(sc, DC_ISR, status);
 
+		if (status & DC_ISR_ABNORMAL)
+			dc_isr_abnormal++;
+		if (status & DC_ISR_NORMAL)
+			dc_isr_normal++;
+
 		if (status & DC_ISR_RX_OK) {
 			int		curpkts;
+
+			dc_isr_rx_ok++;
 			curpkts = ifp->if_ipackets;
 			dc_rxeof(sc);
 			if (curpkts == ifp->if_ipackets) {
-				while (dc_rx_resync(sc))
+				while (dc_rx_resync(sc)) {
+					dc_rx_resync1++;
 					dc_rxeof(sc);
+				}
 			}
 		}
 
-		if (status & (DC_ISR_TX_OK | DC_ISR_TX_NOBUF))
+	 	if (status & (DC_ISR_TX_OK | DC_ISR_TX_NOBUF)) {
+		 	if (status & DC_ISR_TX_OK)
+				dc_isr_tx_ok++;
+		 	if (status & DC_ISR_TX_NOBUF)
+				dc_isr_tx_nobuf++;
 			dc_txeof(sc);
+		}
 
 		if (status & DC_ISR_TX_IDLE) {
+			dc_isr_tx_idle++;
 			dc_txeof(sc);
 			if (sc->dc_cdata.dc_tx_cnt) {
 				DC_SETBIT(sc, DC_NETCFG, DC_NETCFG_TX_ON);
@@ -3124,21 +3182,30 @@
 			}
 		}
 
-		if (status & DC_ISR_TX_UNDERRUN)
+		if (status & DC_ISR_TX_UNDERRUN) {
+			dc_isr_tx_underrun++;
 			dc_tx_underrun(sc);
+		}
 
 		if ((status & DC_ISR_RX_WATDOGTIMEO)
 		    || (status & DC_ISR_RX_NOBUF)) {
 			int		curpkts;
+			if (status & DC_ISR_RX_WATDOGTIMEO)
+				dc_isr_rx_watdogtimeo++;
+			if (status & DC_ISR_RX_NOBUF)
+				dc_isr_rx_nobuf++;
 			curpkts = ifp->if_ipackets;
 			dc_rxeof(sc);
 			if (curpkts == ifp->if_ipackets) {
-				while (dc_rx_resync(sc))
+				while (dc_rx_resync(sc)) {
+					dc_rx_resync2++;
 					dc_rxeof(sc);
+				}
 			}
 		}
 
 		if (status & DC_ISR_BUS_ERR) {
+			dc_isr_bus_err++;
 			dc_reset(sc);
 			dc_init_locked(sc);
 		}
--- dc.releng6.patch ends here ---

--- dc.current.patch begins here ---
Index: sys/dev/dc/if_dc.c
===================================================================
RCS file: /usr/ncvs/src/sys/dev/dc/if_dc.c,v
retrieving revision 1.193
diff -u -r1.193 if_dc.c
--- sys/dev/dc/if_dc.c	22 Nov 2007 02:44:58 -0000	1.193
+++ sys/dev/dc/if_dc.c	23 Nov 2007 16:07:17 -0000
@@ -104,6 +104,7 @@
 #include <sys/kernel.h>
 #include <sys/module.h>
 #include <sys/socket.h>
+#include <sys/sysctl.h>
 
 #include <net/if.h>
 #include <net/if_arp.h>
@@ -333,6 +334,47 @@
 
 static devclass_t dc_devclass;
 
+SYSCTL_NODE(_hw, OID_AUTO, dc, CTLFLAG_RD, 0, "if_dc parameters");
+
+static int dc_ints;
+SYSCTL_INT(_hw_dc, OID_AUTO, ints, CTLFLAG_RW, &dc_ints, 0, "");
+
+static int dc_isr_abnormal;
+SYSCTL_INT(_hw_dc, OID_AUTO, abnormal, CTLFLAG_RW, &dc_isr_abnormal, 0, "");
+
+static int dc_isr_bus_err;
+SYSCTL_INT(_hw_dc, OID_AUTO, bus_err, CTLFLAG_RW, &dc_isr_bus_err, 0, "");
+
+static int dc_isr_normal;
+SYSCTL_INT(_hw_dc, OID_AUTO, normal, CTLFLAG_RW, &dc_isr_normal, 0, "");
+
+static int dc_isr_rx_nobuf;
+SYSCTL_INT(_hw_dc, OID_AUTO, rx_nobuf, CTLFLAG_RW, &dc_isr_rx_nobuf, 0, "");
+
+static int dc_isr_rx_ok;
+SYSCTL_INT(_hw_dc, OID_AUTO, rx_ok, CTLFLAG_RW, &dc_isr_rx_ok, 0, "");
+
+static int dc_isr_rx_watdogtimeo;
+SYSCTL_INT(_hw_dc, OID_AUTO, rx_watdogtimeo, CTLFLAG_RW, &dc_isr_rx_watdogtimeo, 0, "");
+
+static int dc_isr_tx_idle;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_idle, CTLFLAG_RW, &dc_isr_tx_idle, 0, "");
+
+static int dc_isr_tx_nobuf;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_nobuf, CTLFLAG_RW, &dc_isr_tx_nobuf, 0, "");
+
+static int dc_isr_tx_ok;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_ok, CTLFLAG_RW, &dc_isr_tx_ok, 0, "");
+
+static int dc_isr_tx_underrun;
+SYSCTL_INT(_hw_dc, OID_AUTO, tx_underrun, CTLFLAG_RW, &dc_isr_tx_underrun, 0, "");
+
+static int dc_rx_resync1;
+SYSCTL_INT(_hw_dc, OID_AUTO, resync1, CTLFLAG_RW, &dc_rx_resync1, 0, "");
+
+static int dc_rx_resync2;
+SYSCTL_INT(_hw_dc, OID_AUTO, resync2, CTLFLAG_RW, &dc_rx_resync2, 0, "");
+
 DRIVER_MODULE(dc, cardbus, dc_driver, dc_devclass, 0, 0);
 DRIVER_MODULE(dc, pci, dc_driver, dc_devclass, 0, 0);
 DRIVER_MODULE(miibus, dc, miibus_driver, miibus_devclass, 0, 0);
@@ -3065,26 +3107,43 @@
 	/* Disable interrupts. */
 	CSR_WRITE_4(sc, DC_IMR, 0x00000000);
 
+	dc_ints++;
+
 	while (((status = CSR_READ_4(sc, DC_ISR)) & DC_INTRS) &&
 	    status != 0xFFFFFFFF &&
 	    (ifp->if_drv_flags & IFF_DRV_RUNNING)) {
 
 		CSR_WRITE_4(sc, DC_ISR, status);
 
+		if (status & DC_ISR_ABNORMAL)
+			dc_isr_abnormal++;
+		if (status & DC_ISR_NORMAL)
+			dc_isr_normal++;
+
 		if (status & DC_ISR_RX_OK) {
 			int		curpkts;
+
+			dc_isr_rx_ok++;
 			curpkts = ifp->if_ipackets;
 			dc_rxeof(sc);
 			if (curpkts == ifp->if_ipackets) {
-				while (dc_rx_resync(sc))
+				while (dc_rx_resync(sc)) {
+					dc_rx_resync1++;
 					dc_rxeof(sc);
+				}
 			}
 		}
 
-		if (status & (DC_ISR_TX_OK | DC_ISR_TX_NOBUF))
+	 	if (status & (DC_ISR_TX_OK | DC_ISR_TX_NOBUF)) {
+		 	if (status & DC_ISR_TX_OK)
+				dc_isr_tx_ok++;
+		 	if (status & DC_ISR_TX_NOBUF)
+				dc_isr_tx_nobuf++;
 			dc_txeof(sc);
+		}
 
 		if (status & DC_ISR_TX_IDLE) {
+			dc_isr_tx_idle++;
 			dc_txeof(sc);
 			if (sc->dc_cdata.dc_tx_cnt) {
 				DC_SETBIT(sc, DC_NETCFG, DC_NETCFG_TX_ON);
@@ -3092,21 +3151,30 @@
 			}
 		}
 
-		if (status & DC_ISR_TX_UNDERRUN)
+		if (status & DC_ISR_TX_UNDERRUN) {
+			dc_isr_tx_underrun++;
 			dc_tx_underrun(sc);
+		}
 
 		if ((status & DC_ISR_RX_WATDOGTIMEO)
 		    || (status & DC_ISR_RX_NOBUF)) {
 			int		curpkts;
+			if (status & DC_ISR_RX_WATDOGTIMEO)
+				dc_isr_rx_watdogtimeo++;
+			if (status & DC_ISR_RX_NOBUF)
+				dc_isr_rx_nobuf++;
 			curpkts = ifp->if_ipackets;
 			dc_rxeof(sc);
 			if (curpkts == ifp->if_ipackets) {
-				while (dc_rx_resync(sc))
+				while (dc_rx_resync(sc)) {
+					dc_rx_resync2++;
 					dc_rxeof(sc);
+				}
 			}
 		}
 
 		if (status & DC_ISR_BUS_ERR) {
+			dc_isr_bus_err++;
 			dc_reset(sc);
 			dc_init_locked(sc);
 		}
--- dc.current.patch ends here ---


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->edwin 
Responsible-Changed-By: edwin 
Responsible-Changed-When: Mon Dec 31 23:24:49 UTC 2007 
Responsible-Changed-Why:  
I will handle it with submitter and mentor. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119205 

From: Edwin Groothuis <edwin@mavetju.org>
To: FreeBSD Gnats Submit <freebsd-gnats-submit@freebsd.org>
Cc:  
Subject: Re: kern/119205: [patch] Collect various stats regarding dc(4) interrupts
Date: Tue, 15 Jan 2008 18:28:16 +1100

 Maybe it should be SYSCTL_ULONG instead of SYSCTL_INT.
 Maybe hide it behind #ifdef DC_DEBUG or something.
 
 
 -- 
 Edwin Groothuis      |            Personal website: http://www.mavetju.org
 edwin@mavetju.org    |              Weblog: http://www.mavetju.org/weblog/
Responsible-Changed-From-To: edwin->freebsd-bugs 
Responsible-Changed-By: edwin 
Responsible-Changed-When: Thu Feb 14 10:34:20 UTC 2008 
Responsible-Changed-Why:  

Give back into the pool until later. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119205 
State-Changed-From-To: open->feedback 
State-Changed-By: eadler 
State-Changed-When: Sun Mar 25 03:33:58 UTC 2012 
State-Changed-Why:  
is this still required and does the patch need to be updated? I don't 
have the hardware to test. 


Responsible-Changed-From-To: freebsd-bugs->eadler 
Responsible-Changed-By: eadler 
Responsible-Changed-When: Sun Mar 25 03:33:58 UTC 2012 
Responsible-Changed-Why:  
is this still required and does the patch need to be updated? I don't 
have the hardware to test. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119205 
State-Changed-From-To: feedback->closed 
State-Changed-By: eadler 
State-Changed-When: Tue Jun 19 06:57:18 UTC 2012 
State-Changed-Why:  
if this is still a problem please send me an email and I'll reopen 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119205 
State-Changed-From-To: closed->open 
State-Changed-By: eadler 
State-Changed-When: Sat Jun 23 21:15:14 UTC 2012 
State-Changed-Why:  
per submitter request 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119205 
Responsible-Changed-From-To: eadler->peterj 
Responsible-Changed-By: peterj 
Responsible-Changed-When: Fri Oct 19 03:40:23 UTC 2012 
Responsible-Changed-Why:  
Take PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=119205 

From: John Baldwin <jhb@freebsd.org>
To: bug-followup@freebsd.org,
 peterj@freebsd.org
Cc:  
Subject: Re: kern/119205: [dc] [patch] Collect various stats regarding dc(4) interrupts
Date: Fri, 19 Oct 2012 09:05:09 -0400

 I would suggest making these per-device instead.  It should be fairly easy to 
 put the counters in the softc and add these to the existing per-device sysctl 
 tree from new-bus.
 
 -- 
 John Baldwin
>Unformatted:
