From nobody@FreeBSD.org  Tue Dec  8 04:19:12 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 5C7C11065693
	for <freebsd-gnats-submit@FreeBSD.org>; Tue,  8 Dec 2009 04:19:12 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 405B08FC27
	for <freebsd-gnats-submit@FreeBSD.org>; Tue,  8 Dec 2009 04:19:12 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id nB84JBoT000793
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 8 Dec 2009 04:19:11 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id nB84JBOX000790;
	Tue, 8 Dec 2009 04:19:11 GMT
	(envelope-from nobody)
Message-Id: <200912080419.nB84JBOX000790@www.freebsd.org>
Date: Tue, 8 Dec 2009 04:19:11 GMT
From: Alexander Sack <asack@niksun.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: aac(4) FIB starvation during bus scan
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         141269
>Category:       kern
>Synopsis:       [aac] aac(4) FIB starvation during bus scan
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    jkim
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Dec 08 04:20:02 UTC 2009
>Closed-Date:    Tue Dec 08 05:39:17 UTC 2009
>Last-Modified:  Tue Dec  8 05:40:05 UTC 2009
>Originator:     Alexander Sack
>Release:        RELENG_8, CURRENT
>Organization:
Niksun
>Environment:
>Description:
aac(4) preallocates 128 FIBs during initialization time as its active command pool.  Unfortunately, on modern controllers with a max targets of 287 and several buses (e.g. a 5085 controller has 6), at xpt_bus_scan time, it runs out very quickly causing lots of requeue/retry cycles increasing boot time.  A sequential bus scan alleviates the problem completely.

Furthermore, the 128 number is half of what the minimum amount of FIBs modern controllers can handle.  As the driver is written today, the minimum number of FIBs is actually 256 (upper bound is 512).  In the interest of minimum command reallocation overhead, it might be wise to preallocate the maximum amount of FIBs which can also mitigate the BUS scan starvation issue described above.
>How-To-Repeat:
Use (2) 5085s in a single chassis and boot -v to see request/retry cycles
>Fix:
My patch I have been testing with a lot of success:

Index: aac.c
===================================================================
RCS file: /home/ncvs/src/sys/dev/aac/aac.c,v
retrieving revision 1.143.2.4
diff -u -r1.143.2.4 aac.c
--- aac.c       5 Nov 2009 18:34:01 -0000       1.143.2.4
+++ aac.c       7 Dec 2009 21:23:43 -0000
@@ -604,7 +604,7 @@
       TAILQ_INIT(&sc->aac_fibmap_tqh);
       sc->aac_commands = malloc(sc->aac_max_fibs * sizeof(struct aac_command),
                                 M_AACBUF, M_WAITOK|M_ZERO);
-       while (sc->total_fibs < AAC_PREALLOCATE_FIBS) {
+       while (sc->total_fibs < sc->aac_max_fibs) {
               if (aac_alloc_commands(sc) != 0)
                       break;
       }
Index: aac_cam.c
===================================================================
RCS file: /home/ncvs/src/sys/dev/aac/aac_cam.c,v
retrieving revision 1.31.2.2
diff -u -r1.31.2.2 aac_cam.c
--- aac_cam.c   5 Nov 2009 18:34:01 -0000       1.31.2.2
+++ aac_cam.c   7 Dec 2009 21:23:43 -0000
@@ -261,7 +261,7 @@
               cpi->target_sprt = 0;

               /* Resetting via the passthrough causes problems. */
-               cpi->hba_misc = PIM_NOBUSRESET;
+               cpi->hba_misc = PIM_NOBUSRESET | PIM_SEQSCAN;
               cpi->hba_eng_cnt = 0;
               cpi->max_target = camsc->inf->TargetsPerBus;
               cpi->max_lun = 8;       /* Per the controller spec */
Index: aacvar.h
===================================================================
RCS file: /home/ncvs/src/sys/dev/aac/aacvar.h,v
retrieving revision 1.52.2.2
diff -u -r1.52.2.2 aacvar.h
--- aacvar.h    2 Nov 2009 16:54:23 -0000       1.52.2.2
+++ aacvar.h    7 Dec 2009 21:23:44 -0000
@@ -57,13 +57,6 @@
 #define AAC_ADAPTER_FIBS       8

 /*
- * FIBs are allocated in page-size chunks and can grow up to the 512
- * limit imposed by the hardware.
- */
-#define AAC_PREALLOCATE_FIBS   128
-#define AAC_NUM_MGT_FIB                8
-
-/*
 * The controller reports status events in AIFs.  We hang on to a number of
 * these in order to pass them out to user-space management tools.
 */


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-amd64->freebsd-scsi 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Dec 8 04:29:54 UTC 2009 
Responsible-Changed-Why:  
Fix synopsis and category and assign. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=141269 
Responsible-Changed-From-To: freebsd-scsi->jkim 
Responsible-Changed-By: jkim 
Responsible-Changed-When: Tue Dec 8 05:06:34 UTC 2009 
Responsible-Changed-Why:  
I will take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=141269 
State-Changed-From-To: open->closed 
State-Changed-By: jkim 
State-Changed-When: Tue Dec 8 05:38:21 UTC 2009 
State-Changed-Why:  
The patch committed (with a minor comment change).  Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=141269 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/141269: commit references a PR
Date: Tue,  8 Dec 2009 05:36:04 +0000 (UTC)

 Author: jkim
 Date: Tue Dec  8 05:35:51 2009
 New Revision: 200251
 URL: http://svn.freebsd.org/changeset/base/200251
 
 Log:
   - Try pre-allocating all FIBs upfront.  Previously we tried pre-allocating
   128 FIBs first and allocated more later if necessary.  Remove now unused
   definitions from the header file[1].
   - Force sequential bus scanning.  It seems parallel scanning is in fact
   slower and causes more harm than good[1].  Adjust a comment to reflect that.
   
   PR:		kern/141269
   Submitted by:	Alexander Sack (asack at niksun dot com)[1]
   Reviewed by:	scottl
 
 Modified:
   head/sys/dev/aac/aac.c
   head/sys/dev/aac/aac_cam.c
   head/sys/dev/aac/aacvar.h
 
 Modified: head/sys/dev/aac/aac.c
 ==============================================================================
 --- head/sys/dev/aac/aac.c	Tue Dec  8 05:32:44 2009	(r200250)
 +++ head/sys/dev/aac/aac.c	Tue Dec  8 05:35:51 2009	(r200251)
 @@ -604,7 +604,7 @@ aac_alloc(struct aac_softc *sc)
  	TAILQ_INIT(&sc->aac_fibmap_tqh);
  	sc->aac_commands = malloc(sc->aac_max_fibs * sizeof(struct aac_command),
  				  M_AACBUF, M_WAITOK|M_ZERO);
 -	while (sc->total_fibs < AAC_PREALLOCATE_FIBS) {
 +	while (sc->total_fibs < sc->aac_max_fibs) {
  		if (aac_alloc_commands(sc) != 0)
  			break;
  	}
 
 Modified: head/sys/dev/aac/aac_cam.c
 ==============================================================================
 --- head/sys/dev/aac/aac_cam.c	Tue Dec  8 05:32:44 2009	(r200250)
 +++ head/sys/dev/aac/aac_cam.c	Tue Dec  8 05:35:51 2009	(r200251)
 @@ -260,8 +260,11 @@ aac_cam_action(struct cam_sim *sim, unio
  		cpi->hba_inquiry = PI_WIDE_16;
  		cpi->target_sprt = 0;
  
 -		/* Resetting via the passthrough causes problems. */
 -		cpi->hba_misc = PIM_NOBUSRESET;
 +		/*
 +		 * Resetting via the passthrough or parallel bus scan
 +		 * causes problems.
 +		 */
 +		cpi->hba_misc = PIM_NOBUSRESET | PIM_SEQSCAN;
  		cpi->hba_eng_cnt = 0;
  		cpi->max_target = camsc->inf->TargetsPerBus;
  		cpi->max_lun = 8;	/* Per the controller spec */
 
 Modified: head/sys/dev/aac/aacvar.h
 ==============================================================================
 --- head/sys/dev/aac/aacvar.h	Tue Dec  8 05:32:44 2009	(r200250)
 +++ head/sys/dev/aac/aacvar.h	Tue Dec  8 05:35:51 2009	(r200251)
 @@ -57,13 +57,6 @@
  #define AAC_ADAPTER_FIBS	8
  
  /*
 - * FIBs are allocated in page-size chunks and can grow up to the 512
 - * limit imposed by the hardware.
 - */
 -#define AAC_PREALLOCATE_FIBS	128
 -#define AAC_NUM_MGT_FIB		8
 -
 -/*
   * The controller reports status events in AIFs.  We hang on to a number of
   * these in order to pass them out to user-space management tools.
   */
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
