From nobody@FreeBSD.org  Tue Nov 10 02:38:15 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id A1942106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 10 Nov 2009 02:38:15 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 8EC1C8FC1D
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 10 Nov 2009 02:38:15 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id nAA2cEJI000375
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 10 Nov 2009 02:38:14 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id nAA2cEdL000374;
	Tue, 10 Nov 2009 02:38:14 GMT
	(envelope-from nobody)
Message-Id: <200911100238.nAA2cEdL000374@www.freebsd.org>
Date: Tue, 10 Nov 2009 02:38:14 GMT
From: Alexander Sack <asack@niksun.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: isp(4) causes threads to hang due to callwheel bucket list corruption
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         140438
>Category:       kern
>Synopsis:       [isp] isp(4) causes threads to hang due to callwheel bucket list corruption
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    mjacob
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Nov 10 02:40:01 UTC 2009
>Closed-Date:    Fri Dec 04 03:35:17 UTC 2009
>Last-Modified:  Fri Dec  4 03:40:01 UTC 2009
>Originator:     Alexander Sack
>Release:        RELENG_7, RELENG_7_2_RELEASE, RELENG_6_1_RELEASE
>Organization:
Niksun
>Environment:
FreeBSD 7.2-STABLE FreeBSD 7.2-STABLE #3: Mon Nov  9 18:23:49 EST 2009  amd6
>Description:
Hardware:

IBM x346 Series with 8GB of RAM
QLA2342 BIOS 1.47 and 1.52 connected to an external SAN
ispfw(4) 3.3.19 as well as 3.3.25 (taken from QLogic website)

The basic signature of the issue is that a process will call usleep() and the nanosleep() system call will never come back!  Quite disturbing.  The way I was reproducing the issue was by executing a test C binary that called usleep() in a tight loop, sample code:

    int n = 1;
    while (n <= times) {
	usleep(usecs);

	if ((n % 1000) == 0) {
	    printf("sleeper: called %d times\n", n);
	}

	if (gotSignal) {
	    printf("sleeper: received sig %d\n", theSignal);
	    theSignal = -1;
	    gotSignal = 0;
	}

	++n;
    }

[this based off of a colleague and friend's initial test program, I added signals so I could restart it and debug the sleepq path]

etc.  Clearly the above should NEVER hang on a system.  The problem stems from the isp(4) driver.  I found that if no fibre channel devices were found or the cards were removed, the system would act normally.  I also discovered that if I swap the 23xx 2GB FC card for a 24xx 4GB FC card, again the symptoms disappear.  Also if a SIGIO was sent to the process, it would restart (which means more than likely it was not sleep queue stuff, i.e. a valid td was there to setrunnable and move on).

Root Cause:

The issue stems from the fact that there is a race condition of who starts the isp(4)'s Lost Device Timer callout or isp_ldt() function.  The callout is started at attach time in the isp_attach() function.  This is called right after the firmware of the card is initialized via mbox command.  The isp_ldt() callout is also started in isp_async() in response to a ISPASYNC_LOOP_DOWN event.   The issue is the ISP_LOCK() is dropped in isp_attach() while interrupts are attached which allows a ISASYNC_LOOP_DOWN event to occur BEFORE the LDT callout is initialized (via callout_init) on the 23xx series of cards.  As a result, callout_reset() is called FIRST out of isp_async() BEFORE callout_init() in isp_attach() which depending on release (6.1 vs. 7.2) causes different callwheel side effects.  The gist of it is the callwheel bucket list gets corrupt where by other callout's that happen to fall on the bucket the isp_ldt() wound up on, get stuck (to go even further, the head of a bucket
  list head pointer gets set to NULL even though there is a valid callwheel_tailq first and last pointers.  When I walked the whole list backwards (i.e. follow last backwards) I found my sleeper on the corrupt list!  softclock() SKIPs the whole bucket list since it believes nothing is actually on it due to the corrupt head pointer being NULL and HENCE, valid callouts on the list never wake up and hang).  Again callout_init() needs to be called before the first callout_reset to avoid these side effects.  Depending on timing, you get different callwheel types of issues.  ALSO bare in mind, if you happen to get a boot where callout_init() does get called in time, everything just works as it should which makes this problem very sporadic.

I have not gone into thinking about how to protecting the callout wheel from misbehaving clients (i.e. perhaps a flag should be added and checked to ensure callout_init() (and variants) was called before allowing callout_reset() to proceed, a simple check has very minimal overhead, my 2 cents).

Just to note, it is my feeling that interrupts really shouldn't be serviced this early anyway but trying to rearrange the logic of isp_reset()->isp_init()->isp_attach() logic is way too much of a change.  This patch was been tested extensively on 7.2 and 6.1 with great success.

I know due to the 8Gbps FC support on RELENG_8 and HEAD, this problem may not exist, I have to verify that in my testing (also the callout stuff is also changed a lot on a per cpu basis so even if the problem DOES exist it may not manifest itself in the same fashion).  Though 7.x is still susceptible to this issue and I feel my patch or a variant should be considered for MFC ASAP.

Please feel free to email me for any further questions or comments!

Thanks!

-aps
>How-To-Repeat:
Run usleep() in a tight loop on a system, it will eventually hang.  I found that if I disconnect and replug in a FC port to cause LDT callout cycle, the box would hang pretty quickly (just got to get my sleep program on the same bucket as isp_ldt()).
>Fix:
The patch submitted below is an attempt to fix the issue while trying to do the minimal amount to keep regression testing low.  

It composes of two main fixes:

- Initialize the callout structures before attach time initialization
- Start LDT only once during attach time (either from the actual attach OR from the IPASYNC_LOOP_DOWN event, whatever comes first, drivers was designed the loop is down until isp_kthread() calls isp_fc_runstate() which will generate NOTIFY messages which will cancel the LDT otherwise it times out as usual). 

With the patch I no longer have any of these issues and my sleep program never hangs (without it, I can get it to hang almost immediately).  Was hoping to get some feedback from the core group before I go ahead and try this in the field.

Patch attached with submission follows:

Index: isp_freebsd.c
===================================================================
RCS file: /home/ncvs/src/sys/dev/isp/isp_freebsd.c,v
retrieving revision 1.146.2.1
diff -u -r1.146.2.1 isp_freebsd.c
--- isp_freebsd.c	13 May 2009 19:26:04 -0000	1.146.2.1
+++ isp_freebsd.c	10 Nov 2009 02:33:32 -0000
@@ -247,8 +247,6 @@
 	 * Create a kernel thread for fibre channel instances.
 	 */
 	if (IS_FC(isp)) {
-		isp_callout_init(&isp->isp_osinfo.ldt);
-		isp_callout_init(&isp->isp_osinfo.gdt);
 		ISP_UNLOCK(isp);
 #if __FreeBSD_version >= 500000  
 		if (kthread_create(isp_kthread, isp, &isp->isp_osinfo.kproc,
@@ -272,11 +270,13 @@
 		 */
 		if (isp->isp_role & ISP_ROLE_INITIATOR) {
 			isp_freeze_loopdown(isp, "isp_attach");
-			isp->isp_osinfo.ldt_running = 1;
-			callout_reset(&isp->isp_osinfo.ldt,
-			    isp_quickboot_time * hz, isp_ldt, isp);
-			isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0,
-			   "Starting Initial Loop Down Timer");
+			if (isp->isp_osinfo.ldt_running == 0) {
+			    	isp->isp_osinfo.ldt_running = 1;
+				callout_reset(&isp->isp_osinfo.ldt,
+			    	   isp_quickboot_time * hz, isp_ldt, isp);
+				isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0,
+				   "Starting Initial Loop Down Timer");
+			}
 		}
 	}
 }
Index: isp_pci.c
===================================================================
RCS file: /home/ncvs/src/sys/dev/isp/isp_pci.c,v
retrieving revision 1.148
diff -u -r1.148 isp_pci.c
--- isp_pci.c	26 Jun 2007 23:08:57 -0000	1.148
+++ isp_pci.c	10 Nov 2009 02:33:33 -0000
@@ -1193,6 +1193,15 @@
 	}
 
 	/*
+	 * Initialize callouts here to ensure no side-effects
+	 * after we start the ISP's firmware.  
+	 */
+	if (IS_FC(isp)) {
+	    	isp_callout_init(&isp->isp_osinfo.ldt);
+	    	isp_callout_init(&isp->isp_osinfo.gdt);
+	}
+
+	/*
 	 * Make sure we're in reset state.
 	 */
 	ISP_LOCK(isp);


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Nov 10 03:03:17 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140438 

From: Alexander Sack <asack@niksun.com>
To: bug-followup@FreeBSD.org, asack@niksun.com
Cc:  
Subject: Re: kern/140438: [isp] isp(4) causes threads to hang due to callwheel bucket list corruption
Date: Wed, 11 Nov 2009 15:03:00 -0500

 --Apple-Mail-7-489706079
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/plain;
 	charset=us-ascii
 
 RELENG_8 has same issue with my 2342 card.  Note that it looks like =
 during the 8Gbps refresh the check for ldt_running was made in both =
 spots (as well as the call to freeze the simq again, cool beans) so this =
 patch is a little smaller but in the same vain.
 
 I appreciate some comments on this.  It has been heavily tested in the =
 labs.
 
 -aps
 
 
 --Apple-Mail-7-489706079
 Content-Disposition: attachment;
 	filename=isp.RELENG_8.diff
 Content-Type: application/octet-stream;
 	name="isp.RELENG_8.diff"
 Content-Transfer-Encoding: 7bit
 
 Index: isp_freebsd.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/dev/isp/isp_freebsd.c,v
 retrieving revision 1.150.2.2
 diff -u -r1.150.2.2 isp_freebsd.c
 --- isp_freebsd.c	13 Aug 2009 01:45:26 -0000	1.150.2.2
 +++ isp_freebsd.c	11 Nov 2009 19:52:49 -0000
 @@ -139,9 +139,6 @@
  		fc->path = path;
  		fc->isp = isp;
  
 -		callout_init_mtx(&fc->ldt, &isp->isp_osinfo.lock, 0);
 -		callout_init_mtx(&fc->gdt, &isp->isp_osinfo.lock, 0);
 -
  		if (THREAD_CREATE(isp_kthread, fc, &fc->kproc, 0, 0, "%s: fc_thrd%d", device_get_nameunit(isp->isp_osinfo.dev), chan)) {
  			xpt_free_path(fc->path);
  			ISP_LOCK(isp);
 Index: isp_pci.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/dev/isp/isp_pci.c,v
 retrieving revision 1.149.2.1
 diff -u -r1.149.2.1 isp_pci.c
 --- isp_pci.c	3 Aug 2009 08:13:06 -0000	1.149.2.1
 +++ isp_pci.c	11 Nov 2009 19:52:50 -0000
 @@ -922,6 +922,17 @@
  	mtx_init(&isp->isp_osinfo.lock, "isp", NULL, MTX_DEF);
  	locksetup++;
  
 +	/*
 +	 * Initialize callouts before we kick start the ISP.
 +	 */
 +	if (IS_FC(isp)) {
 +		for (cmap = 0; cmap < isp->isp_nchan; cmap++) {
 +			struct isp_fc *fc = ISP_FC_PC(isp, cmap);
 +			callout_init_mtx(&fc->ldt, &isp->isp_osinfo.lock, 0);
 +			callout_init_mtx(&fc->gdt, &isp->isp_osinfo.lock, 0);
 +		}
 +	}
 +
  	if (isp_setup_intr(dev, irq, ISP_IFLAGS, NULL, isp_platform_intr, isp, &pcs->ih)) {
  		device_printf(dev, "could not setup interrupt\n");
  		goto bad;
 
 --Apple-Mail-7-489706079
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii
 
 
 
 --Apple-Mail-7-489706079--
Responsible-Changed-From-To: freebsd-scsi->mjacob 
Responsible-Changed-By: jkim 
Responsible-Changed-When: Wed Nov 11 21:21:59 UTC 2009 
Responsible-Changed-Why:  
mjacob may be able to handle this bug faster. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140438 

From: Alexander Sack <asack@niksun.com>
To: Alexander Sack <asack@niksun.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/140438: [isp] isp(4) causes threads to hang due to callwheel bucket list corruption
Date: Wed, 11 Nov 2009 17:31:52 -0500

 --Apple-Mail-16-498638602
 Content-Transfer-Encoding: quoted-printable
 Content-Type: text/plain;
 	charset=us-ascii
 
 
 On Nov 11, 2009, at 3:03 PM, Alexander Sack wrote:
 
 > RELENG_8 has same issue with my 2342 card.  Note that it looks like =
 during the 8Gbps refresh the check for ldt_running was made in both =
 spots (as well as the call to freeze the simq again, cool beans) so this =
 patch is a little smaller but in the same vain.
 >=20
 > I appreciate some comments on this.  It has been heavily tested in the =
 labs.
 >=20
 > -aps
 >=20
 > <isp.RELENG_8.diff>
 
 RELENG_7 notes:
 
 I would like the RELENG_7 patch to also initialize callouts BEFORE =
 bus_setup_intr etc. just to be safe.  At runtime I see no difference but =
 it is more technically prudent to do so.  Revised patch is attached.
 
 RELENG_7 vs RELENG_8 notes:
 
 I noticed in RELENG_8 the call to isp_loopdown_freeze() is under the =
 ldt_running at isp attach_chan() time.  In the interest of minimal =
 amount of changes, I did not incorporate this logic in RELENG_7.  I =
 focused totally on the callout race issue.
 
 One could do the samething in RELENG_7 but again, smallest amount of =
 changes. =20
 
 If we do want to buy back that change, then its like this:
 
 RELENG_8:
 153                 if ((FCPARAM(isp, chan)->role & ISP_ROLE_INITIATOR) =
 && fc->ldt_running =3D=3D 0) {
 154                         isp_freeze_loopdown(isp, chan, =
 "isp_attach");
 155                         fc->ldt_running =3D 1;
 156                 =20
 
 Could do this too:
 
 RELENG_7:
 271                 if (isp->isp_role & ISP_ROLE_INITIATOR && =
 isp->isp_osinfo.ldt_running =3D=3D 0) {
 
 etc.
 
 Again at runtime it seems no interrupt is available until the ISP is =
 kickstarted via mbox commands in isp_reset().
 
 Thanks!
 
 -aps
 
 
 --Apple-Mail-16-498638602
 Content-Disposition: attachment;
 	filename=isp.RELENG_7.diff
 Content-Type: application/octet-stream;
 	name="isp.RELENG_7.diff"
 Content-Transfer-Encoding: 7bit
 
 Index: isp_pci.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/dev/isp/isp_pci.c,v
 retrieving revision 1.148
 diff -u -r1.148 isp_pci.c
 --- isp_pci.c	26 Jun 2007 23:08:57 -0000	1.148
 +++ isp_pci.c	11 Nov 2009 22:22:26 -0000
 @@ -1171,6 +1171,14 @@
  	mtx_init(&isp->isp_osinfo.lock, "isp", NULL, MTX_DEF);
  	locksetup++;
  #endif
 +	
 +	/*
 +	 * Initialize callouts before kickstarting ISP. 
 +	 */
 +	if (IS_FC(isp)) {
 +		isp_callout_init(&isp->isp_osinfo.ldt);
 +		isp_callout_init(&isp->isp_osinfo.gdt);
 +	}
  
  	if (isp_setup_intr(dev, irq, ISP_IFLAGS, NULL, isp_platform_intr, isp,
  	    &pcs->ih)) {
 
 --Apple-Mail-16-498638602
 Content-Transfer-Encoding: 7bit
 Content-Type: text/plain;
 	charset=us-ascii
 
 
 
 
 
 --Apple-Mail-16-498638602--
State-Changed-From-To: open->closed 
State-Changed-By: mjacob 
State-Changed-When: Fri Dec 4 03:34:31 UTC 2009 
State-Changed-Why:  
Fixed in -current. The suggested fix isn't quite what was needed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=140438 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/140438: commit references a PR
Date: Fri,  4 Dec 2009 03:34:25 +0000 (UTC)

 Author: mjacob
 Date: Fri Dec  4 03:34:12 2009
 New Revision: 200089
 URL: http://svn.freebsd.org/changeset/base/200089
 
 Log:
   Fix cases where we've managed to get a Loop UP event prior to initializing
   the loop down counter, as well as other things. This was brought to my
   attention with a different fix, more for RELENG_7- this one covers the
   multiple channel case.
   
   PR:		140438
   MFC after:	1 month
 
 Modified:
   head/sys/dev/isp/isp_freebsd.c
   head/sys/dev/isp/isp_freebsd.h
 
 Modified: head/sys/dev/isp/isp_freebsd.c
 ==============================================================================
 --- head/sys/dev/isp/isp_freebsd.c	Thu Dec  3 23:57:06 2009	(r200088)
 +++ head/sys/dev/isp/isp_freebsd.c	Fri Dec  4 03:34:12 2009	(r200089)
 @@ -133,33 +133,37 @@ isp_attach_chan(ispsoftc_t *isp, struct 
  		}
  #endif
  	} else {
 +		fcparam *fcp = FCPARAM(isp, chan);
  		struct isp_fc *fc = ISP_FC_PC(isp, chan);
  
 +		ISP_LOCK(isp);
  		fc->sim = sim;
  		fc->path = path;
  		fc->isp = isp;
 +		fc->ready = 1;
  
  		callout_init_mtx(&fc->ldt, &isp->isp_osinfo.lock, 0);
  		callout_init_mtx(&fc->gdt, &isp->isp_osinfo.lock, 0);
 -
 -		if (THREAD_CREATE(isp_kthread, fc, &fc->kproc, 0, 0, "%s: fc_thrd%d", device_get_nameunit(isp->isp_osinfo.dev), chan)) {
 -			xpt_free_path(fc->path);
 -			ISP_LOCK(isp);
 -			xpt_bus_deregister(cam_sim_path(fc->sim));
 -			ISP_UNLOCK(isp);
 -			cam_sim_free(fc->sim, FALSE);
 -		}
  		/*
  		 * We start by being "loop down" if we have an initiator role
  		 */
 -		ISP_LOCK(isp);
 -		if ((FCPARAM(isp, chan)->role & ISP_ROLE_INITIATOR) && fc->ldt_running == 0) {
 +		if (fcp->role & ISP_ROLE_INITIATOR) {
  			isp_freeze_loopdown(isp, chan, "isp_attach");
 -			fc->ldt_running = 1;
  			callout_reset(&fc->ldt, isp_quickboot_time * hz, isp_ldt, fc);
  			isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "Starting Initial Loop Down Timer @ %lu", (unsigned long) time_uptime);
  		}
  		ISP_UNLOCK(isp);
 +		if (THREAD_CREATE(isp_kthread, fc, &fc->kproc, 0, 0, "%s: fc_thrd%d", device_get_nameunit(isp->isp_osinfo.dev), chan)) {
 +			xpt_free_path(fc->path);
 +			ISP_LOCK(isp);
 +			if (callout_active(&fc->ldt)) {
 +				callout_stop(&fc->ldt);
 +			}
 +			xpt_bus_deregister(cam_sim_path(fc->sim));
 +			ISP_UNLOCK(isp);
 +			cam_sim_free(fc->sim, FALSE);
 +			return (ENOMEM);
 +		}
  #ifdef	ISP_INTERNAL_TARGET
  		ISP_SET_PC(isp, chan, proc_active, 1);
  		if (THREAD_CREATE(isp_target_thread_fc, fc, &fc->target_proc, 0, 0, "%s: isp_test_tgt%d", device_get_nameunit(isp->isp_osinfo.dev), chan)) {
 @@ -3935,12 +3939,12 @@ isp_gdt(void *arg)
  		isp_prt(isp, ISP_LOGCONFIG, prom3, chan, lp->portid, tgt, "Gone Device Timeout");
  		isp_make_gone(isp, chan, tgt);
  	}
 -	if (more_to_do) {
 -		fc->gdt_running = 1;
 -		callout_reset(&fc->gdt, hz, isp_gdt, fc);
 -	} else {
 -		isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "Chan %d stopping Gone Device Timer", chan);
 -		fc->gdt_running = 0;
 +	if (fc->ready) {
 +		if (more_to_do) {
 +			callout_reset(&fc->gdt, hz, isp_gdt, fc);
 +		} else {
 +			isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "Chan %d stopping Gone Device Timer", chan);
 +		}
  	}
  }
  
 @@ -4017,6 +4021,7 @@ isp_kthread(void *arg)
  	ispsoftc_t *isp = fc->isp;
  	int chan = fc - isp->isp_osinfo.pc.fc;
  	int slp = 0;
 +
  	mtx_lock(&isp->isp_osinfo.lock);
  
  	for (;;) {
 @@ -4802,6 +4807,7 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  	char *msg = NULL;
  	target_id_t tgt;
  	fcportdb_t *lp;
 +	struct isp_fc *fc;
  	struct cam_path *tmppath;
  	va_list ap;
  
 @@ -4886,7 +4892,6 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		/* FALLTHROUGH */
  	case ISPASYNC_LOOP_DOWN:
  	{
 -		struct isp_fc *fc;
  		if (msg == NULL) {
  			msg = "LOOP Down";
  		}
 @@ -4894,20 +4899,21 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		bus = va_arg(ap, int);
  		va_end(ap);
  
 -		FCPARAM(isp, bus)->link_active = 1;
 +		FCPARAM(isp, bus)->link_active = 0;
  
  		fc = ISP_FC_PC(isp, bus);
 -		/*
 -		 * We don't do any simq freezing if we are only in target mode
 -		 */
 -		if (fc->role & ISP_ROLE_INITIATOR) {
 -			if (fc->path) {
 -				isp_freeze_loopdown(isp, bus, msg);
 -			}
 -			if (fc->ldt_running == 0) {
 -				fc->ldt_running = 1;
 -				callout_reset(&fc->ldt, fc->loop_down_limit * hz, isp_ldt, fc);
 -				isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "starting Loop Down Timer @ %lu", (unsigned long) time_uptime);
 +		if (cmd == ISPASYNC_LOOP_DOWN && fc->ready) {
 +			/*
 +			 * We don't do any simq freezing if we are only in target mode
 +			 */
 +			if (fc->role & ISP_ROLE_INITIATOR) {
 +				if (fc->path) {
 +					isp_freeze_loopdown(isp, bus, msg);
 +				}
 +				if (!callout_active(&fc->ldt)) {
 +					callout_reset(&fc->ldt, fc->loop_down_limit * hz, isp_ldt, fc);
 +					isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "starting Loop Down Timer @ %lu", (unsigned long) time_uptime);
 +				}
  			}
  		}
  		isp_prt(isp, ISP_LOGINFO, "Chan %d: %s", bus, msg);
 @@ -4917,6 +4923,7 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		va_start(ap, cmd);
  		bus = va_arg(ap, int);
  		va_end(ap);
 +		fc = ISP_FC_PC(isp, bus);
  		/*
  		 * Now we just note that Loop has come up. We don't
  		 * actually do anything because we're waiting for a
 @@ -4924,8 +4931,8 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		 * thread to look at the state of the loop again.
  		 */
  		FCPARAM(isp, bus)->link_active = 1;
 -		ISP_FC_PC(isp, bus)->loop_dead = 0;
 -		ISP_FC_PC(isp, bus)->loop_down_time = 0;
 +		fc->loop_dead = 0;
 +		fc->loop_down_time = 0;
  		isp_prt(isp, ISP_LOGINFO, "Chan %d Loop UP", bus);
  		break;
  	case ISPASYNC_DEV_ARRIVED:
 @@ -4933,8 +4940,9 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		bus = va_arg(ap, int);
  		lp = va_arg(ap, fcportdb_t *);
  		va_end(ap);
 +		fc = ISP_FC_PC(isp, bus);
  		lp->reserved = 0;
 -		if ((ISP_FC_PC(isp, bus)->role & ISP_ROLE_INITIATOR) && (lp->roles & (SVC3_TGT_ROLE >> SVC3_ROLE_SHIFT))) {
 +		if ((fc->role & ISP_ROLE_INITIATOR) && (lp->roles & (SVC3_TGT_ROLE >> SVC3_ROLE_SHIFT))) {
  			int dbidx = lp - FCPARAM(isp, bus)->portdb;
  			int i;
  
 @@ -4967,6 +4975,7 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		bus = va_arg(ap, int);
  		lp = va_arg(ap, fcportdb_t *);
  		va_end(ap);
 +		fc = ISP_FC_PC(isp, bus);
  		lp->reserved = 0;
  		if (isp_change_is_bad) {
  			lp->state = FC_PORTDB_STATE_NIL;
 @@ -5013,6 +5022,7 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		bus = va_arg(ap, int);
  		lp = va_arg(ap, fcportdb_t *);
  		va_end(ap);
 +		fc = ISP_FC_PC(isp, bus);
  		/*
  		 * If this has a virtual target and we haven't marked it
  		 * that we're going to have isp_gdt tell the OS it's gone,
 @@ -5025,10 +5035,9 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  			lp->reserved = 1;
  			lp->new_reserved = ISP_FC_PC(isp, bus)->gone_device_time;
  			lp->state = FC_PORTDB_STATE_ZOMBIE;
 -			if (ISP_FC_PC(isp, bus)->gdt_running == 0) {
 +			if (fc->ready && !callout_active(&fc->gdt)) {
  				isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "Chan %d starting Gone Device Timer", bus);
 -				ISP_FC_PC(isp, bus)->gdt_running = 1;
 -				callout_reset(&ISP_FC_PC(isp, bus)->gdt, hz, isp_gdt, ISP_FC_PC(isp, bus));
 +				callout_reset(&fc->gdt, hz, isp_gdt, fc);
  			}
  			tgt = lp->dev_map_idx - 1;
  			isp_prt(isp, ISP_LOGCONFIG, prom2, bus, lp->portid, lp->handle, roles[lp->roles], "gone zombie at", tgt, (uint32_t) (lp->port_wwn >> 32), (uint32_t) lp->port_wwn);
 @@ -5053,6 +5062,7 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  			nlstate = reason = 0;
  		}
  		va_end(ap);
 +		fc = ISP_FC_PC(isp, bus);
  
  		if (evt == ISPASYNC_CHANGE_PDB) {
  			msg = "Chan %d Port Database Changed";
 @@ -5065,16 +5075,15 @@ isp_async(ispsoftc_t *isp, ispasync_t cm
  		/*
  		 * If the loop down timer is running, cancel it.
  		 */
 -		if (ISP_FC_PC(isp, bus)->ldt_running) {
 +		if (fc->ready && callout_active(&fc->ldt)) {
  			isp_prt(isp, ISP_LOGSANCFG|ISP_LOGDEBUG0, "Stopping Loop Down Timer @ %lu", (unsigned long) time_uptime);
 -			ISP_FC_PC(isp, bus)->ldt_running = 0;
 -			callout_stop(&ISP_FC_PC(isp, bus)->ldt);
 +			callout_stop(&fc->ldt);
  		}
  		isp_prt(isp, ISP_LOGINFO, msg, bus);
 -		if (ISP_FC_PC(isp, bus)->role & ISP_ROLE_INITIATOR) {
 +		if (fc->role & ISP_ROLE_INITIATOR) {
  			isp_freeze_loopdown(isp, bus, msg);
  		}
 -		wakeup(ISP_FC_PC(isp, bus));
 +		wakeup(fc);
  		break;
  	}
  #ifdef	ISP_TARGET_MODE
 
 Modified: head/sys/dev/isp/isp_freebsd.h
 ==============================================================================
 --- head/sys/dev/isp/isp_freebsd.h	Thu Dec  3 23:57:06 2009	(r200088)
 +++ head/sys/dev/isp/isp_freebsd.h	Fri Dec  4 03:34:12 2009	(r200089)
 @@ -177,9 +177,9 @@ struct isp_fc {
  		hysteresis	: 8,
  		role		: 2,
  		gdt_running	: 1,
 -		ldt_running	: 1,
  		loop_dead	: 1,
 -		fcbsy		: 1;
 +		fcbsy		: 1,
 +		ready		: 1;
  	struct callout ldt;	/* loop down timer */
  	struct callout gdt;	/* gone device timer */
  #ifdef	ISP_TARGET_MODE
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
>Unformatted:
