From os@brain.cc.rsu.ru  Tue Feb 19 21:54:25 2008
Return-Path: <os@brain.cc.rsu.ru>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D1CE816A46E
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 19 Feb 2008 21:54:25 +0000 (UTC)
	(envelope-from os@brain.cc.rsu.ru)
Received: from brain.cc.rsu.ru (brain.cc.rsu.ru [195.208.252.154])
	by mx1.freebsd.org (Postfix) with ESMTP id B098113C459
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 19 Feb 2008 21:54:24 +0000 (UTC)
	(envelope-from os@brain.cc.rsu.ru)
Received: from brain.cc.rsu.ru (localhost [127.0.0.1])
	by brain.cc.rsu.ru (8.14.2/8.14.2) with ESMTP id m1JLsLiN010249;
	Wed, 20 Feb 2008 00:54:21 +0300 (MSK)
	(envelope-from os@brain.cc.rsu.ru)
Received: (from os@localhost)
	by brain.cc.rsu.ru (8.14.2/8.14.2/Submit) id m1JLsLeA010248;
	Wed, 20 Feb 2008 00:54:21 +0300 (MSK)
	(envelope-from os)
Message-Id: <200802192154.m1JLsLeA010248@brain.cc.rsu.ru>
Date: Wed, 20 Feb 2008 00:54:21 +0300 (MSK)
From: Oleg Sharoiko <os@rsu.ru>
Reply-To: Oleg Sharoiko <os@rsu.ru>
To: FreeBSD-gnats-submit@freebsd.org
Cc: Oleg Sharoiko <os@rsu.ru>
Subject: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         120858
>Category:       kern
>Synopsis:       [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    scottl
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Feb 19 22:00:04 UTC 2008
>Closed-Date:    
>Last-Modified:  Wed Oct 12 00:50:11 UTC 2011
>Originator:     Oleg Sharoiko
>Release:        FreeBSD 7.0-RC1 amd64
>Organization:
Southern federal university
>Environment:
System: FreeBSD blade-3-1.r61.net 7.0-PRERELEASE FreeBSD 7.0-PRERELEASE #1: Tue Feb 19 19:00:51 UTC 2008 os@blade-3-1.r61.net:/usr/obj/usr/src/sys/GENERIC amd64


	
>Description:
	Had a chance to play with FreeBSD on IBM BladeCenter H and
EMC CLARiiON CX3-40. Moderate I/O activity on ufs filesystem gave
me a reproducable ufs_dirbad panic and corrupted root directory of
filesystem.

>How-To-Repeat:
	Assuming da0 is a LUN on CX3:

	newfs -U /dev/da0s1a
	mount /dev/da0s1a /mnt
	tar cf - -C / --one-file-system . usr var tmp | tar xvf -C /mnt

>Fix:
	Lowering queue depth down to 63 (with camcontrol tags da0 -N 63)
fixes this issue. I did several successfull make -9 buildworld and 
make -j buildkerel. With queue depth 64 I still got panics. Here is a patch
which sets maxtags to 63 for volumes on CLARiiON.

	The LUNs from CLARiON are identified by camcontrol devlist
as follows:

<DGC RAID 5 0324>                  at scbus0 target 0 lun 0 (pass0,da0)
<DGC RAID 5 0324>                  at scbus0 target 1 lun 0 (pass1,da1)
<DGC RAID 5 0324>                  at scbus1 target 0 lun 0 (pass2,da2)
<DGC RAID 5 0324>                  at scbus1 target 1 lun 0 (pass3,da3)

	I'm not sure if DGC is a sufficently narrow pattern for
xpt_quirk_table, but can't suggest anything better.

Index: cam_xpt.c
===================================================================
RCS file: /home/ncvs/src/sys/cam/cam_xpt.c,v
retrieving revision 1.190.2.1
diff -u -r1.190.2.1 cam_xpt.c
--- cam_xpt.c	31 Jan 2008 09:31:41 -0000	1.190.2.1
+++ cam_xpt.c	19 Feb 2008 20:33:30 -0000
@@ -632,6 +632,15 @@
 		CAM_QUIRK_NOSERIAL, /*mintags*/0, /*maxtags*/0
 	},
 	{
+		/*
+		 * It looks like EMC CLARiiON CX3 model 40 silently
+		 * drops commands when there are more than 63 commands
+		 * in a queue
+		 */
+		{ T_ANY, SIP_MEDIA_FIXED|SIP_MEDIA_REMOVABLE, "DGC", "*", "*" },
+		/*quirks*/0, /*mintags*/2, /*maxtags*/63
+	},
+	{
 		/* Default tagged queuing parameters for all devices */
 		{
 		  T_ANY, SIP_MEDIA_REMOVABLE|SIP_MEDIA_FIXED,
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->scottl 
Responsible-Changed-By: remko 
Responsible-Changed-When: Wed Feb 20 06:36:17 UTC 2008 
Responsible-Changed-Why:  
Hi scott this might be something for you (or thomas), can you give 
it a look please? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=120858 

From: Scott Long <scottl@samsco.org>
To: bug-followup@FreeBSD.org, os@rsu.ru
Cc:  
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
Date: Wed, 20 Feb 2008 08:10:07 -0700

 Can you post the inquiry information from this device, either from the 
 dmesg output for from running camcontrol?

From: Oleg Sharoiko <os@rsu.ru>
To: bug-followup@FreeBSD.org, os@rsu.ru
Cc:  
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON
	CX3-40
Date: Wed, 20 Feb 2008 23:52:10 +0300

 Here are lines from dmesg:
 
 da0 at isp0 bus 0 target 0 lun 0
 da0: <DGC RAID 5 0324> Fixed Direct Access SCSI-4 device 
 da0: Serial Number CK200073700517
 da0: 200.000MB/s transfers
 da0: Command Queueing Enabled
 da0: 102400MB (209715200 512 byte sectors: 255H 63S/T 13054C)
 
 --
 Oleg Sharoiko
 

From: Oleg Sharoyko <osharoiko@gmail.com>
To: Scott Long <scottl@samsco.org>, bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
Date: Wed, 10 Nov 2010 22:56:07 +0300

 --00163646d57ca900000494b83f95
 Content-Type: text/plain; charset=UTF-8
 
 Hi Scott,
 
 would you please, if is't possible, bring kern/120858
 (http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/120858) back to your
 attention. I have an updated patch against the recent current
 (attached to this mail) and it also looks like the actual limit is
 only 32 tags but not 63 as I have thought earlier. Please let me know
 if you need more information on this issue.
 
 -- 
 Oleg Sharoyko
 
 --00163646d57ca900000494b83f95
 Content-Type: text/x-patch; charset=US-ASCII; name="clariion.diff"
 Content-Disposition: attachment; filename="clariion.diff"
 Content-Transfer-Encoding: base64
 X-Attachment-Id: f_ggcml64i0
 
 SW5kZXg6IHN5cy9jYW0vc2NzaS9zY3NpX3hwdC5jCj09PT09PT09PT09PT09PT09PT09PT09PT09
 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHN5cy9jYW0vc2Nz
 aS9zY3NpX3hwdC5jCShyZXZpc2lvbiAyMTUwOTgpCisrKyBzeXMvY2FtL3Njc2kvc2NzaV94cHQu
 Ywkod29ya2luZyBjb3B5KQpAQCAtNTI1LDYgKzUyNSwxNSBAQAogCQlDQU1fUVVJUktfTk9MVU5T
 LCAvKm1pbnRhZ3MqLzAsIC8qbWF4dGFncyovMAogCX0sCiAJeworCQkvKgorCQkgKiBJdCBsb29r
 cyBsaWtlIEVNQyBDTEFSaWlPTiBDWDMgbW9kZWwgNDAgc2lsZW50bHkKKwkJICogZHJvcHMgY29t
 bWFuZHMgd2hlbiB0aGVyZSBhcmUgbW9yZSB0aGFuIDMyIGNvbW1hbmRzCisJCSAqIGluIGEgcXVl
 dWUKKwkJICovCisJCXsgVF9BTlksIFNJUF9NRURJQV9GSVhFRHxTSVBfTUVESUFfUkVNT1ZBQkxF
 LCAiREdDIiwgIioiLCAiKiIgfSwKKwkJLypxdWlya3MqLzAsIC8qbWludGFncyovMiwgLyptYXh0
 YWdzKi8zMgorCX0sCisJewogCQkvKiBEZWZhdWx0IHRhZ2dlZCBxdWV1aW5nIHBhcmFtZXRlcnMg
 Zm9yIGFsbCBkZXZpY2VzICovCiAJCXsKIAkJICBUX0FOWSwgU0lQX01FRElBX1JFTU9WQUJMRXxT
 SVBfTUVESUFfRklYRUQsCg==
 --00163646d57ca900000494b83f95--

From: Panagiotis Christias <p.christias@noc.ntua.gr>
To: bug-followup@FreeBSD.org, os@rsu.ru
Cc:  
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
Date: Tue, 05 Apr 2011 10:49:35 +0300

 A kind request/reminder regarding this patch. 32 tag openings is right 
 number for CLARiiON devices as reported in EMC's documentation (I can 
 provide more details).
 
 Regards,
 Panagiotis
 
 -- 
 Panagiotis J. Christias    Network Management Center
 p.christias@noc.ntua.gr    National Technical Univ. of Athens, GREECE

From: Pavel Timofeev <timp87@gmail.com>
To: bug-followup@FreeBSD.org, os@rsu.ru
Cc:  
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
Date: Tue, 4 Oct 2011 15:27:36 +0400

 --00151747c09addab7904ae77609c
 Content-Type: text/plain; charset=UTF-8
 
 Excuse me, I'm currently testing FreeBSD 9 BETA-3 amd64 with EMC CLARiiON
 CX3-40 via iSCSI.
 
 And I have problems! It works very unstable and sometimes gives "panic:
 ffs_alloccg: map corrupted".
 
 Have you any advices?
 
 
 
 Panagiotis Christias, where can I find such documentation?
 
 --00151747c09addab7904ae77609c
 Content-Type: text/html; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 
 
 <p class=3D"MsoPlainText"><span style=3D"mso-ansi-language:EN-US" lang=3D"E=
 N-US">Excuse
 me, I&#39;m currently testing FreeBSD 9 BETA-3 amd64 with EMC CLARiiON CX3-=
 40 via
 iSCSI.</span></p>
 
 <p class=3D"MsoPlainText"><span style=3D"mso-ansi-language:EN-US" lang=3D"E=
 N-US">And I
 have problems! It works very unstable and sometimes gives &quot;panic:
 ffs_alloccg: map corrupted&quot;.</span></p>
 
 <p class=3D"MsoPlainText"><span style=3D"mso-ansi-language:EN-US" lang=3D"E=
 N-US">Have you
 any advices?</span></p>
 
 <p class=3D"MsoPlainText"><span style=3D"mso-ansi-language:EN-US" lang=3D"E=
 N-US">=C2=A0</span></p>
 
 <p class=3D"MsoPlainText"><span style=3D"mso-ansi-language:EN-US" lang=3D"E=
 N-US">Panagiotis
 Christias, where can I find such documentation?</span></p>
 
 
 --00151747c09addab7904ae77609c--

From: Oleg Sharoyko <osharoiko@gmail.com>
To: Pavel Timofeev <timp87@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
Date: Tue, 11 Oct 2011 21:12:25 +0100

 Hello Pavel,
 
 On 4 October 2011 12:27, Pavel Timofeev <timp87@gmail.com> wrote:
 
 > Excuse me, I'm currently testing FreeBSD 9 BETA-3 amd64 with EMC CLARiiON
 > CX3-40 via iSCSI.
 > And I have problems! It works very unstable and sometimes gives "panic:
 > ffs_alloccg: map corrupted".
 > Have you any advices?
 
 I'm afraid I can't offer much help. I don't have access to the
 equipment any more and have never played with iSCSI. You could try
 debugging it, but you should be ready to dig the code. Another thing
 you could try is to limit the number of tags to a very low value using
 `camcontrol tags' and check if that helps. I'd recommend recreating a
 file system with newfs after changing tags and before running any
 tests. If setting a low value helps you can try binary search to find
 the maximum number of tags that works for you. I would expect a
 significant drop in performance when running with a low (like 2 or
 near that) number of tags and better results with more tags.
 
 Hope that helps.
 Regards
 -- 
 Oleg

From: Pavel Timofeev <timp87@gmail.com>
To: Oleg Sharoyko <osharoiko@gmail.com>
Cc: bug-followup@freebsd.org
Subject: Re: kern/120858: [patch] [cam] panic: ufs_dirbad with CLARiiON CX3-40
Date: Wed, 12 Oct 2011 04:49:09 +0400

 --0015174736725cb4f104af0f64b1
 Content-Type: text/plain; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 Thank you, Oleg!
 But bug was found.
 http://www.freebsd.org/cgi/query-pr.cgi?pr=3D160943
 It covers many devices from different vendors.
 I've tested various opening tags value. Without this patch system crashes
 always.
 But if it patched increasing opening tag gets more performance and doesn't
 affect stability.
 12.10.2011 0:12 =D0=BF=D0=BE=D0=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=
 =B5=D0=BB=D1=8C "Oleg Sharoyko" <osharoiko@gmail.com> =D0=BD=D0=B0=D0=BF=D0=
 =B8=D1=81=D0=B0=D0=BB:
 
 > Hello Pavel,
 >
 > On 4 October 2011 12:27, Pavel Timofeev <timp87@gmail.com> wrote:
 >
 > > Excuse me, I'm currently testing FreeBSD 9 BETA-3 amd64 with EMC CLARii=
 ON
 > > CX3-40 via iSCSI.
 > > And I have problems! It works very unstable and sometimes gives "panic:
 > > ffs_alloccg: map corrupted".
 > > Have you any advices?
 >
 > I'm afraid I can't offer much help. I don't have access to the
 > equipment any more and have never played with iSCSI. You could try
 > debugging it, but you should be ready to dig the code. Another thing
 > you could try is to limit the number of tags to a very low value using
 > `camcontrol tags' and check if that helps. I'd recommend recreating a
 > file system with newfs after changing tags and before running any
 > tests. If setting a low value helps you can try binary search to find
 > the maximum number of tags that works for you. I would expect a
 > significant drop in performance when running with a low (like 2 or
 > near that) number of tags and better results with more tags.
 >
 > Hope that helps.
 > Regards
 > --
 > Oleg
 >
 
 --0015174736725cb4f104af0f64b1
 Content-Type: text/html; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 <p>Thank you, Oleg!<br>
 But bug was found. <br>
 <a href=3D"http://www.freebsd.org/cgi/query-pr.cgi?pr=3D160943">http://www.=
 freebsd.org/cgi/query-pr.cgi?pr=3D160943</a><br>
 It covers many devices from different vendors.<br>
 I&#39;ve tested various opening tags value. Without this patch system crash=
 es always.<br>
 But if it patched increasing opening tag gets more performance and doesn&#3=
 9;t affect stability.</p>
 <div class=3D"gmail_quote">12.10.2011 0:12 =D0=BF=D0=BE=D0=BB=D1=8C=D0=B7=
 =D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C &quot;Oleg Sharoyko&quot; &lt;<a=
  href=3D"mailto:osharoiko@gmail.com">osharoiko@gmail.com</a>&gt; =D0=BD=D0=
 =B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB:<br type=3D"attribution"><blockquote clas=
 s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
 ding-left:1ex">
 Hello Pavel,<br>
 <br>
 On 4 October 2011 12:27, Pavel Timofeev &lt;<a href=3D"mailto:timp87@gmail.=
 com">timp87@gmail.com</a>&gt; wrote:<br>
 <br>
 &gt; Excuse me, I&#39;m currently testing FreeBSD 9 BETA-3 amd64 with EMC C=
 LARiiON<br>
 &gt; CX3-40 via iSCSI.<br>
 &gt; And I have problems! It works very unstable and sometimes gives &quot;=
 panic:<br>
 &gt; ffs_alloccg: map corrupted&quot;.<br>
 &gt; Have you any advices?<br>
 <br>
 I&#39;m afraid I can&#39;t offer much help. I don&#39;t have access to the<=
 br>
 equipment any more and have never played with iSCSI. You could try<br>
 debugging it, but you should be ready to dig the code. Another thing<br>
 you could try is to limit the number of tags to a very low value using<br>
 `camcontrol tags&#39; and check if that helps. I&#39;d recommend recreating=
  a<br>
 file system with newfs after changing tags and before running any<br>
 tests. If setting a low value helps you can try binary search to find<br>
 the maximum number of tags that works for you. I would expect a<br>
 significant drop in performance when running with a low (like 2 or<br>
 near that) number of tags and better results with more tags.<br>
 <br>
 Hope that helps.<br>
 Regards<br>
 --<br>
 Oleg<br>
 </blockquote></div>
 
 --0015174736725cb4f104af0f64b1--
>Unformatted:
