From nobody@FreeBSD.org  Fri Sep 23 16:31:16 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 2D83A106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 23 Sep 2011 16:31:16 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 041308FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 23 Sep 2011 16:31:16 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p8NGVFNF095273
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 23 Sep 2011 16:31:15 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p8NGVFvk095271;
	Fri, 23 Sep 2011 16:31:15 GMT
	(envelope-from nobody)
Message-Id: <201109231631.p8NGVFvk095271@red.freebsd.org>
Date: Fri, 23 Sep 2011 16:31:15 GMT
From: Craig Boston <cb@severious.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: iSCSI initiator ignores block offset causing silent data corruption
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         160943
>Category:       kern
>Synopsis:       [iscsi] [patch] iSCSI initiator ignores block offset causing silent data corruption
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kib
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Sep 23 16:40:09 UTC 2011
>Closed-Date:    Thu Oct 20 19:18:07 UTC 2011
>Last-Modified:  Fri Oct 21 06:30:13 UTC 2011
>Originator:     Craig Boston
>Release:        8.2 Stable
>Organization:
>Environment:
FreeBSD XXX 8.2-STABLE FreeBSD 8.2-STABLE #0: Wed Sep 21 14:58:49 CDT 2011     root@XXX:/compile/obj/compile/src/sys/GENERIC  amd64
>Description:
This is the result of troubleshooting silent data corruption issues when setting up an EqualLogic DS4000 iSCSI unit. The corruption was detected during testing as checksum errors in ZFS soon after the pool was created, but I was also able to reproduce the problem with UFS. It is identical to the issue reported here:

http://lists.freebsd.org/pipermail/freebsd-scsi/2010-June/004403.html

Down to the very same inode number being corrupt after a fresh newfs / fsck cycle.

After examining ktrace output from newfs and fsck to determine which block was different when read than when written, I cross referenced that with a network dump and identified the following exchange for a 64k write (simplified):

Initiator:  SCSI Write 128 blocks (i.e. length = 0x10000)
Target:     Ready to Transfer, desried data length = 0x0c000
Initiator:  Data out, length = 0x0c000 [correct data]
Target:     Ready to Transfer, buffer offset = 0x0c000, desired data length = 0x04000
Initiator:  Data out, length = 0x04000 [WRONG DATA!]

In the second data transfer from the initiator, it ignores the buffer offset, and instead sends the first 0x04000 bytes again. This results in incorrect data being written to the disk.

I'm not sure exactly why the EqualLogic unit sometimes sends R2Ts with a length of 0x10000 and sometimes uses 0x0c0000 (maybe related to its internal striping scheme). Such behavior is unusual, but perfectly valid according to the RFC. Whatever the reason, it's a bug that we don't correctly follow the iSCSI spec.

The attached patch corrects this.
>How-To-Repeat:
Get an EqualLogic DS4000 or DS6500, connect to it over iSCSI, and watch your data be silently corrupted as it's written.

Alternatively, modify a software iSCSI target to break transfers into smaller pieces by sending R2Ts with different offsets / lengths.
>Fix:
--- sys/dev/iscsi/initiator/iscsi_subr.c.orig   2011-09-23 10:38:12.000000000 -0500
+++ sys/dev/iscsi/initiator/iscsi_subr.c        2011-09-23 11:17:03.000000000 -0500
@@ -84,6 +84,7 @@
               caddr_t          bp = csio->data_ptr;

               bo = ntohl(r2t->bo);
+              bp += MIN(bo, edtl - ddtl);
               bleft = ddtl;

               if(sp->opt.maxXmitDataSegmentLength > 0) // danny's RFC

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-scsi 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Oct 10 04:40:36 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=160943 

From: Danny Braniss <danny@cs.huji.ac.il>
To: bug-followup@FreeBSD.org, cb@severious.net
Cc:  
Subject: Re: kern/160943: [iscsi] [patch] iSCSI initiator ignores block offset
 causing silent data corruption
Date: Mon, 10 Oct 2011 10:42:09 +0200

 --002215b03dbe3b6fbc04aeedc4c9
 Content-Type: text/plain; charset=ISO-8859-1
 
 good catch! I've been hunting this bug and was so close :-)
 it also fixes Clariion CX3-40
 
 need someone with a committer bit to apply it,
 thanks
 
 danny
 
 --002215b03dbe3b6fbc04aeedc4c9
 Content-Type: text/html; charset=ISO-8859-1
 
 <div dir="ltr">good catch! I&#39;ve been hunting this bug and was so close :-)<br>it also fixes Clariion CX3-40<br><br>need someone with a committer bit to apply it,<br>thanks<br><br>danny<br><br></div>
 
 --002215b03dbe3b6fbc04aeedc4c9--

From: Pavel Timofeev <timp87@gmail.com>
To: bug-followup@FreeBSD.org, cb@severious.net
Cc:  
Subject: Re: kern/160943: [iscsi] [patch] iSCSI initiator ignores block offset
 causing silent data corruption
Date: Mon, 10 Oct 2011 15:59:45 +0400

 --001517448242e10c2404aef0867c
 Content-Type: text/plain; charset=UTF-8
 
 This patch works for 9-BETA3!
 
 --001517448242e10c2404aef0867c
 Content-Type: text/html; charset=UTF-8
 
 This patch works for 9-BETA3!<br>
 
 --001517448242e10c2404aef0867c--
State-Changed-From-To: open->patched 
State-Changed-By: kib 
State-Changed-When: Mon Oct 10 13:27:04 UTC 2011 
State-Changed-Why:  
Grab, fix committed for HEAD. 


Responsible-Changed-From-To: freebsd-scsi->kib 
Responsible-Changed-By: kib 
Responsible-Changed-When: Mon Oct 10 13:27:04 UTC 2011 
Responsible-Changed-Why:  
Grab, fix committed for HEAD. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=160943 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/160943: commit references a PR
Date: Mon, 10 Oct 2011 13:27:02 +0000 (UTC)

 Author: kib
 Date: Mon Oct 10 13:26:53 2011
 New Revision: 226208
 URL: http://svn.freebsd.org/changeset/base/226208
 
 Log:
   Do not ignore block offsets.
   
   PR:	kern/160943
   Submitted by:	Craig Boston <cb severious net>
   Reviewed by:	Danny Braniss <danny cs huji ac il>
   MFC after:	1 week
 
 Modified:
   head/sys/dev/iscsi/initiator/iscsi_subr.c
 
 Modified: head/sys/dev/iscsi/initiator/iscsi_subr.c
 ==============================================================================
 --- head/sys/dev/iscsi/initiator/iscsi_subr.c	Mon Oct 10 13:18:55 2011	(r226207)
 +++ head/sys/dev/iscsi/initiator/iscsi_subr.c	Mon Oct 10 13:26:53 2011	(r226208)
 @@ -84,6 +84,7 @@ iscsi_r2t(isc_session_t *sp, pduq_t *opq
  	       caddr_t		bp = csio->data_ptr;
  
  	       bo = ntohl(r2t->bo);
 +	       bp += MIN(bo, edtl - ddtl);
  	       bleft = ddtl;
  
  	       if(sp->opt.maxXmitDataSegmentLength > 0) // danny's RFC
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: Rolf Grossmann <rg@xamine.com>
To: <bug-followup@FreeBSD.org>, <cb@severious.net>
Cc:  
Subject: Re: kern/160943: [iscsi] [patch] iSCSI initiator ignores block offset
 causing silent data corruption
Date: Mon, 10 Oct 2011 17:38:48 +0200

 Thank you so much for finding this bug!
 
 I can verify that this does fix the problem I reported in the message
 you quoted. Now I can finally use my PS6500 with FreeBSD. :)
 
 Thanks again, Rolf.
 

From: Pavel Timofeev <timp87@gmail.com>
To: bug-followup@FreeBSD.org, cb@severious.net
Cc:  
Subject: Re: kern/160943: [iscsi] [patch] iSCSI initiator ignores block offset
 causing silent data corruption
Date: Mon, 10 Oct 2011 20:12:16 +0400

 --0015174027a6f8683604aef40d91
 Content-Type: text/plain; charset=UTF-8
 
 I hope this patch will be in FreeBSD 8/9 STABLE and 9.0 RELEASE
 
 --0015174027a6f8683604aef40d91
 Content-Type: text/html; charset=UTF-8
 
 I hope this patch will be in FreeBSD 8/9 STABLE and 9.0 RELEASE<br>
 
 --0015174027a6f8683604aef40d91--

From: Pavel Timofeev <timp87@gmail.com>
To: bug-followup@FreeBSD.org, cb@severious.net
Cc:  
Subject: Re: kern/160943: [iscsi] [patch] iSCSI initiator ignores block offset
 causing silent data corruption
Date: Sun, 16 Oct 2011 18:27:12 +0400

 --00151747948e48f41704af6b491e
 Content-Type: text/plain; charset=UTF-8
 
 It's very very long-awaited fix for iscsi_initiator.
 Finally, our EMC Clariions can work with FreeBSD!
 And as Rolf Grossmann and Craig Boston said this patch fix work with
 EqualLogic devices too.
 
 I stress tested this patch most of this weekend. Everything is fine with
 Clariion!
 And patch doesn't affect stability for other iscsi targets (net/istgt,
 tested too)!
 Please, commit it to 9 stable.
 
 --00151747948e48f41704af6b491e
 Content-Type: text/html; charset=UTF-8
 Content-Transfer-Encoding: quoted-printable
 
 It&#39;s very very long-awaited fix for iscsi_initiator.<br>Finally, our EM=
 C Clariions can work with FreeBSD!<br>And as Rolf Grossmann and Craig Bosto=
 n said this patch fix work with EqualLogic devices too.<br><br>I stress tes=
 ted this patch most of this weekend. Everything is fine with Clariion!<br>
 And patch doesn&#39;t affect stability for other iscsi targets (net/istgt, =
 tested too)!<br>Please, commit it to 9 stable.<br>
 
 --00151747948e48f41704af6b491e--

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/160943: commit references a PR
Date: Thu, 20 Oct 2011 18:39:14 +0000 (UTC)

 Author: kib
 Date: Thu Oct 20 18:39:01 2011
 New Revision: 226576
 URL: http://svn.freebsd.org/changeset/base/226576
 
 Log:
   MFC r226208:
   Do not ignore block offsets.
   
   PR:	kern/160943
   Approved by:	re (bz)
 
 Modified:
   stable/9/sys/dev/iscsi/initiator/iscsi_subr.c
 Directory Properties:
   stable/9/sys/   (props changed)
 
 Modified: stable/9/sys/dev/iscsi/initiator/iscsi_subr.c
 ==============================================================================
 --- stable/9/sys/dev/iscsi/initiator/iscsi_subr.c	Thu Oct 20 17:33:22 2011	(r226575)
 +++ stable/9/sys/dev/iscsi/initiator/iscsi_subr.c	Thu Oct 20 18:39:01 2011	(r226576)
 @@ -84,6 +84,7 @@ iscsi_r2t(isc_session_t *sp, pduq_t *opq
  	       caddr_t		bp = csio->data_ptr;
  
  	       bo = ntohl(r2t->bo);
 +	       bp += MIN(bo, edtl - ddtl);
  	       bleft = ddtl;
  
  	       if(sp->opt.maxXmitDataSegmentLength > 0) // danny's RFC
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: kib 
State-Changed-When: Thu Oct 20 19:17:26 UTC 2011 
State-Changed-Why:  
stable{8,9} patched. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=160943 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/160943: commit references a PR
Date: Thu, 20 Oct 2011 19:17:07 +0000 (UTC)

 Author: kib
 Date: Thu Oct 20 19:16:52 2011
 New Revision: 226579
 URL: http://svn.freebsd.org/changeset/base/226579
 
 Log:
   MFC r226208:
   Do not ignore block offsets.
   
   PR:	kern/160943
 
 Modified:
   stable/8/sys/dev/iscsi/initiator/iscsi_subr.c
 Directory Properties:
   stable/8/sys/   (props changed)
 
 Modified: stable/8/sys/dev/iscsi/initiator/iscsi_subr.c
 ==============================================================================
 --- stable/8/sys/dev/iscsi/initiator/iscsi_subr.c	Thu Oct 20 18:54:05 2011	(r226578)
 +++ stable/8/sys/dev/iscsi/initiator/iscsi_subr.c	Thu Oct 20 19:16:52 2011	(r226579)
 @@ -84,6 +84,7 @@ iscsi_r2t(isc_session_t *sp, pduq_t *opq
  	       caddr_t		bp = csio->data_ptr;
  
  	       bo = ntohl(r2t->bo);
 +	       bp += MIN(bo, edtl - ddtl);
  	       bleft = ddtl;
  
  	       if(sp->opt.maxXmitDataSegmentLength > 0) // danny's RFC
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: Pavel Timofeev <timp87@gmail.com>
To: bug-followup@FreeBSD.org, cb@severious.net
Cc:  
Subject: Re: kern/160943: [iscsi] [patch] iSCSI initiator ignores block offset
 causing silent data corruption
Date: Fri, 21 Oct 2011 10:28:01 +0400

 --0015174be1f6c69de504afc92cbb
 Content-Type: text/plain; charset=UTF-8
 
 iiihaaa, BIG thanks!
 
 --0015174be1f6c69de504afc92cbb
 Content-Type: text/html; charset=UTF-8
 
 iiihaaa, BIG thanks!<br>
 
 --0015174be1f6c69de504afc92cbb--
>Unformatted:
