From nobody@FreeBSD.org  Tue Sep 22 23:30:57 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D2290106568B
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 22 Sep 2009 23:30:57 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id A7FD28FC26
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 22 Sep 2009 23:30:57 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n8MNUuSL003119
	for <freebsd-gnats-submit@FreeBSD.org>; Tue, 22 Sep 2009 23:30:56 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n8MNUumO003118;
	Tue, 22 Sep 2009 23:30:56 GMT
	(envelope-from nobody)
Message-Id: <200909222330.n8MNUumO003118@www.freebsd.org>
Date: Tue, 22 Sep 2009 23:30:56 GMT
From: Daniel Grund <bugs@dgrund.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: zfs marked as production ready but it used a deprecated checksum algorithm
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         139072
>Category:       kern
>Synopsis:       [zfs] zfs marked as production ready but it used a deprecated checksum algorithm
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    pjd
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Sep 22 23:40:01 UTC 2009
>Closed-Date:    Wed May 05 07:28:24 UTC 2010
>Last-Modified:  Wed May 05 07:28:24 UTC 2010
>Originator:     Daniel Grund
>Release:        8.0-RC1
>Organization:
-
>Environment:
8.0-RC1
>Description:
According to http://bitbucket.org/osunix/osunix-gate/changeset/02e1ddcc9be7/ the default ZFS checksum algorithm in FreeBSD (fletcher2) is deprecated in OpenSolaris because some data losses might be overlooked.
>How-To-Repeat:

>Fix:
fletcher4 should also be the default for FreeBSD 8.0. At least this weakness should be documented for production ready filesystem.

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Wed Sep 23 05:43:28 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139072 
Responsible-Changed-From-To: freebsd-fs->pjd 
Responsible-Changed-By: pjd 
Responsible-Changed-When: ro 23 wrz 2009 09:19:57 UTC 
Responsible-Changed-Why:  
I'll take this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139072 
State-Changed-From-To: open->patched 
State-Changed-By: pjd 
State-Changed-When: ptk 25 wrz 2009 18:20:21 UTC 
State-Changed-Why:  
Default checksum algorithm was changed to fletcher4. 
Note that Solaris (also considered production-ready) still uses fletcher2. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139072 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/139072: commit references a PR
Date: Fri, 25 Sep 2009 18:20:05 +0000 (UTC)

 Author: pjd
 Date: Fri Sep 25 18:19:50 2009
 New Revision: 197497
 URL: http://svn.freebsd.org/changeset/base/197497
 
 Log:
   Switch to fletcher4 as the default checksum algorithm. Fletcher2 was proven to
   be a bit weak and OpenSolaris also switched to fletcher4.
   
   PR:		kern/139072
   Reported by:	Daniel Grund <bugs@dgrund.de>
   MFC after:	3 days
 
 Modified:
   head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/fletcher.c
   head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h
 
 Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/fletcher.c
 ==============================================================================
 --- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/fletcher.c	Fri Sep 25 18:08:38 2009	(r197496)
 +++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/fletcher.c	Fri Sep 25 18:19:50 2009	(r197497)
 @@ -19,11 +19,111 @@
   * CDDL HEADER END
   */
  /*
 - * Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
 + * Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
   * Use is subject to license terms.
   */
  
 -#pragma ident	"%Z%%M%	%I%	%E% SMI"
 +/*
 + * Fletcher Checksums
 + * ------------------
 + *
 + * ZFS's 2nd and 4th order Fletcher checksums are defined by the following
 + * recurrence relations:
 + *
 + *	a  = a    + f
 + *	 i    i-1    i-1
 + *
 + *	b  = b    + a
 + *	 i    i-1    i
 + *
 + *	c  = c    + b		(fletcher-4 only)
 + *	 i    i-1    i
 + *
 + *	d  = d    + c		(fletcher-4 only)
 + *	 i    i-1    i
 + *
 + * Where
 + *	a_0 = b_0 = c_0 = d_0 = 0
 + * and
 + *	f_0 .. f_(n-1) are the input data.
 + *
 + * Using standard techniques, these translate into the following series:
 + *
 + *	     __n_			     __n_
 + *	     \   |			     \   |
 + *	a  =  >     f			b  =  >     i * f
 + *	 n   /___|   n - i		 n   /___|	 n - i
 + *	     i = 1			     i = 1
 + *
 + *
 + *	     __n_			     __n_
 + *	     \   |  i*(i+1)		     \   |  i*(i+1)*(i+2)
 + *	c  =  >     ------- f		d  =  >     ------------- f
 + *	 n   /___|     2     n - i	 n   /___|	  6	   n - i
 + *	     i = 1			     i = 1
 + *
 + * For fletcher-2, the f_is are 64-bit, and [ab]_i are 64-bit accumulators.
 + * Since the additions are done mod (2^64), errors in the high bits may not
 + * be noticed.  For this reason, fletcher-2 is deprecated.
 + *
 + * For fletcher-4, the f_is are 32-bit, and [abcd]_i are 64-bit accumulators.
 + * A conservative estimate of how big the buffer can get before we overflow
 + * can be estimated using f_i = 0xffffffff for all i:
 + *
 + * % bc
 + *  f=2^32-1;d=0; for (i = 1; d<2^64; i++) { d += f*i*(i+1)*(i+2)/6 }; (i-1)*4
 + * 2264
 + *  quit
 + * %
 + *
 + * So blocks of up to 2k will not overflow.  Our largest block size is
 + * 128k, which has 32k 4-byte words, so we can compute the largest possible
 + * accumulators, then divide by 2^64 to figure the max amount of overflow:
 + *
 + * % bc
 + *  a=b=c=d=0; f=2^32-1; for (i=1; i<=32*1024; i++) { a+=f; b+=a; c+=b; d+=c }
 + *  a/2^64;b/2^64;c/2^64;d/2^64
 + * 0
 + * 0
 + * 1365
 + * 11186858
 + *  quit
 + * %
 + *
 + * So a and b cannot overflow.  To make sure each bit of input has some
 + * effect on the contents of c and d, we can look at what the factors of
 + * the coefficients in the equations for c_n and d_n are.  The number of 2s
 + * in the factors determines the lowest set bit in the multiplier.  Running
 + * through the cases for n*(n+1)/2 reveals that the highest power of 2 is
 + * 2^14, and for n*(n+1)*(n+2)/6 it is 2^15.  So while some data may overflow
 + * the 64-bit accumulators, every bit of every f_i effects every accumulator,
 + * even for 128k blocks.
 + *
 + * If we wanted to make a stronger version of fletcher4 (fletcher4c?),
 + * we could do our calculations mod (2^32 - 1) by adding in the carries
 + * periodically, and store the number of carries in the top 32-bits.
 + *
 + * --------------------
 + * Checksum Performance
 + * --------------------
 + *
 + * There are two interesting components to checksum performance: cached and
 + * uncached performance.  With cached data, fletcher-2 is about four times
 + * faster than fletcher-4.  With uncached data, the performance difference is
 + * negligible, since the cost of a cache fill dominates the processing time.
 + * Even though fletcher-4 is slower than fletcher-2, it is still a pretty
 + * efficient pass over the data.
 + *
 + * In normal operation, the data which is being checksummed is in a buffer
 + * which has been filled either by:
 + *
 + *	1. a compression step, which will be mostly cached, or
 + *	2. a bcopy() or copyin(), which will be uncached (because the
 + *	   copy is cache-bypassing).
 + *
 + * For both cached and uncached data, both fletcher checksums are much faster
 + * than sha-256, and slower than 'off', which doesn't touch the data at all.
 + */
  
  #include <sys/types.h>
  #include <sys/sysmacros.h>
 
 Modified: head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h
 ==============================================================================
 --- head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h	Fri Sep 25 18:08:38 2009	(r197496)
 +++ head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/zio.h	Fri Sep 25 18:19:50 2009	(r197497)
 @@ -76,7 +76,7 @@ enum zio_checksum {
  	ZIO_CHECKSUM_FUNCTIONS
  };
  
 -#define	ZIO_CHECKSUM_ON_VALUE	ZIO_CHECKSUM_FLETCHER_2
 +#define	ZIO_CHECKSUM_ON_VALUE	ZIO_CHECKSUM_FLETCHER_4
  #define	ZIO_CHECKSUM_DEFAULT	ZIO_CHECKSUM_ON
  
  enum zio_compress {
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: pjd 
State-Changed-When: śro 5 maj 2010 07:28:04 UTC 
State-Changed-Why:  
Change was merged few months ago. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=139072 
>Unformatted:
