From nobody@FreeBSD.ORG Fri Mar  5 02:59:07 1999
Return-Path: <nobody@FreeBSD.ORG>
Received: by hub.freebsd.org (Postfix, from userid 32767)
	id 8BB4D14DDC; Fri,  5 Mar 1999 02:59:07 -0800 (PST)
Message-Id: <19990305105907.8BB4D14DDC@hub.freebsd.org>
Date: Fri,  5 Mar 1999 02:59:07 -0800 (PST)
From: hokada@isl.melco.co.jp
Sender: nobody@FreeBSD.ORG
To: freebsd-gnats-submit@freebsd.org
Subject: Tagged queueing makes IBM DCAS-34330 slow
X-Send-Pr-Version: www-1.0

>Number:         10398
>Category:       kern
>Synopsis:       Tagged queueing makes IBM DCAS-34330 slow
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    ken
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Fri Mar  5 03:00:01 PST 1999
>Closed-Date:    Sat Mar 13 21:19:06 PST 1999
>Last-Modified:  Sat Mar 13 21:20:22 PST 1999
>Originator:     Hideaki Okada
>Release:        3.1-RELEASE
>Organization:
MITSUBISHI Electric
>Environment:
FreeBSD waikiki.isl.melco.co.jp 3.1-RELEASE FreeBSD 3.1-RELEASE #5: Sat Feb 20 01:25:27 JST 1999     hokada@waikiki.isl.melco.co.jp:/usr/src/sys/compile/WAIKIKI  i386

>Description:
My IBM DCAS-34330 hard drive is connected to Adaptec AHA-2940 adapter,
and I found write performance is low.
Simple throughput measurement like 'dd if=/dev/zero of=test bs=64k count=1000'
shows ahout 3MB/s even though it should be 7-8MB/s .
However, there is no trouble on read performance.
Here is bonnie's results:

               -------Sequential Output-------- ---Sequential Input-- --Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
2.2.8R    100  6677 83.1  7542 22.8  3313 11.3  8519 93.4  7813 14.0 250.0  4.9
3.1R      100  7269 90.7  3199 10.5  2283  8.7  7450 95.7  7548 15.0 262.5  5.4
3.1R(*)   100  7376 91.3  6823 21.5  2628  9.3  6931 88.7  7469 14.3 240.6  4.7

(*) tagged queuing is turned off by modifying cam_xpt.c

>How-To-Repeat:
I reported this on FreeBSD-users-jp@jp.freebsd.org mailing list,
and the other report on this problem is posted.
Its configuration is:
    IBM DCAS-34330W connected to AHA-2940UW

>Fix:
I put initializer of xpt_quirk_table at cam_xpt.c, and I got bonnie's result as reported above.

             {
                     /*
                      * Slow when tagged queueing is enabled. (3MB/sec versus
                      * 8MB/sec.)
                      */
                     { T_DIRECT, SIP_MEDIA_FIXED, "IBM", "DCAS-34330", "S65A" },
                     /*quirks*/0, /*mintags*/0, /*maxtags*/0
             },

>Release-Note:
>Audit-Trail:

From: "Kenneth D. Merry" <ken@plutotech.com>
To: hokada@isl.melco.co.jp
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10398: Tagged queueing makes IBM DCAS-34330 slow
Date: Fri, 5 Mar 1999 11:53:43 -0700 (MST)

 hokada@isl.melco.co.jp wrote...
 > 
 > >Number:         10398
 > >Category:       kern
 > >Synopsis:       Tagged queueing makes IBM DCAS-34330 slow
 > >Confidential:   no
 > >Severity:       serious
 > >Priority:       medium
 > >Responsible:    freebsd-bugs
 > >State:          open
 > >Quarter:        
 > >Keywords:       
 > >Date-Required:
 > >Class:          change-request
 > >Submitter-Id:   current-users
 > >Arrival-Date:   Fri Mar  5 03:00:01 PST 1999
 > >Closed-Date:
 > >Last-Modified:
 > >Originator:     Hideaki Okada
 > >Release:        3.1-RELEASE
 > >Organization:
 > MITSUBISHI Electric
 > >Environment:
 > FreeBSD waikiki.isl.melco.co.jp 3.1-RELEASE FreeBSD 3.1-RELEASE #5: Sat Feb 20 01:25:27 JST 1999     hokada@waikiki.isl.melco.co.jp:/usr/src/sys/compile/WAIKIKI  i386
 > 
 > >Description:
 > My IBM DCAS-34330 hard drive is connected to Adaptec AHA-2940 adapter,
 > and I found write performance is low.
 > Simple throughput measurement like 'dd if=/dev/zero of=test bs=64k count=1000'
 > shows ahout 3MB/s even though it should be 7-8MB/s .
 > However, there is no trouble on read performance.
 > Here is bonnie's results:
 > 
 >                -------Sequential Output-------- ---Sequential Input-- --Random--
 >                -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
 > Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
 > 2.2.8R    100  6677 83.1  7542 22.8  3313 11.3  8519 93.4  7813 14.0 250.0  4.9
 > 3.1R      100  7269 90.7  3199 10.5  2283  8.7  7450 95.7  7548 15.0 262.5  5.4
 > 3.1R(*)   100  7376 91.3  6823 21.5  2628  9.3  6931 88.7  7469 14.3 240.6  4.7
 > 
 > (*) tagged queuing is turned off by modifying cam_xpt.c
 > 
 > >How-To-Repeat:
 > I reported this on FreeBSD-users-jp@jp.freebsd.org mailing list,
 > and the other report on this problem is posted.
 > Its configuration is:
 >     IBM DCAS-34330W connected to AHA-2940UW
 > 
 > >Fix:
 > I put initializer of xpt_quirk_table at cam_xpt.c, and I got bonnie's result as reported above.
 > 
 >              {
 >                      /*
 >                       * Slow when tagged queueing is enabled. (3MB/sec versus
 >                       * 8MB/sec.)
 >                       */
 >                      { T_DIRECT, SIP_MEDIA_FIXED, "IBM", "DCAS-34330", "S65A" },
 >                      /*quirks*/0, /*mintags*/0, /*maxtags*/0
 >              },
 
 
 Thanks for reporting this.
 
 I would like to get some more information on the problem, though, before
 disabling tagged queueing altogether for this drive.  I have found that
 some drives behave differently depending on whether or not write caching is
 turned on.  Also, it may be that we don't have to disable tagged queueing
 completely, but just limit the number of tags to some low number.
 
 The Seagate Medalist Pro is a good example of that.  It has horrible write
 performance when the number of tags is >= 3.  But, at 2 tags or just 1
 (i.e. tagged queueing disabled), its write performance is just fine.
 
 So, the first thing to try is your bonnie test above, with write caching
 enabled and with write caching disabled.
 
 To see if write caching is enabled or disabled, do something like this:
 
 camcontrol modepage -n da -u 4 -v -m 8
 
 Look at the 'WCE' (Write Cache Enable) bit.  If it is 1, write caching is
 enabled.  If it is 0, write caching is disabled.  To change the value:
 
 camcontrol modepage -n da -u 4 -v -m 8 -e -P 3
 
 Obviously, if your disk is not da4, you'll need to change the -u argument
 above to the unit number for your disk.
 
 The next thing to try is to change the number of tags on the fly and
 try to determine whether there is a point where performance drops off
 completely.
 
 I will send you some patches to camcontrol in private email that will allow
 you to change the number of tags on the fly, and disable tagged queueing
 altogether.
 
 Ken
 -- 
 Kenneth Merry
 ken@plutotech.com
 

From: Hideaki Okada <hokada@isl.melco.co.jp>
To: "Kenneth D. Merry" <ken@plutotech.com>
Cc: freebsd-gnats-submit@FreeBSD.ORG, hokada@isl.melco.co.jp
Subject: Re: kern/10398: Tagged queueing makes IBM DCAS-34330 slow 
Date: Wed, 10 Mar 1999 10:48:21 +0900

 Ken,
 
 Thanks for your support.
 
 > So, the first thing to try is your bonnie test above, with write caching
 > enabled and with write caching disabled.
 
 > The next thing to try is to change the number of tags on the fly and
 > try to determine whether there is a point where performance drops off
 > completely.
 
 Here are results of bonnie, measured configurations are:
 	WCE=1, tags=1, 2, 3, 4, 8, 16, 32, 64
 	WCE=0, tags=1,       4,    16,     64
 
               -------Sequential Output-------- ---Sequential Input-- --Random--
               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
            MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
 CONFIGURATION
 Number of Tags
 write cache enabled
 NO        100  7222 89.2  6801 21.8  2347  8.5  7330 93.3  7368 14.6 226.5  4.7
 2         100  7263 90.3  6357 20.3  2730  9.9  7025 90.5  7321 14.9 209.4  4.6
 3         100  7115 88.1  6406 20.8  2289  8.9  7307 93.9  7335 15.0 212.6  4.5
 4         100  7281 90.0  6204 20.8  2278  9.1  7267 93.7  7350 15.3 217.6  4.8
 8         100  7236 89.7  6007 19.4  2284  8.7  7239 93.1  7374 14.9 213.4  4.5
 16        100  6775 83.7  6110 19.5  2283  8.7  7239 93.1  7380 14.8 217.8  4.8
 32        100  7265 89.9  4385 13.9  2274  8.7  7302 93.7  7324 14.5 218.9  4.5
 64        100  6731 83.3  3038  9.8  2271  8.7  7337 94.6  7356 14.7 213.6  4.4
 
 write cache disabled
 NO        100  1638 19.9  1268  3.9  1960  7.1  7377 94.2  7356 14.3 218.4  4.4
 4         100  5032 62.0  3559 11.5  2277  8.9  7262 93.6  7374 15.1 216.8  4.6
 16        100  5705 70.3  4599 14.6  2255  8.8  7290 93.5  7374 15.1 209.5  4.6
 64        100  2847 35.0  2752  8.8  2261  8.6  7360 94.0  7374 14.7 215.3  4.6
 
 Because it is easier to read how configuration affects,
 attached results of dd if=/dev/zero of=foo bs=64k count=1000
 
 (WCE=1)
 tags bytes/sec
 NO  7258435
 2   6593304
 4   6551596
 8   6119427
 16  5853919
 32  5200000
 64  3048527
 
 I hope this mail has enough information for you.
 
 thanks,
 Hideaki Okada
 

From: "Kenneth D. Merry" <ken@plutotech.com>
To: hokada@isl.melco.co.jp (Hideaki Okada)
Cc: freebsd-gnats-submit@FreeBSD.ORG
Subject: Re: kern/10398: Tagged queueing makes IBM DCAS-34330 slow
Date: Thu, 11 Mar 1999 14:50:22 -0700 (MST)

 Hideaki Okada wrote...
 > 
 > Ken,
 > 
 > Thanks for your support.
 
 Sorry I took so long to respond.
 
 > > So, the first thing to try is your bonnie test above, with write caching
 > > enabled and with write caching disabled.
 > 
 > > The next thing to try is to change the number of tags on the fly and
 > > try to determine whether there is a point where performance drops off
 > > completely.
 > 
 > Here are results of bonnie, measured configurations are:
 > 	WCE=1, tags=1, 2, 3, 4, 8, 16, 32, 64
 > 	WCE=0, tags=1,       4,    16,     64
 > 
 >               -------Sequential Output-------- ---Sequential Input-- --Random--
 >               -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
 >            MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec %CPU
 > CONFIGURATION
 > Number of Tags
 > write cache enabled
 > NO        100  7222 89.2  6801 21.8  2347  8.5  7330 93.3  7368 14.6 226.5  4.7
 > 2         100  7263 90.3  6357 20.3  2730  9.9  7025 90.5  7321 14.9 209.4  4.6
 > 3         100  7115 88.1  6406 20.8  2289  8.9  7307 93.9  7335 15.0 212.6  4.5
 
 Yeah, it looks like there is a reasonable dropoff in write performance from
 1 transaction to two.  It also looks like turning off write caching is a
 very bad thing to do with that drive.
 
 I'll put in a quirk entry to disable tagged queueing.
 
 Ken
 -- 
 Kenneth Merry
 ken@plutotech.com
 
Responsible-Changed-From-To: freebsd-bugs->ken 
Responsible-Changed-By: ken 
Responsible-Changed-When: Thu Mar 11 14:12:50 PST 1999 
Responsible-Changed-Why:  
To remind me to commit the quirk entry. 
State-Changed-From-To: open->closed 
State-Changed-By: ken 
State-Changed-When: Sat Mar 13 21:19:06 PST 1999 
State-Changed-Why:  
Quirk entry checked into -current, revision 1.49 of cam_xpt.c.  Checked into 
RELENG_3, revision 1.42.2.5 of cam_xpt.c. 
Thanks for your problem report and thanks for spending the time to profile 
the performance of this drive. 
>Unformatted:
