From freaky@voi.aagh.net  Mon Jul  9 12:39:03 2007
Return-Path: <freaky@voi.aagh.net>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 6289416A421
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  9 Jul 2007 12:39:03 +0000 (UTC)
	(envelope-from freaky@voi.aagh.net)
Received: from spork.qfe3.net (spork.qfe3.net [212.13.207.101])
	by mx1.freebsd.org (Postfix) with ESMTP id 2D23613C455
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  9 Jul 2007 12:39:03 +0000 (UTC)
	(envelope-from freaky@voi.aagh.net)
Received: from [81.104.144.87] (helo=voi.aagh.net)
	by spork.qfe3.net with esmtp (Exim 4.66 (FreeBSD))
	(envelope-from <freaky@voi.aagh.net>)
	id 1I7s9E-000Imz-FF
	for FreeBSD-gnats-submit@freebsd.org; Mon, 09 Jul 2007 13:15:24 +0100
Received: from freaky by voi.aagh.net with local (Exim 4.67 (FreeBSD))
	(envelope-from <freaky@voi.aagh.net>)
	id 1I7s9E-0009fT-7n
	for FreeBSD-gnats-submit@freebsd.org; Mon, 09 Jul 2007 13:15:24 +0100
Message-Id: <E1I7s9E-0009fT-7n@voi.aagh.net>
Date: Mon, 09 Jul 2007 13:15:24 +0100
From: Thomas Hurst <tom@hur.st>
Sender: Thomas Hurst <freaky@voi.aagh.net>
Reply-To: Thomas Hurst <tom@hur.st>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Anomalous performance with multiple arrays and amr(4)
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         114438
>Category:       kern
>Synopsis:       [amr] Anomalous performance with multiple arrays and amr(4)
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jul 09 12:40:04 GMT 2007
>Closed-Date:    
>Last-Modified:  Wed Jan 16 19:00:05 UTC 2008
>Originator:     Thomas Hurst
>Release:        FreeBSD 6.2-STABLE amd64
>Organization:
>Environment:
System: FreeBSD voi.nightsdawn.sf 6.2-STABLE FreeBSD 6.2-STABLE #0: Wed Jul 4 13:46:52 BST 2007 root@voi.nightsdawn.sf:/usr/obj/usr/src/sys/VOI amd64

8GB dual dual core Opteron

amr0: <LSILogic LSI MegaRAID SATA300-8X PCI-X> Firmware 814B, BIOS H431, 128MB RAM
amrd0: 190713MB (390580224 sectors) RAID 1 (optimal)
amrd1: 381485MB (781281280 sectors) RAID 1 (optimal)
amrd2: 381485MB (781281280 sectors) RAID 1 (optimal)
>Description:
Performing heavy random IO on any one array causes other arrays to
stall intermitently.  For example:

find /usr >/dev/null & # on amrd0
dd if=/dev/amrd1 of=/dev/null bs=128k

gstat shows amrd0 doing a few hundred ops/sec and up to a few MB/s
amrd1 runs at the expected 65MB/s for ~10 seconds, then stops for 2-3,
before returning to 65MB/s.

This behavior is seen with lighter serial transfers; I normally notice it
while streaming movies or music at 100-1000k/s and daily run kicks in,
causing annoying pauses in playback without the use of fairly large buffers.

Multiple serial transfers on different arrays do not have the same problem;
I can dd from or to all arrays without unexplained pauses.

ata(4) disks on the system are unaffected.
>How-To-Repeat:
See above.
>Fix:




>Release-Note:
>Audit-Trail:

From: Thomas Hurst <tom@hur.st>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/114438: [amr] Anomalous performance with multiple arrays
	and amr(4)
Date: Wed, 16 Jan 2008 18:33:55 +0000

 Further investigation would seem to implicate atime updates; every time
 reads from one array freeze, the other array has an IO queue depth (as
 reported by gstat) of ~1000 and is performing a bunch of writes.
 
 Ignoring the meta-issue of all those atime updates, it would seem the
 large work queue to one array on the card starves out other arrays.
 
 My recent tests for this have been on 7.0-RC1 as of today.
 
 -- 
 Thomas 'Freaky' Hurst
     http://hur.st/
>Unformatted:
