From nobody@FreeBSD.org  Sat Dec 10 18:26:00 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id BDB9516A41F
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 10 Dec 2005 18:26:00 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 559D143D4C
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 10 Dec 2005 18:26:00 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id jBAIQ0Jw049892
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 10 Dec 2005 18:26:00 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id jBAIQ0Rl049891;
	Sat, 10 Dec 2005 18:26:00 GMT
	(envelope-from nobody)
Message-Id: <200512101826.jBAIQ0Rl049891@www.freebsd.org>
Date: Sat, 10 Dec 2005 18:26:00 GMT
From: Simon <ender@tog.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Server reboot after "FAILURE - out of memory in ata_raid_init_request"
X-Send-Pr-Version: www-2.3

>Number:         90206
>Category:       kern
>Synopsis:       [ata] [reboot] Server reboot after "FAILURE - out of memory in ata_raid_init_request"
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Dec 10 18:30:02 GMT 2005
>Closed-Date:    
>Last-Modified:  Fri Jun 19 22:29:13 UTC 2009
>Originator:     Simon
>Release:        6.0-RELEASE
>Organization:
KSD
>Environment:
FreeBSD bnet 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005     ender@bnet:/usr/obj/usr/src/sys/BNET  i386
>Description:
A server daemon I run keeps its database in memory, and after a scheduled
time flushes the dirty database files from memory to harddrives. The
database files are stored as millions of tiny CDB binary files in 1
directory. I use RAID 0+1 on a promise raid card. When the server kicks
off a flush and starts doing a large amount of IO, i get the following
errors in dmesg:

Dec 10 13:51:39 <kern.crit> bnet kernel: DOH! ata_alloc_request failed!
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet last message repeated 4 times
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE g- out of memory i_vn fsat_da_one(raid):_iarni0st_1dre[WquRIestTE(offset=2030
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs
Dec 10 13:51:39 <kern.crit> bnet kernel: _done()FA:aILr0UREs1 -d[ oWRIut oTEf (omeffmoseryt= 2in ata0329_rai03d_94init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_rai62d_4, inleit_rengquthest=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290411008, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel:
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_dFAoInLUe():RE - arou0st 1dof[ mWRemITory E(inoffs aet=t2a_raid0329_init0427392
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE_re(oquffseest=t203290460160, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290509312, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=20329057
Dec 10 13:51:39 <kern.crit> bnet kernel: 4848, lenFAILgtURh=E - 163ou84t )]of error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203290787840, length=16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203675779072, length=16384)]error me= mory5 in ata_r
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: F
Dec 10 13:51:39 <kern.crit> bnet kernel: AILURE - out go_fv fmse_mdoornye (i)n: aart0as_1rda[iWdR_IiTnEi(to_frfesqeute=st
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memory i2n0 3a8t6a8_5r2a0i4d4_8i,n ilte_nrgetqhu=est
Dec 10 13:51:39 <kern.crit> bnet kernel: 16384)]error = 5
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - ougt_ ovff mse_modroy nien (a)ta:_arra0isd1_di[nWiRtI_TrEe(qofufesestt=
Dec 10 13:51:39 <kern.crit> bnet kernel: 203868536832, length=F16A3I84L)U]ReErr o-r  =o u5t
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=203 8o6f8 5m5e3m2o1r6y,  ilne nagttah_=raid_1i6n3i8t
Dec 10 13:51:39 <kern.crit> bnet kernel: FAILURE - out of memorry  i=n  5a
Dec 10 13:51:39 <kern.crit> bnet kernel: ta_raid_init_request
Dec 10 13:51:39 <kern.crit> bnet kernel: g_vfs_done():ar0s1d[WRITE(offset=F2A04I0L61U2R78E2 0-8 ,o ulte nogft hm=emory 1in ata
Dec 10 13:51:39 <kern.crit> bnet kernel:

<snip>

Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=12288)]error = 5
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5
Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5
Dec 10 13:52:02 <kern.crit> bnet kernel: FAILURE - out of memory in ata_raid_init_request
Dec 10 13:52:02 <kern.crit> bnet kernel: g_vfs_done():ar0s1a[WRITE(offset=417742848, length=14336)]error = 5
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

<crash>

Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1992-2005 The FreeBSD Project.
Dec 10 13:53:24 <kern.crit> bnet kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Dec 10 13:53:24 <kern.crit> bnet kernel: The Regents of the University of California. All rights reserved.
Dec 10 13:53:24 <kern.crit> bnet kernel: FreeBSD 6.0-RELEASE #0: Sat Dec 10 02:13:22 UTC 2005

Kernel:
http://ender.tog.net/crash/BNET
dmesg:
http://ender.tog.net/crash/dmesg.today
loader.conf:
http://ender.tog.net/crash/loader.conf
messages:
http://ender.tog.net/crash/messages.0

This problem was also repeated on just RAID 0 array. Unfortunatally I do
not have a kernel dump file at the moment due to the 

bnet# dumpon /dev/ar0
dumpon: ioctl(DIOCSKERNELDUMP): Operation not supported by device

limitation, but hopefully will have another disk in this server to get a
proper kernel dump soon. 
>How-To-Repeat:
Run a daemon that uses a lot of disk IO on a raid array.         
>Fix:
              
>Release-Note:
>Audit-Trail:

From: Alex Trull <alex@trull.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/90206: [ata] [crash] Server reboot after "FAILURE - out
 of memory in ata_raid_init_request"
Date: Sun, 15 Apr 2007 21:30:30 +0100

 I see the opener of this PR no longer has a valid email address. I am 
 experiencing the same symptoms on 6.2-Release amd64 + sil raid1. Can I 
 help at all by providing dumps or access to a developer to my system ?  
 If anyone is interested in solving this bug I would be happy to donate 
 hardware (I checked the request list and no developer has asked for an 
 ata raid1 card + hdds to test this with). This issue does not appear to 
 be universal (atleast, it doesn't appear to be fixed yet so I imagine it 
 isn't universal).
 
 I get these errors from nightly periodic filesystem checks on a system 
 with multiple jails - the IO of each jail performing its periodics 
 within the same few minutes causes the errors to show up in dmesg and an 
 occasional reboot.
 
 Regards,
 
 Alex Trull

From: Travis Mikalson <bofh@terranova.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/90206: [ata] [crash] Server reboot after "FAILURE - out
 of memory in ata_raid_init_request"
Date: Tue, 24 Apr 2007 18:38:28 -0400

 I get these errors periodically (without a crash so far) on my FreeBSD 
 6.2 system still. My server is amd64 but using:
 ar0: 76319MB <nVidia MediaShield RAID1> status: READY
 
 It's a Tyan barebones server (Tyan Transport GT20) with an nvidia nforce 
 pro 3400 chipset whose built-in SATA RAID I am using.
 
 I probably haven't managed a crash yet because my I/O is so light.
 
 So far we have a Silicon Image, a Promise Fasttrak and an nvidia 
 chipset's RAID1 mirroring causing the same problem.
 Indications are that this is a generic problem with FreeBSD 6's software 
 ATA RAID1 stuff.
 
 DOH! ata_alloc_request failed!
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 FAILURE - out of memory in ata_raid_init_request
 g_vfs_done():ar0s1e[WRITE(offset=75324096512, length=16384)]error = 5
 g_vfs_done():ar0s1e[WRITE(offset=75324112896, length=16384)]error = 5
 g_vfs_done():ar0s1e[WRITE(offset=75324129280, length=16384)]error = 5
 g_vfs_done():ar0s1e[WRITE(offset=75324145664, length=16384)]error = 5
 g_vfs_done():ar0s1e[WRITE(offset=75324162048, length=16384)]error = 5
 swap_pager: I/O error - pageout failed; blkno 63550,size 4096, error 5
 swap_pager: I/O error - pageout failed; blkno 78419,size 49152, error 5
 swap_pager: I/O error - pageout failed; blkno 69285,size 12288, error 5
 swap_pager: I/O error - pageout failed; blkno 69875,size 12288, error 5
 
 
 -- 
 TerraNovaNet Internet Services - Key Largo, FL
 Voice: (305)453-4011 x101   Fax: (305)451-5991
 http://www.terranova.net/
 ----------------------------------------------
 Life's not fair, but the root password helps.
>Unformatted:
