From nobody@FreeBSD.org  Mon Apr  2 22:02:19 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 7EF9516A406
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  2 Apr 2007 22:02:19 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 6FD6A13C484
	for <freebsd-gnats-submit@FreeBSD.org>; Mon,  2 Apr 2007 22:02:19 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l32M2JNh055319
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 2 Apr 2007 22:02:19 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l32LvHHL054246;
	Mon, 2 Apr 2007 21:57:17 GMT
	(envelope-from nobody)
Message-Id: <200704022157.l32LvHHL054246@www.freebsd.org>
Date: Mon, 2 Apr 2007 21:57:17 GMT
From: George Breahna<george@top-consulting.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Kernel panic caused by UFS
X-Send-Pr-Version: www-3.0

>Number:         111156
>Category:       kern
>Synopsis:       [ufs] Kernel panic caused by UFS
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 02 22:10:04 GMT 2007
>Closed-Date:    Tue Jun 12 08:34:03 GMT 2007
>Last-Modified:  Tue Jun 12 08:34:03 GMT 2007
>Originator:     George Breahna
>Release:        6.1-RELEASE-p12
>Organization:
G-Tech IT Consulting
>Environment:
FreeBSD store016.1-RELEASE-p12 FreeBSD 6.1-RELEASE-p12 #0: Mon Jan 22 11:20:03 UTC 2007     pulsar@:/usr/src/sys/i386/compile/GTECH  i386

>Description:
The machine contains a 3ware 9550SX-4LP card. It has attached to it 4 x
500GB disks in RAID 10 config. Output from 3ware CLI is all normal, RAID
ok, drives OK. The drives and the 3ware card have been updated to the
latest firmware available from manufacturer.

This machine is mainly used for backups of an e-mail server.

Within 1 hour of starting the backups, the following error appears:

Mar 31 00:29:05 store01 syslogd: kernel boot file is /boot/kernel/kernel
Mar 31 00:29:05 store01 kernel: start = 0, len = 11431, fs = /usr
Mar 31 00:29:05 store01 kernel: panic: ffs_alloccg: map corrupted
Mar 31 00:29:05 store01 kernel: Uptime: 2d22h57m30s
Mar 31 00:29:05 store01 kernel: Cannot dump. No dump device defined.
Mar 31 00:29:05 store01 kernel: Automatic reboot in 15 seconds - press a key on the console to abort


The first time it happened I rebooted in single user mode and rand a
fsck -y on all partitions. It didn't encounter any errors except for the
following type:

fsck: /dev/da0s1f: INCORRECT BLOCK COUNT I=28342929 (8 should be 0) (CORRECTED)

After that the system reboots fine, works excellent until the backups are
started again. I should mention that the server ran fine for up to 22
days, sitting idle. The day I put it into production these issues appeared.

I did a bit of research and the only other case where I found a very
similar problem is with NetBSD, documented here:

http://www.nabble.com/kern-35580%3A-creating-directories-on-large-UFS2--%3E-boom-tf3203420.html#a8895438


Some output from df:
Filesystem  1K-blocks      Used     Avail Capacity  iused     ifree %iused  Mounted on
/dev/da0s1a    988398    122102    787226    13%     4425    136885    3%   /
devfs               1         1         0   100%        0         0  100%   /dev
/dev/da0s1d    988398        60    909268     0%       25    141285    0%   /tmp
/dev/da0s1f 940800816 508178166 357358586    59% 15881895 105717079   13%   /usr
/dev/da0s1e    988398    452108    457220    50%     2506    138804    2%   /var







>How-To-Repeat:
Create many directories/files on the file system.
>Fix:

>Release-Note:
>Audit-Trail:

From: Kris Kennaway <kris@obsecurity.org>
To: George Breahna <george@top-consulting.net>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/111156: Kernel panic caused by UFS
Date: Mon, 2 Apr 2007 19:50:37 -0400

 Please follow the directions in the developers handbook on kernel
 debugging and obtain a crashdump and debugging backtrace.  This
 information is required to complete your PR before a developer can
 proceed with it.
 
 Thanks,
 Kris

From: Astrodog <astrodog@gmail.com>
To: bug-followup@FreeBSD.org, george@top-consulting.net
Cc:  
Subject: Re: misc/111156: Kernel panic caused by UFS
Date: Wed, 4 Apr 2007 08:21:27 -0500

 Tested with 9550SX controller I have locally, unable to reproduce. If you
 follow Kris's instructions, we can probably track this problem down fairly
 quickly
 
 --- Harrison

From: "George Breahna" <george@top-consulting.net>
To: "'Astrodog'" <astrodog@gmail.com>,
	<bug-followup@FreeBSD.org>
Subject: RE: misc/111156: Kernel panic caused by UFS
Date: Thu, 5 Apr 2007 12:41:12 -0400

 Alrigt, I will try and reproduce the bug this weekend. When I had the
 backups running nightly it happened twice in two consecutive days with the
 same error.
 
 I enabled dumpdev and will provide a backtrace this weekend.
 
 George
 
 Senior Engineer
 G-Tech Consulting
 1-888-232-2378 x 111
 http://www.top-consulting.net
 
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Sun Apr 22 07:51:24 UTC 2007 
State-Changed-Why:  
Note that the submitter was asked for feedback. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=111156 
State-Changed-From-To: feedback->closed 
State-Changed-By: linimon 
State-Changed-When: Tue Jun 12 08:33:28 UTC 2007 
State-Changed-Why:  
Feedback timeout (1 month).  To submitter: if you can still reproduce 
this, we can reopen this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=111156 
>Unformatted:
