From nobody@FreeBSD.org  Fri Sep 19 06:26:17 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id E29D0106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 19 Sep 2008 06:26:17 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id D03408FC0A
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 19 Sep 2008 06:26:17 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.2/8.14.2) with ESMTP id m8J6QHM6020603
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 19 Sep 2008 06:26:17 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.2/8.14.1/Submit) id m8J6QGfp020602;
	Fri, 19 Sep 2008 06:26:16 GMT
	(envelope-from nobody)
Message-Id: <200809190626.m8J6QGfp020602@www.freebsd.org>
Date: Fri, 19 Sep 2008 06:26:16 GMT
From: Alex <titov-av@ptt.spb.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: System hang on ZFS input-output
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         127492
>Category:       kern
>Synopsis:       [zfs] System hang on ZFS input-output
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    pjd
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Sep 19 06:30:00 UTC 2008
>Closed-Date:    
>Last-Modified:  Sun Apr 29 08:40:11 UTC 2012
>Originator:     Alex
>Release:        7.1
>Organization:
JSC "P.T.T."
>Environment:
FreeBSD x6.line1.ru 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #1: Mon Sep 15 11:27:50 MSD 2008     root@x6.line1.ru:/usr/obj/usr/src/sys/KMIST  amd64 
>Description:
We use RAID controller Smart Array P400 (driver ciss)in our system.
Five hardware stripe arrays was combined into raidz (zfs).

#df -h
coolpool         6.7T    1.7T    5.0T    25%    /www

/www - used by proftpd and httpd daemons
System is working properly during some period of time(day-week), but processes which are accessing to the ZFS volume turn into "D  uninterruptible wait" state and hang on.

www      86576  0.0  0.5 121560 19556  ??  D     7:37AM   0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      86577  0.0  0.5 121560 19556  ??  D     7:37AM   0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
www      86578  0.0  0.5 121560 19556  ??  D     7:38AM   0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT

Any requests to the ZFS volume cause the process hang.
"Reboot"-command is executed normally up to the moment when the system produce "syncing disks" and hang also.

P.S. 7.0 - release is working in the same way.

>How-To-Repeat:
We don't know. Possible high disc load.
>Fix:
Hardware reset. O_o

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-amd64->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 18 02:32:51 UTC 2009 
Responsible-Changed-Why:  
probably not amd64-specific. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=127492 
State-Changed-From-To: open->feedback 
State-Changed-By: pjd 
State-Changed-When: pon 14 wrz 2009 13:07:16 UTC 
State-Changed-Why:  
Can you still see this behaviour with FreeBSD 8? 


Responsible-Changed-From-To: freebsd-fs->pjd 
Responsible-Changed-By: pjd 
Responsible-Changed-When: pon 14 wrz 2009 13:07:16 UTC 
Responsible-Changed-Why:  
I'll take this one. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=127492 

From: rainer@ultra-secure.de
To: bug-followup@FreeBSD.org
Cc: titov-av@ptt.spb.ru
Subject: Re: kern/127492: [zfs] System hang on ZFS input-output
Date: Tue, 25 May 2010 03:25:05 +0200

 Hello,
 
 we see the same problem:
 I/O to ZFS filesystems hangs, lots of processes in "D" state (mostly httpd).
 
 We used 8.0, but it still showed in 8.1-PRELEASE (AMD64), HPQ-DL380G5, 8GB
 RAM.
 The system has well over 500 ZFS filesystems.
 
 The same problem showed in an 8-STABLE with a lot less filesystems
 (single-digit, IIRC), but with jails on ZFS (via ezjail).
 
 Can someone give us a hint how one would debug this or give meaningful
 output for the developers?
 
 After a reboot, the server works for some time. It will freeze more often
 at 03:00 AM, but not exclusively.
 
 We are _very_ interested in solving this problem...
 
 
 

From: Chris St Denis <chris@ctgameinfo.com>
To: bug-followup@FreeBSD.org, titov-av@ptt.spb.ru
Cc:  
Subject: Re: kern/127492: [zfs] System hang on ZFS input-output
Date: Mon, 19 Mar 2012 05:33:30 -0700

 I too think I am being affected by this bug. Any process that attempts 
 IO on the affected filesystem ends up stuck in a state of zfsvfs (don't 
 have a PS capture, but D state sound about right)
 
 I submitted a detailed bug report to the FreeNAS bug tracker since that 
 is the distro I am running, but it looks like a kernel bug (probably a 
 dead lock?) so it belongs here.
 
 https://support.freenas.org/ticket/1310
 
 
 If it doesn't sound like the same bug as this I'll open a new PR for it 
 here.

From: Chris St Denis <chris@ctgameinfo.com>
To: bug-followup@FreeBSD.org, titov-av@ptt.spb.ru
Cc:  
Subject: Re: kern/127492: [zfs] System hang on ZFS input-output
Date: Sun, 29 Apr 2012 01:24:33 -0700

 I am willing to provide a kernel dump if it will help resolve this bug, 
 but I can not figure out how to get it to work.
 
 Even with a dump device set, if I force a panic after the io lockup with 
 sysctl debug.kdb.panic or debug.kdb.enter it does not create a dump.
>Unformatted:
