From karl@tickerforum.org  Mon Aug  2 20:20:09 2010
Return-Path: <karl@tickerforum.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id ADC631065676
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  2 Aug 2010 20:20:09 +0000 (UTC)
	(envelope-from karl@tickerforum.org)
Received: from tickerforum.org (tickerforum.org [67.23.181.70])
	by mx1.freebsd.org (Postfix) with ESMTP id 774AD8FC08
	for <FreeBSD-gnats-submit@freebsd.org>; Mon,  2 Aug 2010 20:20:09 +0000 (UTC)
Received: from tickerforum.org (localhost [127.0.0.1])
	by tickerforum.org (8.14.3/8.14.3) with ESMTP id o72JkwM5057403
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 2 Aug 2010 15:46:58 -0400 (EDT)
	(envelope-from karl@tickerforum.org)
Received: (from root@localhost)
	by tickerforum.org (8.14.3/8.14.3/Submit) id o72JkwdW057402;
	Mon, 2 Aug 2010 15:46:58 -0400 (EDT)
	(envelope-from karl)
Message-Id: <201008021946.o72JkwdW057402@tickerforum.org>
Date: Mon, 2 Aug 2010 15:46:58 -0400 (EDT)
From: Karl <karl@denninger.net>
Reply-To: Karl <karl@denninger.net>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: mksnap_ffs hang/deadlock
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         149208
>Category:       kern
>Synopsis:       mksnap_ffs(8) hang/deadlock
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Aug 02 20:30:08 UTC 2010
>Closed-Date:    
>Last-Modified:  Fri Dec  9 05:00:17 UTC 2011
>Originator:     Karl
>Release:        FreeBSD 8.0-STABLE amd64
>Organization:
Cuda Systems LLC
>Environment:
System: FreeBSD tickerforum.org 8.0-STABLE FreeBSD 8.0-STABLE #3: Fri Dec 18 17:41:53 EST 2009 karl@tickerforum.org:/usr/obj/usr/src/sys/GENERIC amd64


>Description:
	Automated nightly backup hung during mksnap_ffs call from dump;
	all I/O to that disk blocked in a "D" state.  System was
	otherwise functional.  There were no errors in the system logs
	or on the console and the raid adapter was fully function.  Raw
	I/O to the disk devices (e.g. reads of the "c" partition of the
	disk that was wedged) remained fully functional during the event.

	Disk adapter is a 3ware 9650; firmware FE9x 4.08.00.006, driver
	3.70.05.001, BBU enabled.  It has no faults recorded against it
	or the drives attached to it.

	Looks like a race condition/deadlock of some sort in the
	mksnap_ffs code.

	The only unique "feature" of this event was that the disk in
	question was a DBMS storage RAID array and had a VERY large
	(many thousands of files) directory on it containing the log
	segments (typically ~1-2MB each, compressed, sequential log
	files numbering many thousand.)  

	System was unable to be cleared; mksnap_ffs could not be killed
	and ultimately the machine had to be forcibly rebooted without a
	sync().


>How-To-Repeat:
	Attempting to replicate at this time.  Will update the PR if
	I am able to do so.  The subject machine had been up for more
	than six months at the time of the fault, and it has not
	reoccurred as of submission of this PR.

	Migration to 8.1 on this cluster is taking place, but I am going
	to put in a significant amount of time in attempting to
	replicate the problem on 8.0-STABLE (same kernel and binaries) 
	on my test cluster to see if I can provoke the failure; I have
	an isolated environment in which I can make a decent attempt at
	recreating the event.

>Fix:

	


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ia64->freebsd-bugs 
Responsible-Changed-By: marcel 
Responsible-Changed-When: Tue Oct 26 06:34:40 UTC 2010 
Responsible-Changed-Why:  
uname(1) output states amd64 as the platform, so ia64 is not the right 
category. Since this is very likely not amd64-specific, change category 
to kern. Update responsible accordingly. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=149208 
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Oct 26 09:36:31 UTC 2010 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=149208 

From: =?windows-1251?B?yu7t/Oru4iDF4uPl7ejp?= <kes-kes@yandex.ru>
To: bug-followup@FreeBSD.org, karl@denninger.net
Cc:  
Subject: Re: kern/149208: mksnap_ffs(8) hang/deadlock
Date: Fri, 9 Dec 2011 06:58:51 +0200

 Hi
 as adviced here
 http://unix.derkeiler.com/Mailing-Lists/FreeBSD/stable/2008-11/msg00210.html
 
 I think you need to add
 > + if (cg % 10 == 0)
 > + ffs_syncvnode(vp, MNT_WAIT);
 in any loop, because of depend on size of disk look can take long
 time.
 
 so:
 --- ffs_snapshot.c.orig 2011-12-09 06:45:11.000000000 +0000
 +++ ffs_snapshot.c      2011-12-09 06:49:43.000000000 +0000
 @@ -305,7 +305,7 @@
         ip->i_flag |= IN_CHANGE | IN_UPDATE;
         error = readblock(vp, bp, numblks - 1);
         bawrite(bp);
 -       if (error != 0)
 +       if (error != 0)    // why not just  if( error ) ???
                 goto out;
         /*
          * Preallocate critical data structures so that we can copy
 @@ -324,6 +324,10 @@
                 if (error)
                         goto out;
                 bawrite(ibp);
 +               if (blkno % 10 == 0)
 +                   ffs_syncvnode(vp, MNT_WAIT);
 +               if (error)           // to be same as line 385
 +                       goto out;
         }
         /*
          * Allocate copies for the superblock and its summary information.
 @@ -341,6 +345,10 @@
                 if (error)
                         goto out;
                 bawrite(nbp);
 +               if (loc % 10 == 0)
 +                   ffs_syncvnode(vp, MNT_WAIT);
 +               if (error)            //to be same as line 385
 +                       goto out;
         }
         /*
          * Allocate all cylinder group blocks.
 @@ -353,6 +361,8 @@
                 bawrite(nbp);
                 if (cg % 10 == 0)
                         ffs_syncvnode(vp, MNT_WAIT);
 +               if (error)           //to be same as line 385
 +                       goto out;.
         }
         /*
          * Copy all the cylinder group maps. Although the
 
 but I not shure. Please confirm
 
 
 -- 
  ,
                            mailto:kes-kes@yandex.ru
 
>Unformatted:
