From nobody@FreeBSD.org  Sat Jul 14 03:34:11 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 7BBAB106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 14 Jul 2012 03:34:11 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 6848E8FC17
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 14 Jul 2012 03:34:11 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q6E3YBIg007879
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 14 Jul 2012 03:34:11 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q6E3YBH3007878;
	Sat, 14 Jul 2012 03:34:11 GMT
	(envelope-from nobody)
Message-Id: <201207140334.q6E3YBH3007878@red.freebsd.org>
Date: Sat, 14 Jul 2012 03:34:11 GMT
From: Tig On <tigger@lvlworld.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: spin lock held too long
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         169838
>Category:       kern
>Synopsis:       [kernel] spin lock held too long
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Jul 14 03:40:03 UTC 2012
>Closed-Date:    
>Last-Modified:  Mon Jul 16 04:33:42 UTC 2012
>Originator:     Tig On
>Release:        FreeBSD 8.3
>Organization:
>Environment:
FreeBSD tiger.lilypie.com 8.3-RELEASE-p3 FreeBSD 8.3-RELEASE-p3 #10: Wed Jul  4 14:33:44 EST 2012     tigger@tiger.lilypie.com:/usr/obj/usr/src/sys/TIGER  i386
>Description:
Once a week, for many years now a back-up sh script runs over the two SCSI drives in 6 similar, but slightly different servers.

The script is very heavy on the drives and will create many tar files with many more small files in each tar. In total about 6.5 million small files across all servers will be tar'd up at the end.

Two weeks ago, the servers were upgraded from 8.2 to 8.3. On the first back-up 4 of the 6 servers went down.

This week, so far only one has gone down. The message on the console is:

spin lock 0xc0cb94b4 (smp rendezvous) held by 0xccaf78a0 (tid 100986) too long
panic: spin lock held too long
cpuid = 3

Debug options are disable on the server, in the kernel conf:
#makeoptions    DEBUG=-g
#options        KDTRACE_HOOKS           # Kernel DTrace hooks
#options        KDB                     # Kernel debugger related code

What other info can I share to help?
>How-To-Repeat:
Sadly, not a completely repeatable issue.

The first crash happens at about 6 hours into the back-up. The 4th crash (last week) happened at about the 24 hour point on the back-up (which takes from 30 to 38 hours).

More than happy to try anything.
>Fix:
Wish I knew :]

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-i386->freebsd-bugs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jul 16 04:33:15 UTC 2012 
Responsible-Changed-Why:  
reclassify. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=169838 
>Unformatted:
