From nobody@FreeBSD.org  Thu Jan 16 19:09:48 2014
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id 18B86931
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Jan 2014 19:09:48 +0000 (UTC)
Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id DDE701E67
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Jan 2014 19:09:47 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id s0GJ9lon081570
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Jan 2014 19:09:47 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id s0GJ9loJ081566;
	Thu, 16 Jan 2014 19:09:47 GMT
	(envelope-from nobody)
Message-Id: <201401161909.s0GJ9loJ081566@oldred.freebsd.org>
Date: Thu, 16 Jan 2014 19:09:47 GMT
From: John Hickey <jjh-freebsd@deterlab.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Kernel Panic after upgrading NFS server from FreeBSD 9.1 to 9.2
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         185827
>Category:       kern
>Synopsis:       [nfs] [panic] Kernel Panic after upgrading NFS server from FreeBSD 9.1 to 9.2
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jan 16 19:10:00 UTC 2014
>Closed-Date:    Sun Apr 20 12:58:24 UTC 2014
>Last-Modified:  Sun Apr 20 12:58:24 UTC 2014
>Originator:     John Hickey
>Release:        FreeBSD 9.2-RELEASE-p3 #0 r260700: Wed Jan 15 22:17:33 PST 2014     root@:/usr/obj/usr/src/sys/USERS92  i386
>Organization:
USC - ISI / DETER Project
>Environment:
FreeBSD 9.2-RELEASE-p3 #0 r260700: Wed Jan 15 22:17:33 PST 2014     root@:/usr/obj/usr/src/sys/USERS92  i386
>Description:
After upgrading a heavily used NFS server from FreeBSD 9.1 i386 to 9.2 i386, we get a repeatable kernel panic around the time the NFS server starts during boot.  This happened at every boot with the same trace until I went back to FreeBSD 9.1:




panic: stack overflow detected; backtrace may be corrupted
cpuid = 5
KDB: stack backtrace:
#0 0xc0b180ef at kdb_backtrace+0x4f
#1 0xc0adf36f at panic+0x16f
#2 0xc0b0a482 at __stack_chk_fail+0x12
#3 0xc0cc9e83 at fha_assign+0x433
#4 0xc0a09ee0 at fhanew_assign+0x20
#5 0xc0ce2237 at svc_run_internal+0x767
#6 0xc0ce25f0 at svc_thread_start+0x10
#7 0xc0aaad9f at fork_exit+0xcf
#8 0xc0f36664 at fork_trampoline+0x8
Uptime: 7m22s
Physical memory: 3558 MB
Dumping 265 MB: 250 234 218 202 186 170 154 138 122 106 90 74 58 42 26 10

#0  doadump (textdump=1) at pcpu.h:249
249     pcpu.h: No such file or directory.
        in pcpu.h
(kgdb) #0  doadump (textdump=1) at pcpu.h:249
#1  0xc0adf0b5 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:449
#2  0xc0adf3b2 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:637
#3  0xc0b0a482 in __stack_chk_fail ()
    at /usr/src/sys/kern/stack_protector.c:17
#4  0xc0cc9e83 in fha_assign (this_thread=0xcf274500, req=0xcf328000,
    softc=0xc12cd860) at /usr/src/sys/nfs/nfs_fha.c:463
#5  0xc0a09ee0 in fhanew_assign (this_thread=0xcf274500, req=0xcf328000)
    at /usr/src/sys/fs/nfsserver/nfs_fha_new.c:271
#6  0xc0ce2237 in svc_run_internal (pool=0xcf272980, ismaster=0)
    at /usr/src/sys/rpc/svc.c:1109
#7  0xc0ce25f0 in svc_thread_start (arg=0xcf272980)
    at /usr/src/sys/rpc/svc.c:1200
#8  0xc0aaad9f in fork_exit (callout=0xc0ce25e0 <svc_thread_start>,
    arg=0xcf272980, frame=0xe876dd08) at /usr/src/sys/kern/kern_fork.c:992
#9  0xc0f36664 in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:279
(kgdb)

>How-To-Repeat:
I am not sure how to repeat this outside of our environment.  We have a large number to NFS clients served by this machine.  It could have to do with a race condition exposed by such a large number of connections happening at boot time.
>Fix:


>Release-Note:
>Audit-Trail:

From: John Hickey <jjh@deterlab.net>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/185827: Kernel Panic after upgrading NFS server from
 FreeBSD 9.1 to 9.2
Date: Thu, 16 Jan 2014 11:53:21 -0800

 This appears to be the same panic as 184771, which has a low priority.
 This panic is a show stopper for our site and we can reproduce it.
 
 John
State-Changed-From-To: open->open 
State-Changed-By: linimon 
State-Changed-When: Sun Apr 20 01:48:45 UTC 2014 
State-Changed-Why:  
Over to maintainer(s). 


Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Apr 20 01:48:45 UTC 2014 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=185827 
State-Changed-From-To: open->closed 
State-Changed-By: rmacklem 
State-Changed-When: Sun Apr 20 12:55:44 UTC 2014 
State-Changed-Why:  

I'm pretty sure this crash was fixed by r259765, which was 
MFC'd to stable/9 as r261061. It was caused by an NFSv2 mount 
and the stack corruption only seemed to affect i386 systems. 
Thanks go to John for his help with isolating the problem. 

If anyone experiences a similar crash with an up to date stable/9 
or 9.3 (when it is released) system, please submit another PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=185827 
>Unformatted:
