From john.marshall@riverwillow.com.au  Sat Oct  5 06:54:24 2013
Return-Path: <john.marshall@riverwillow.com.au>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTP id 43C35EC
	for <FreeBSD-gnats-submit@freebsd.org>; Sat,  5 Oct 2013 06:54:24 +0000 (UTC)
	(envelope-from john.marshall@riverwillow.com.au)
Received: from mail1.riverwillow.net.au (mail1.riverwillow.net.au [IPv6:2001:8000:1000:1801::36])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id C3FFE2FC6
	for <FreeBSD-gnats-submit@freebsd.org>; Sat,  5 Oct 2013 06:54:22 +0000 (UTC)
Received: from rwsrv08.gfn.riverwillow.net.au (localhost [IPv6:::1])
	by mail1.riverwillow.net.au (8.14.7/8.14.7) with ESMTP id r956rw9a017433
	for <FreeBSD-gnats-submit@freebsd.org>; Sat, 5 Oct 2013 16:53:58 +1000 (AEST)
Received: (from john@localhost)
	by rwsrv08.gfn.riverwillow.net.au (8.14.7/8.14.7/Submit) id r956rwl5017432;
	Sat, 5 Oct 2013 16:53:58 +1000 (AEST)
Message-Id: <201310050653.r956rwl5017432@rwsrv08.gfn.riverwillow.net.au>
Date: Sat, 5 Oct 2013 16:53:58 +1000 (AEST)
From: John Marshall <john.marshall@riverwillow.com.au>
Reply-To: John Marshall <john.marshall@riverwillow.com.au>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: [PANIC] 9.2-RELEASE vm_page_unwire: page n's wire count is zero
X-Send-Pr-Version: 3.114
X-GNATS-Notify:

>Number:         182661
>Category:       kern
>Synopsis:       [PANIC] 9.2-RELEASE vm_page_unwire: page n's wire count is zero
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Oct 05 07:00:00 UTC 2013
>Closed-Date:    Mon Nov 11 01:06:11 UTC 2013
>Last-Modified:  Mon Nov 11 01:06:11 UTC 2013
>Originator:     John Marshall
>Release:        FreeBSD 9.2-RELEASE amd64
>Organization:
Riverwillow Pty Ltd
>Environment:

System: FreeBSD ozsrv04.riverwillow.net.au 9.2-RELEASE FreeBSD 9.2-RELEASE #1 r255982: Wed Oct  2 15:03:22 AEST 2013     root@ozsrv04.riverwillow.net.au:/usr/obj/kits/src/sys/OZSRV04  amd64

System: FreeBSD rwsrv08.gfn.riverwillow.net.au 9.2-STABLE FreeBSD 9.2-STABLE #0 r256029: Fri Oct 4 07:22:42 AEST 2013 root@rwsrv08.gfn.riverwillow.net.au:/usr/obj/usr/src/sys/RWSRV08 amd64

	
>Description:

Since 9.2-RC4 I have observed this panic many times on each of two
servers running FreeBSD/amd64.  The panic usually occurs during system
shutdown when either ntpd or watchdogd are stopping.  Sometimes the
panic will occur when stopping either of those daemons independently of
a system shutdown.  I also saw this panic - once only - while the sftp
server was active accepting transfer of large (GB) files over the LAN.

I have about 30 dumps (from two systems) that look like the following.
Entry points and offsets are always identical.  Backtraces matching the
following account for all but one of the panics which I have seen, and
all appear to be triggered by ntpd or watchdogd exiting either while the
system is running or, usually, during shutdown.  These 30+ panics were
observed on 9.2-RC4, 9.2-RELEASE and 9.2-STABLE.  I took one of the
systems back to a fresh releng/9.1 (9.1-RELEASE-p7) and was unable to
trigger the panic.

  panic: vm_page_unwire: page <n>'s wire count is zero
  cpuid = <n> <-- anything from 0-7
  KDB: stack backtrace:
  #0 <n> at kdb_backtrace+0x68
  #1 <n> at panic+0x21a
  #2 <n> at vm_page_unwire+0x102
  #3 <n> at vm_fault_unwire+0xd2
  #4 <n> at vm_map_delete+0x171
  #5 <n> at vm_map_remove+0x5f
  #6 <n> at vmspace_exit+0xc9
  #7 <n> at exit1+0x72d
  #8 <n> at sys_sys_exit+0xe
  #9 <n> at amd64_syscall+0x3bf
  #10 <n> at Xfast_syscall+0xf7

The following is the exception.  This was when a 9.2-RC4 system panicked
with sftp server active.
 
  panic: vm_page_unwire: page 0xfffffe0239ecfc48's wire count is zero
  cpuid = 5
  KDB: stack backtrace:
  #0 0xffffffff80490268 at kdb_backtrace+0x68
  #1 0xffffffff8045630a at panic+0x21a
  #2 0xffffffff8068bbc2 at vm_page_unwire+0x102
  #3 0xffffffff804d857e at vfs_vmio_release+0x10e
  #4 0xffffffff804dadc8 at getnewbuf+0x468
  #5 0xffffffff804dbb2f at getblk+0x5df
  #6 0xffffffff80632bb1 at ffs_balloc_ufs2+0x15c1
  #7 0xffffffff8065a8d6 at ffs_write+0x246
  #8 0xffffffff8070a18f at VOP_WRITE_APV+0x11f
  #9 0xffffffff80505e77 at vn_write+0x1f7
  #10 0xffffffff80503b51 at vn_io_fault+0xb1
  #11 0xffffffff804a387c at dofilewrite+0x9c
  #12 0xffffffff804a3b54 at kern_writev+0x54
  #13 0xffffffff804a3bf5 at sys_write+0x65
  #14 0xffffffff806afd6f at amd64_syscall+0x3bf
  #15 0xffffffff8069a817 at Xfast_syscall+0xf7

See this thread on the -stable mailing list for more information.
http://lists.freebsd.org/pipermail/freebsd-stable/2013-September/075317.html

Please contact me for access to core.txt files or for extracting
information from dumps.  I still have most of the dumps and some of the
corresponding kernel.symbols files.

>How-To-Repeat:

I have only seen this FreeBSD/amd64 machines with 8 logical CPU's.  I
have not seen this on i386 (but none of my i386 systems have more than 2
logical CPU's).

 - Start ntpd
 - Stop ntpd (panics sometimes)

 - Start ntpd
 - Shutdown the system
 - Often panics when ntpd exiting
   
   Stopping ntpd.
   Waiting for PIDS: 1430
   panic: vm_page_unwire: page 0xfffffe0239144f38's wire count is zero
   cpuid = 7
   KDB: stack backtrace:
   #0 0xffffffff80490268 at kdb_backtrace+0x68
   #1 0xffffffff8045630a at panic+0x21a
   #2 0xffffffff8068bbc2 at vm_page_unwire+0x102
   #3 0xffffffff80678702 at vm_fault_unwire+0xd2
   (etc.)

>Fix:

	


>Release-Note:
>Audit-Trail:

From: John Marshall <john.marshall@riverwillow.com.au>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/182661: [PANIC] 9.2-RELEASE vm_page_unwire: page n's wire
 count is zero
Date: Tue, 8 Oct 2013 18:47:36 +1100

 At the request of kib@ I tested with stable/9 and head and found the
 panic introduced in the following commits.
 
   stable/9  r244654
   head      r240285
 
 -- 
 John Marshall

From: John Marshall <john.marshall@riverwillow.com.au>
To: FreeBSD-gnats-submit@FreeBSD.org
Cc: kostikbel@gmail.com, peter@holm.cc
Subject: Re: kern/182661: [PANIC] 9.2-RELEASE vm_page_unwire: page n's wire
 count is zero
Date: Thu, 17 Oct 2013 17:35:40 +1100

 pho@ confirmed that the presence of nullfs mounts was a necessary
 ingredient in this panic.  A patch was produced and committed to head by
 kib@ as r256211.  I applied the same patch to releng/9.2 and have had
 the patched 9.2 running on two production servers for over a week now.
 All looks good.  The patch has now been MFC'd to stable/9 by kib@ as
 r256590.
 
 Unless there's a good reason to keep this open, I think this PR could
 now be closed.
 
 Thank you kib@ and pho@ for your work on this.
 
 -- 
 John Marshall
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Mon Nov 11 01:05:43 UTC 2013 
State-Changed-Why:  
patch has now been MFC'd to stable/9. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=182661 
>Unformatted:
