From nobody@FreeBSD.org  Sat Sep 17 20:20:52 2011
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 752B01065670
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 17 Sep 2011 20:20:52 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 642DE8FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 17 Sep 2011 20:20:52 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p8HKKq8b026359
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 17 Sep 2011 20:20:52 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p8HKKqcm026358;
	Sat, 17 Sep 2011 20:20:52 GMT
	(envelope-from nobody)
Message-Id: <201109172020.p8HKKqcm026358@red.freebsd.org>
Date: Sat, 17 Sep 2011 20:20:52 GMT
From: Allen Landsidel <landsidel.allen@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: panic: VPUTX: negative ref count with FUSE
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         160790
>Category:       kern
>Synopsis:       [fusefs] [panic] VPUTX: negative ref count with FUSE
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-fs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Sep 17 20:30:10 UTC 2011
>Closed-Date:    
>Last-Modified:  Wed Jun 26 20:10:01 UTC 2013
>Originator:     Allen Landsidel
>Release:        8.2-STABLE
>Organization:
>Environment:
FreeBSD is01 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu Sep 15 15:30:52 EDT 2011     root@:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
Unexpected panic from nowhere.  Had been storing files with MooseFS at 10-20 mbit/sec for several hours when all connections to machine were lost.  A visit to the console showed a panic.

System did not automatically reboot after the panic, hard reset was required.

May be related to / dupe of http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/156797 though ZFS was not involved.

Full panic is:

vputx: negative ref count
0xffffff01573bf1d8: tag fuse, type VREC
    usecount 0, writecount 0, refcount 0 mountedhere 0
    flags (VI_FREE)
  VI_LOCKed  lock type fuse: UNLOCKED
nodeid: 0, parent_nid: 38125, fh_counter: 0, nlookup: 1, flags: 0
panic: vputx: negative ref cnt
cpuid = 3
KDB: stack backtrace:
#0 0xffffffff805fdda0 at kdb_backtrace+0x60
#1 0xffffffff805cbbd4 at panic_0x1b4
#2 0xffffffff8065e211 at vputx+0x101
#3 0xffffffff8065e40e at vput+0xe
#4 0xffffffff80660c50 at kern_statat_vnhook+0x100
#5 0xffffffff80660d95 at kern_statat+0x15
#6 0xffffffff80660dbc at kern_lstat+0x1c
#7 0xffffffff80660e5d at lstat+0x2d
#8 0xffffffff8060abde at syscallenter+0x2fe
#9 0xffffffff808bd961 at syscall+0x41
#10 0xffffffff808a5ed2 at Xfast_syscall+0xe2
Uptime: 2h47m36s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
>How-To-Repeat:
Exact steps are unknown.  The system had been up running apache and MooseFS (a FUSE based distributed filesystem) for several hours before the panic.  Apache only used to serve images via the mounted FUSE filesystem.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Sep 18 02:46:03 UTC 2011 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=160790 

From: Allen Landsidel <landsidel.allen@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with
 FUSE
Date: Sun, 18 Sep 2011 07:40:36 -0400

 The crash is repeatable though the exact steps are unknown.  Three 
 identical crashes within 8-9 hours, running the same workload.

From: Allen Landsidel <landsidel.allen@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with
 FUSE
Date: Thu, 31 May 2012 15:30:47 -0400

 This bug continues to occur on a new system, different 'hardware'.  The 
 system is running MooseFS and was migrated some time ago to an VMWare 
 ESX 5 VM, 64bit.
 
 OS version is: 8.2-STABLE FreeBSD 8.2-STABLE #0: Fri Oct 14 00:11:10 UTC 
 2011     
 root@freebsd-82-64.master.concord.internal:/usr/obj/usr/src/sys/GENERIC  
 amd64
 
 My initial suspicion is there is some kind of race involved, as the 
 error only seems to occur during times of heavy write load.
 
 The virtualized system does reboot on the panic however, though the 
 physical machine the initial bug report regards did not.
 
 Panic:
 panic: vputx: negative ref cnt
 cpuid = 1
 KDB: stack backtrace:
 #0 0xffffffff80607050 at kdb_backtrace+0x60
 #1 0xffffffff805d4e74 at panic+0x1b4
 #2 0xffffffff80666e31 at vputx+0x101
 #3 0xffffffff8066702e at vput+0xe
 #4 0xffffffff8067206c at vn_open_cred+0x56c
 #5 0xffffffff8067210c at vn_open+0x1c
 #6 0xffffffff8066fc03 at kern_openat+0x163
 #7 0xffffffff8066ff89 at kern_open+0x19
 #8 0xffffffff8066ffa8 at open+0x18
 #9 0xffffffff808c6301 at amd64_syscall+0x301
 #10 0xffffffff808aee0c at Xfast_syscall+0xfc
 panic: bufwrite: buffer is not busy???
 

From: Allen Landsidel <landsidel.allen@gmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/160790: [fusefs] [panic] VPUTX: negative ref count with
 FUSE
Date: Wed, 26 Jun 2013 16:08:39 -0400

 On RELENG_9 (9.1 stable), the crash is near instant now, and does not 
 take hours to manifest.  With a MooseFS FUSE filesystem mounted, simply 
 deleting a single top level directory results in an instant kernel panic.
 
 Compiled with a debug kernel so I could catch it without having to be 
 watching 24/7, I get a panic:
 
 Fatal trap 9: general protection fault while in kernel mode
 cpuid = 1; apic id = 01
 ...
 current process = 904 (rm)
 [ thread pid 904 tid 100074 ]
 Stopped at M_FUSEFH: movb 0xffffffffff8182a5,%al
 
 Screenshots of the panic and backtrace here:
 
 http://i.imgur.com/LHwpdas.png
 http://i.imgur.com/5fpNd7E.png
 http://i.imgur.com/8O2yfV7.png
>Unformatted:
