From nobody@FreeBSD.org  Mon Apr 29 05:31:18 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 69716C2A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 29 Apr 2013 05:31:18 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [69.147.83.34])
	by mx1.freebsd.org (Postfix) with ESMTP id 413081628
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 29 Apr 2013 05:31:18 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r3T5VIPo023399
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 29 Apr 2013 05:31:18 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r3T5VIbu023396;
	Mon, 29 Apr 2013 05:31:18 GMT
	(envelope-from nobody)
Message-Id: <201304290531.r3T5VIbu023396@red.freebsd.org>
Date: Mon, 29 Apr 2013 05:31:18 GMT
From: Alex Strange <astrange@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: 8.3 nfsv4 client reports "nfsv4 client/server protocol prob err=10026"
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         178231
>Category:       kern
>Synopsis:       [nfs] 8.3 nfsv4 client reports "nfsv4 client/server protocol prob err=10026"
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-fs
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 29 05:40:01 UTC 2013
>Closed-Date:    
>Last-Modified:  Mon May 20 21:08:00 UTC 2013
>Originator:     Alex Strange
>Release:        8.3-release
>Organization:
>Environment:
FreeBSD sys 8.3-RELEASE FreeBSD 8.3-RELEASE #0: Tue Oct 30 02:00:23 EDT 2012     mrvacbob@sys:/usr/obj/usr/src/sys/CUSTOM  amd64

>Description:
I have a busy web server which mounts two other systems over NFSv4 and writes files to them. All three systems are running FreeBSD 8.3 amd64.

dmesg frequently fills up with this:
nfsv4 client/server protocol prob err=10026
nfsv4 client/server protocol prob err=10026


which causes file writes to fail on specific worker processes. Killing the right process while the logging is going fixes the issue for a while, but this would cause requests to error so we'd prefer to avoid it.

nfs activity:
% nfsstat -w 1 -e
        GtAttr Lookup Rdlink   Read  Write Rename Access  Rddir

Client:   1106    880      0      0    424     60    880      0
Server:      0      0      0      0      0      0      0      0

nfs related sysctls:
vfs.newnfs.downdelayinitial: 12
vfs.newnfs.downdelayinterval: 30
vfs.newnfs.keytab_enctype: 1
vfs.newnfs.skip_wcc_data_onerr: 1
vfs.newnfs.nfs3_jukebox_delay: 10
vfs.newnfs.reconnects: 0
vfs.newnfs.bufpackets: 4
vfs.newnfs.callback_addr: 
vfs.newnfs.nfs4acl_enable: 1
vfs.newnfs.realign_count: 0
vfs.newnfs.realign_test: 0
vfs.newnfs.nfs_directio_allow_mmap: 1
vfs.newnfs.nfs_directio_enable: 0
vfs.newnfs.clean_pages_on_close: 1
vfs.newnfs.commit_on_close: 0
vfs.newnfs.prime_access_cache: 0
vfs.newnfs.access_cache_timeout: 60
vfs.newnfs.diskless_rootpath: 
vfs.newnfs.diskless_valid: 0
vfs.newnfs.nfs_ip_paranoia: 1
vfs.newnfs.defect: 0
vfs.newnfs.iodmax: 4
vfs.newnfs.iodmin: 0
vfs.newnfs.iodmaxidle: 120
vfs.newnfs.disable_checkutf8: 0
vfs.newnfs.server_max_nfsvers: 4
vfs.newnfs.server_min_nfsvers: 2
vfs.newnfs.nfs_privport: 0
vfs.newnfs.enable_locallocks: 0
vfs.newnfs.issue_delegations: 0
vfs.newnfs.commit_miss: 0
vfs.newnfs.commit_blks: 0
vfs.newnfs.mirrormnt: 1
vfs.newnfs.minthreads: 1
vfs.newnfs.maxthreads: 4
vfs.newnfs.threads: 0
vfs.newnfs.request_space_used: 0
vfs.newnfs.request_space_used_highest: 0
vfs.newnfs.request_space_high: 47185920
vfs.newnfs.request_space_low: 31457280
vfs.newnfs.request_space_throttled: 0
vfs.newnfs.request_space_throttle_count: 0

Applicable fstab:
boards:/boards /www/boards nfs rw,noatime,nfsv4,async,nolockd,late,intr,bg,timeout=5 0 0
images:/images /www/images nfs rw,noatime,nfsv4,async,nolockd,late,intr,bg,timeout=5 0 0
images:/thumbs /www/thumbs nfs rw,noatime,nfsv4,async,nolockd,late,intr,bg,timeout=5 0 0
>How-To-Repeat:
Wait.
>Fix:


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-fs 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon May 20 03:29:21 UTC 2013 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=178231 
State-Changed-From-To: open->feedback 
State-Changed-By: rmacklem 
State-Changed-When: Mon May 20 21:06:30 UTC 2013 
State-Changed-Why:  

I have emailed the reporter and asked him to test without 
the "intr" mount option, which could cause this. The error 
indicates that the client got "out of sync" with the server 
while doing open/lock operations, which are serialized via 
a sequence number. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=178231 
>Unformatted:
