From marcolz@rave.ilse.net  Thu Mar 24 17:16:27 2005
Return-Path: <marcolz@rave.ilse.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 6934A16A4CE
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 24 Mar 2005 17:16:27 +0000 (GMT)
Received: from rave.ilse.net (pip0-6.ilse.nl [62.69.162.176])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 0A7AF43D4C
	for <FreeBSD-gnats-submit@freebsd.org>; Thu, 24 Mar 2005 17:16:26 +0000 (GMT)
	(envelope-from marcolz@rave.ilse.net)
Received: from rave.ilse.net (localhost.ilse.nl [127.0.0.1])
	by rave.ilse.net (8.13.3/8.13.1) with ESMTP id j2OHGMkJ086632;
	Thu, 24 Mar 2005 18:16:22 +0100 (CET)
	(envelope-from marcolz@rave.ilse.net)
Received: (from marcolz@localhost)
	by rave.ilse.net (8.13.3/8.13.1/Submit) id j2OHGMrc086629;
	Thu, 24 Mar 2005 18:16:22 +0100 (CET)
	(envelope-from marcolz)
Message-Id: <200503241716.j2OHGMrc086629@rave.ilse.net>
Date: Thu, 24 Mar 2005 18:16:22 +0100 (CET)
From: Marc Olzheim <zlo@zlo.nu>
Reply-To: Marc Olzheim <zlo@zlo.nu>
To: FreeBSD-gnats-submit@freebsd.org
Cc: Sven Berkvens <sven@berkvens.net>
Subject: Deadlock or starvation doing heavy NFS writes with writev
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         79208
>Category:       kern
>Synopsis:       [nfs] Deadlock or starvation doing heavy NFS writes with writev
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Mar 24 17:20:02 GMT 2005
>Closed-Date:    Sat Nov 17 03:19:14 UTC 2007
>Last-Modified:  Sat Nov 17 03:19:14 UTC 2007
>Originator:     Marc Olzheim
>Release:        FreeBSD 5.4-PRERELEASE i386
>Organization:
ilse mediz
>Environment:
System: FreeBSD techno.ilse.net 5.4-PRERELEASE FreeBSD 5.4-PRERELEASE #3: Wed Mar 23 16:18:45 CET 2005     root@rave.ilse.net:/usr/obj/usr/src/sys/SE3DEBUG  i386

	grep nfs /etc/rc.conf:

	nfs_client_enable="YES"
	nfs_reserved_port_only="YES"
	nfs_server_enable="YES"

	Boot message is at
	http://www.stack.nl/~marcolz/FreeBSD/techno.bootlog.txt


>Description:
	Doing 60 MB of writev() over NFS, results in exactly 58638336
	bytes (57264K) ariving at the NFS server, _during_ which the
	machine becomes unresponsive to almost everything but pings...
	The resposiveness stops before 10 MB are written, maybe even as
	soon as the writev() starts.

	Running an "rtprio 0 cat" on console gives me a cat that
	responds to ^T (stty status ^T) with:

	load: 0.72  cmd: cat 673 [flswai] 0.00u 0.00s 0% 652k
	load: 8.87  cmd: cat 673 [flswai] 0.00u 0.29s 0% 652k

	and that still consumes input, but doesn't output anything
	anymore.


>How-To-Repeat:
	writev 60 foo on an NFS mounted filesystem.

Source:
----------------------------------------------------------------------
/* Copyright (C) 2005 by ilse technology - Marc Olzheim */

#include	<err.h>
#include	<fcntl.h>
#include	<stdio.h>
#include	<stdlib.h>
#include	<strings.h>
#include	<sys/stat.h>
#include	<sys/types.h>
#include	<sys/uio.h>
#include	<unistd.h>

int
main(int argc, char *argv[])
{
	int		fd, i, iovcnt;
	struct iovec	*iov;
	char		buffer[1048576];
	ssize_t		written;

	if (3 != argc)
	{
		fprintf(stderr, "Usage: %s <megabytes> <file>\n", argv[0]);
		return(1);
	}

	if (-1 == (fd = open(argv[2], O_WRONLY | O_TRUNC | O_CREAT, S_IWUSR | S_IRUSR)))
		perror("open()");

	iovcnt = atoi(argv[1]);

	iov = malloc(iovcnt * sizeof(struct iovec));
	if (!iov)
		perror("malloc()");

	for (i = 0; i < iovcnt; i++)
	{
		iov[i].iov_base = buffer;
		iov[i].iov_len = sizeof(buffer);
	}

	bzero(buffer, sizeof(buffer));

	written = writev(fd, iov, iovcnt);
	if (written == (ssize_t)-1)
		err(1, "writev()");

	if (written != (ssize_t)(iovcnt * sizeof(buffer)))
		err(1, "Short write (%ld of %lu)",
			(long)written,
			(unsigned long)(iovcnt * sizeof(buffer)));

	if (close(fd))
		perror("close()");

	return(0);
}
----------------------------------------------------------------------
>Fix:
>Release-Note:
>Audit-Trail:

From: Marc Olzheim <zlo@zlo.nu>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc: Marc Olzheim <zlo@zlo.nu>, sven@berkvens.net
Subject: Re: kern/79208: Deadlock or starvation doing heavy NFS writes with writev
Date: Fri, 25 Mar 2005 14:12:13 +0100

 --TRYliJ5NKNqkz5bu
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On a FreeBSD 6.0-CURRENT (Fri Mar 25 11:09:51 CET 2005) machine,
 the same happens.
 
 This machine has less memory 512 MB instead of 4GB, so the limit is
 lower than 60 MB: 31 MB is enough. It produces a file of 31506432 bytes
 on the NFS server, during which the machine becomes unresponsive and
 then never recovers.
 
 ping/ping6 still works, but that's all. Power cycling the machine is all
 that's left...
 
 Oh yeah, forgot to mention: this is as mortal user, not as root, so it's
 a simple local Denial of Service attack. :-(
 
 On FreeBSD-4.11 it works as expected, no hangs.
 
 Marc
 
 --TRYliJ5NKNqkz5bu
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.2.4 (FreeBSD)
 
 iD8DBQFCRA4tezjnobFOgrERAqnUAJ96HLgAJVDPpcks01JbodFIlk/UGACgqCcI
 RTdiBc/Q1zXrriwideSu2Oc=
 =jF8g
 -----END PGP SIGNATURE-----
 
 --TRYliJ5NKNqkz5bu--

From: Marc Olzheim <zlo@zlo.nu>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc: FreeBSD-stable@FreeBSD.org, Sven Berkvens <sven@berkvens.net>
Subject: Re: kern/79208: Deadlock or starvation doing heavy NFS writes with writev
Date: Mon, 4 Apr 2005 15:20:33 +0200

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 FreeBSD 5.3-RELEASE-p6 (SMP) #0: Wed Mar 30 14:28:31 CEST 2005
 on i386 is vulnerable as well, so it has been b0rken for some time now,
 I'm afraid.
 
 Marc
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.2.4 (FreeBSD)
 
 iD8DBQFCUT8hezjnobFOgrERAg7xAKCVF1sLLaKzu7Mm5HaD96LXJFajfgCfYK+g
 lga7RE/3bmqEM5tr6Ru4EGY=
 =xpuk
 -----END PGP SIGNATURE-----

From: Marc Olzheim <marcolz@stack.nl>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc: FreeBSD-stable@FreeBSD.org, Sven Berkvens <sven@berkvens.net>
Subject: Re: kern/79208: Deadlock or starvation doing heavy NFS writes with writev
Date: Tue, 5 Apr 2005 15:21:07 +0200

 Hmm, network services like an ircd, keep functioning until they need to
 write a log entry, so it seems that the only thing that's "hanging", is
 the VFS layer.
 
 Marc
Responsible-Changed-From-To: freebsd-bugs->cel 
Responsible-Changed-By: cel 
Responsible-Changed-When: Fri May 12 21:03:56 UTC 2006 
Responsible-Changed-Why:  


http://www.freebsd.org/cgi/query-pr.cgi?pr=79208 
Responsible-Changed-From-To: cel->freebsd-bugs 
Responsible-Changed-By: cel 
Responsible-Changed-When: Mon Mar 12 15:30:07 UTC 2007 
Responsible-Changed-Why:  
Back to the public pool. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79208 
State-Changed-From-To: open->feedback 
State-Changed-By: kmacy 
State-Changed-When: Fri Nov 16 07:30:15 UTC 2007 
State-Changed-Why:  

Sounds like Giant contention. Is this still an issue on RELENG_7? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79208 
State-Changed-From-To: feedback->closed 
State-Changed-By: kmacy 
State-Changed-When: Sat Nov 17 03:18:55 UTC 2007 
State-Changed-Why:  

Submitter says that this is fixed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=79208 
>Unformatted:
