From nobody@FreeBSD.org  Fri Nov 25 13:00:27 2005
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 2181E16A41F
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 25 Nov 2005 13:00:27 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 96B5743D94
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 25 Nov 2005 13:00:04 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id jAPD03o5053537
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 25 Nov 2005 13:00:03 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id jAPD03uj053536;
	Fri, 25 Nov 2005 13:00:03 GMT
	(envelope-from nobody)
Message-Id: <200511251300.jAPD03uj053536@www.freebsd.org>
Date: Fri, 25 Nov 2005 13:00:03 GMT
From: Philippe Lang <philippe.lang@attiksystem.ch>
To: freebsd-gnats-submit@FreeBSD.org
Subject: impossible to kill a jail
X-Send-Pr-Version: www-2.3

>Number:         89528
>Category:       kern
>Synopsis:       [jail] [patch] impossible to kill a jail
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bz
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Nov 25 13:10:02 GMT 2005
>Closed-Date:    Sat Jan 10 23:34:22 UTC 2009
>Last-Modified:  Sat Jan 10 23:34:22 UTC 2009
>Originator:     Philippe Lang
>Release:        6.0
>Organization:
Attik System
>Environment:
FreeBSD xeon.attiksystem.ch 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Sun Nov 13 11:01:58 CET 2005     plang@xeon.attiksystem.ch:/usr/obj/usr/src/sys/SMP  i386
>Description:
/etc/rc.d/jail restart

or

/etc/rc.d/jail stop

.. apprently leave something in the jail system.

jls

.. mentions about old jails still running in the system.
>How-To-Repeat:
Create a few jails, start them, stop and restart them. Sometimes, jails are not killed. I tried to kill network sockets remaining in the system, but a reboot is the only way of getting rid of old jail ids.
>Fix:

>Release-Note:
>Audit-Trail:

From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>
To: Philippe Lang <philippe.lang@attiksystem.ch>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/89528: impossible to kill a jail
Date: Fri, 25 Nov 2005 13:47:21 +0000 (UTC)

 On Fri, 25 Nov 2005, Philippe Lang wrote:
 
 > /etc/rc.d/jail stop
 >
 > .. apprently leave something in the jail system.
 >
 > jls
 >
 > .. mentions about old jails still running in the system.
 >> How-To-Repeat:
 > Create a few jails, start them, stop and restart them. Sometimes, jails are not killed. I tried to kill network sockets remaining in the system, but a reboot is the only way of getting rid of old jail ids.
 
 I guess looking at netstat there are some connections still hanging
 around and if you just wait long enough you won't see the jail anymore
 in jls and you won't see the finishing connections in netstat - right?
 
 -- 
 Bjoern A. Zeeb				bzeeb at Zabbadoz dot NeT

From: Maxim Konovalov <maxim@macomnet.ru>
To: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>
Cc: bug-followup@freebsd.org
Subject: Re: misc/89528: impossible to kill a jail
Date: Fri, 25 Nov 2005 18:01:59 +0300 (MSK)

 On Fri, 25 Nov 2005, 14:20-0000, Bjoern A. Zeeb wrote:
 
 > On Fri, 25 Nov 2005, Bjoern A. Zeeb wrote:
 >
 > > The following reply was made to PR misc/89528; it has been noted by GNATS.
 > >
 > > From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>
 > > To: Philippe Lang <philippe.lang@attiksystem.ch>
 > > Cc: freebsd-gnats-submit@FreeBSD.org
 > > Subject: Re: misc/89528: impossible to kill a jail
 > > Date: Fri, 25 Nov 2005 13:47:21 +0000 (UTC)
 > >
 > > On Fri, 25 Nov 2005, Philippe Lang wrote:
 > >
 > > > /etc/rc.d/jail stop
 > > >
 > > > .. apprently leave something in the jail system.
 > > >
 > > > jls
 > > >
 > > > .. mentions about old jails still running in the system.
 > > > > How-To-Repeat:
 > > > Create a few jails, start them, stop and restart them. Sometimes, jails
 > > > are not killed. I tried to kill network sockets remaining in the system,
 > > > but a reboot is the only way of getting rid of old jail ids.
 > >
 > > I guess looking at netstat there are some connections still hanging
 > > around and if you just wait long enough you won't see the jail anymore
 > > in jls and you won't see the finishing connections in netstat - right?
 >
 > ok after another round of private mail exchange I have reprodcued it.
 > I am running with pjd@s latest multi-IP jail patch and I can see the
 > same problem. No open or closing sockets in netstat.
 
 I'm sure this is just another leaked reference in the (socket?) code
 somewhere.  Try to run vmstat -z before you create a jail and after
 you kill it.
 
 -- 
 Maxim Konovalov

From: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>
To: Philippe Lang <philippe.lang@attiksystem.ch>
Cc: bug-followup@FreeBSD.org
Subject: RE: misc/89528: impossible to kill a jail
Date: Sat, 3 Dec 2005 09:31:30 +0000 (UTC)

 On Fri, 25 Nov 2005, Philippe Lang wrote:
 
 >> I just checked with my own machine where i had restarted a jail two days
 >> ago and it's still listed in jls though the sockets are gone (seen from
 >> the base system kind of view of course).
 >
 > Same situation here.
 
 I tried to reproduce the problem on a second box with HEAD from some
 days ago and failed up to now.
 
 What I have found so far doing live system debugging ist that on the
 machine showing this problem the initial jail (id 1) that should
 be gone already still has a reference count of 2:
 
 (kgdb) print allprison->lh_first->pr_list->le_next->pr_list->le_next->pr_list->le_next->pr_list->le_next->pr_list->le_next->pr_id
 $9 = 1
 (kgdb) print allprison->lh_first->pr_list->le_next->pr_list->le_next->pr_list->le_next->pr_list->le_next->pr_list->le_next->pr_ref
 $10 = 2
 
 So it seems we are leaking some creds somewhere.
 
 
 What I should know from you:
 
 a) which branch are you running? RELENG_{5,6} or HEAD
 b) if it's not a RELEASE which exact build dates?
     (something like uname -s -v would be a good starting point)
 c) are you running with any patches like multi-IP jail patches?
 d) Do you know a way to successfully reproduce it all the time?
 
 
 Greetings
 Bjoern
 
 -- 
 Bjoern A. Zeeb				bzeeb at Zabbadoz dot NeT
Responsible-Changed-From-To: freebsd-bugs->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Sat Dec 3 09:43:28 GMT 2005 
Responsible-Changed-Why:  
I am already working on this so take the PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org, philippe.lang@attiksystem.ch
Cc:  
Subject: Re: kern/89528 : [jail] impossible to kill a jail
Date: Wed, 4 Jan 2006 17:10:31 +0000 (UTC)

 The problem is a ucred leak which is caused by devfs/?ty refcounting.
 The bad news is that it will be hard to fix and might take some (long)
 time for the appropriate people to work on this.
 I will keep an eye on this but do not expect a solution in the near
 future.
State-Changed-From-To: open->suspended 
State-Changed-By: bz 
State-Changed-When: Wed Sep 6 08:02:41 UTC 2006 
State-Changed-Why:  
This problem will last for some more time and it's nothing I 
can fix or is a jail problem. Reflect by changing state to suspended. 
I'll keep an eye on that problem. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 

From: Matthias Lederhofer <matled@gmx.net>
To: "Bjoern A. Zeeb" <bzeeb@zabbadoz.net>,
	Philippe Lang <philippe.lang@attiksystem.ch>,
	bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Tue, 19 Sep 2006 20:52:58 +0200

 Yesterday I found this bug too and tried a bit how to reproduce it.
 Today I took a fresh install of release 6.1 and could reproduce it for
 the cd version, RELENG_6_1 and RELENG_6:
 
 Create a jail directory, make installworld/distribution, give it a
 ports tree, copy over resolv.conf.
 
 I used the /etc/rc.d/jail script with the following configuration in
 /etc/rc.conf:
 jail_example_rootdir="/jail"
 jail_example_hostname="test"
 jail_example_interface=INTERFACE
 jail_example_ip=JAIL_IP
 jail_example_exec_start="/bin/sh /etc/rc"
 jail_example_exec_stop="/bin/sh /etc/rc.shutdown"
 jail_example_devfs_enable="YES"
 jail_example_fdescfs_enable="YES"
 jail_example_procfs_enable="NO"
 jail_example_mount_enable="NO"
 jail_example_devfs_ruleset="devfsrules_jail"
 jail_example_fstab=""
 jail_example_flags="-l -U root"
 
 Start the jail, install portupgrade (used to reproduce this bug)
 # /etc/rc.d/jail onestart example
 # jexec 1 pkg_add -r portupgrade
 
 And then the command actually making the jail immortal:
 # jexec 1 portinstall -PPF www/apache2
 If the package is not downloaded this did not work for me, so delete
 the package for the next try.  It did not matter if I wait until all
 connections in netstat are gone or not.  This was not always
 reproducable multiple times without a reboot.
 
 To answer all questions from Bjoern:
 > a) which branch are you running? RELENG_{5,6} or HEAD
 release 6.1 (cd), RELENG_6_1, RELENG_6
 > b) if it's not a RELEASE which exact build dates?
 > (something like uname -s -v would be a good starting point)
 RELENG_6/RELENG_6_1 were from today:
 FreeBSD 6.2-PRERELEASE #1: Tue Sep 19 13:46:18 CEST 2006 w/ GENERIC
 (the other one is overwritten)
 > c) are you running with any patches like multi-IP jail patches?
 Nope, just used the cd and cvsup standard/stable/ports-supfile.
 > d) Do you know a way to successfully reproduce it all the time?
 See above.
 
 
 Matthias

From: Ed Schouten <ed@fxq.nl>
To: bug-followup@FreeBSD.org, philippe.lang@attiksystem.ch
Cc: freebsd-hackers@freebsd.org
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Thu, 4 Jan 2007 21:14:34 +0100

 --OE5XN2KVoD5QaTkR
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 Hello everyone,
 
 I decided to investigate this bug because I think the bug is quite
 irritating. After adding some ddb show commands to the source and
 reading a lot of code, I think I understand the problem:
 
 The tty code doesn't leak any ucreds, it's the devfs code that
 crhold()'s an ucred structure. When a new pty is needed, the tty_pty
 code allocates a new pty. It also runs make_dev_cred(), to which it
 passes the thread's ucred. This function calls make_dev_credv(), which
 finally runs crhold().
 
 As long as pty's have been allocated that have been created by threads
 in a jail, the prison structure has more references, causing the zombie
 jails to exist.
 
 Yours,
 --=20
  Ed Schouten <ed@fxq.nl>
  WWW: http://g-rave.nl/
 
 --OE5XN2KVoD5QaTkR
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (FreeBSD)
 
 iD8DBQFFnWAq52SDGA2eCwURAukZAJ4lGKkBlyXrtMLY/nN1EpH35f68hgCdHWSS
 /KmDk8nFZrT/tyvNyQu2Zek=
 =6L9c
 -----END PGP SIGNATURE-----
 
 --OE5XN2KVoD5QaTkR--

From: Ed Schouten <ed@fxq.nl>
To: bug-followup@FreeBSD.org, philippe.lang@attiksystem.ch
Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Thu, 4 Jan 2007 21:49:52 +0100

 --aE2Tjr+Wh4rS13mJ
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 * Ed Schouten <ed@fxq.nl> wrote:
 > As long as pty's have been allocated that have been created by threads
 > in a jail, the prison structure has more references, causing the zombie
 > jails to exist.
 
 We could change the make_dev_credv() routine to crcopy() everything
 except the prison when we're creating a node in a jail. The following
 patch fixes the zombie jail bug on my machine:
 
 --- src/sys/kern/kern_conf.c	Fri Oct 20 09:59:50 2006
 +++ src/sys/kern/kern_conf.c	Thu Jan  4 21:36:44 2007
 @@ -42,6 +42,7 @@
  #include <sys/ctype.h>
  #include <sys/tty.h>
  #include <sys/ucred.h>
 +#include <sys/jail.h>
  #include <machine/stdarg.h>
 =20
  #include <fs/devfs/devfs_int.h>
 @@ -563,7 +564,15 @@
  	=09
  	dev->si_flags |=3D SI_NAMED;
  	if (cr !=3D NULL)
 -		dev->si_cred =3D crhold(cr);
 +		if (cr->cr_prison =3D=3D NULL) {
 +			dev->si_cred =3D crhold(cr);
 +		} else {
 +			/* Don't let the node depend on a prison */
 +			dev->si_cred =3D crget();
 +			crcopy(dev->si_cred, cr);
 +			prison_free(dev->si_cred->cr_prison);
 +			dev->si_cred->cr_prison =3D NULL;
 +		}
  	else
  		dev->si_cred =3D NULL;
  	dev->si_uid =3D uid;
 
 Could other people experiencing this problem as well give this patch a
 try? Thanks a lot!
 
 Yours,
 --=20
  Ed Schouten <ed@fxq.nl>
  WWW: http://g-rave.nl/
 
 --aE2Tjr+Wh4rS13mJ
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (FreeBSD)
 
 iD8DBQFFnWhw52SDGA2eCwURAnf9AJ47ZpkxiM8s2KznHU+NncdjiqjHpACcCzJQ
 jfxOzPnh9zjiOAxisCY8e5A=
 =TOCY
 -----END PGP SIGNATURE-----
 
 --aE2Tjr+Wh4rS13mJ--

From: Ed Schouten <ed@fxq.nl>
To: cokane@cokane.org
Cc: bug-followup@freebsd.org, philippe.lang@attiksystem.ch,
	FreeBSD Hackers <freebsd-hackers@freebsd.org>
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Thu, 4 Jan 2007 22:37:47 +0100

 --W1G4cAX3lNqeBRc9
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 * Coleman Kane <zombyfork@gmail.com> wrote:
 > Does this behavior still occur if you set sysctl kern.pts.enable=3D1 ?
 
 Well, I haven't tested that, but it should be fixed as well, because it
 also calls make_dev_cred().
 
 > Is this at all related to why I have been experiencing zombies left behind
 > for any process that alloc's its own tty (such as gnome-terminal [actually
 > gnome-pty-helper])? If I CTRL-D to end a gnome-terminal session, it will
 > hang all of the gnome-terminals I have open and I typically have to reboot
 > to clear out the zombies that remain. I can't open any more apps that use
 > gnome-pty-helper to allocate ttys unless I attempt to kill it and start it
 > anew (and I am not even completely sure if that works).
 
 As far as I know, this is unrelated. The patch in my previous mail only
 fixes device node creation in jails.
 
 --=20
  Ed Schouten <ed@fxq.nl>
  WWW: http://g-rave.nl/
 
 --W1G4cAX3lNqeBRc9
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (FreeBSD)
 
 iD8DBQFFnXOr52SDGA2eCwURArUyAJ4wenRYkZqHyHCyWPdJvkA3kW25YACdF+cE
 IxKmII5ArYEfxDb+iOJ4C24=
 =HoIt
 -----END PGP SIGNATURE-----
 
 --W1G4cAX3lNqeBRc9--

From: "Coleman Kane" <zombyfork@gmail.com>
To: "Ed Schouten" <ed@fxq.nl>
Cc: bug-followup@freebsd.org, philippe.lang@attiksystem.ch, 
	"FreeBSD Hackers" <freebsd-hackers@freebsd.org>
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Thu, 4 Jan 2007 14:34:16 -0700

 ------=_Part_4911_2287395.1167946456593
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline
 
 On 1/4/07, Ed Schouten <ed@fxq.nl> wrote:
 >
 > * Ed Schouten <ed@fxq.nl> wrote:
 > > As long as pty's have been allocated that have been created by threads
 > > in a jail, the prison structure has more references, causing the zombie
 > > jails to exist.
 >
 > We could change the make_dev_credv() routine to crcopy() everything
 > except the prison when we're creating a node in a jail. The following
 > patch fixes the zombie jail bug on my machine:
 >
 > --- src/sys/kern/kern_conf.c    Fri Oct 20 09:59:50 2006
 > +++ src/sys/kern/kern_conf.c    Thu Jan  4 21:36:44 2007
 > @@ -42,6 +42,7 @@
 > #include <sys/ctype.h>
 > #include <sys/tty.h>
 > #include <sys/ucred.h>
 > +#include <sys/jail.h>
 > #include <machine/stdarg.h>
 >
 > #include <fs/devfs/devfs_int.h>
 > @@ -563,7 +564,15 @@
 >
 >         dev->si_flags |= SI_NAMED;
 >         if (cr != NULL)
 > -               dev->si_cred = crhold(cr);
 > +               if (cr->cr_prison == NULL) {
 > +                       dev->si_cred = crhold(cr);
 > +               } else {
 > +                       /* Don't let the node depend on a prison */
 > +                       dev->si_cred = crget();
 > +                       crcopy(dev->si_cred, cr);
 > +                       prison_free(dev->si_cred->cr_prison);
 > +                       dev->si_cred->cr_prison = NULL;
 > +               }
 >         else
 >                 dev->si_cred = NULL;
 >         dev->si_uid = uid;
 >
 > Could other people experiencing this problem as well give this patch a
 > try? Thanks a lot!
 >
 > Yours,
 > --
 > Ed Schouten <ed@fxq.nl>
 > WWW: http://g-rave.nl/
 >
 >
 >
 Does this behavior still occur if you set sysctl kern.pts.enable=1 ?
 
 Is this at all related to why I have been experiencing zombies left behind
 for any process that alloc's its own tty (such as gnome-terminal [actually
 gnome-pty-helper])? If I CTRL-D to end a gnome-terminal session, it will
 hang all of the gnome-terminals I have open and I typically have to reboot
 to clear out the zombies that remain. I can't open any more apps that use
 gnome-pty-helper to allocate ttys unless I attempt to kill it and start it
 anew (and I am not even completely sure if that works).
 
 --
 Coleman Kane
 
From: Ed Schouten <ed@fxq.nl>
To: bug-followup@FreeBSD.org, philippe.lang@attiksystem.ch
Cc:  
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Fri, 5 Jan 2007 09:44:52 +0100

 --0OOz7ZB592LYQf07
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 * Ed Schouten <ed@fxq.nl> wrote:
 > We could change the make_dev_credv() routine to crcopy() everything
 > except the prison when we're creating a node in a jail. The following
 > patch fixes the zombie jail bug on my machine:
 >=20
 > [...]
 
 I've cleaned up the patch by adding a new routine, change_prison(),
 which changes the prison in an ucred. The patch is available at:
 
 	http://g-rave.nl/junk/freebsd-jail-nodevdep.diff
 
 Yours,
 --=20
  Ed Schouten <ed@fxq.nl>
  WWW: http://g-rave.nl/
 
 --0OOz7ZB592LYQf07
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (FreeBSD)
 
 iD8DBQFFnhAE52SDGA2eCwURApnYAJ0YFELm5lmqKxMDGXglgGxbepvZPQCfftEU
 04yzmHC9kDohmFTi6HITjwc=
 =e1LR
 -----END PGP SIGNATURE-----
 
 --0OOz7ZB592LYQf07--

From: John Baldwin <jhb@freebsd.org>
To: freebsd-hackers@freebsd.org
Cc: Ed Schouten <ed@fxq.nl>, bug-followup@freebsd.org,
        philippe.lang@attiksystem.ch
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Fri, 5 Jan 2007 14:27:44 -0500

 On Thursday 04 January 2007 15:14, Ed Schouten wrote:
 > Hello everyone,
 > 
 > I decided to investigate this bug because I think the bug is quite
 > irritating. After adding some ddb show commands to the source and
 > reading a lot of code, I think I understand the problem:
 > 
 > The tty code doesn't leak any ucreds, it's the devfs code that
 > crhold()'s an ucred structure. When a new pty is needed, the tty_pty
 > code allocates a new pty. It also runs make_dev_cred(), to which it
 > passes the thread's ucred. This function calls make_dev_credv(), which
 > finally runs crhold().
 > 
 > As long as pty's have been allocated that have been created by threads
 > in a jail, the prison structure has more references, causing the zombie
 > jails to exist.
 
 Why aren't the pty's destroyed?  Once all references to the pty are closed it  
 should be destroyed and the resulting devfs_free() should drop the reference.
 Is the pty somehow stuck on the dead_cdevsw?
 
 -- 
 John Baldwin

From: Ed Schouten <ed@fxq.nl>
To: John Baldwin <jhb@freebsd.org>
Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>, bug-followup@freebsd.org,
	philippe.lang@attiksystem.ch
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Sat, 6 Jan 2007 01:06:32 +0100

 --YiEDa0DAkWCtVeE4
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 Content-Transfer-Encoding: quoted-printable
 
 * John Baldwin <jhb@freebsd.org> wrote:
 > Why aren't the pty's destroyed?  Once all references to the pty are close=
 d it =20
 > should be destroyed and the resulting devfs_free() should drop the refere=
 nce.
 > Is the pty somehow stuck on the dead_cdevsw?
 
 Ouch! I found a comment in tty_pty.c that it doesn't deallocate pty's
 anymore. Not a single destroy_dev() is called by tty_pty.c. This means
 the FreeBSD kernel leaks a lot of cdev's and thus ucred's and my patch
 is working around another bug.
 
 Yours,
 --=20
  Ed Schouten <ed@fxq.nl>
  WWW: http://g-rave.nl/
 
 --YiEDa0DAkWCtVeE4
 Content-Type: application/pgp-signature
 Content-Disposition: inline
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.6 (FreeBSD)
 
 iD8DBQFFnugI52SDGA2eCwURAggEAJsF9WpUHCnKZaGvtoGp45MFTZRbDwCfexxV
 UAdEvGoYVyK0gLlbI+8p8VU=
 =Uo2h
 -----END PGP SIGNATURE-----
 
 --YiEDa0DAkWCtVeE4--

From: Juergen Unger <j.unger@addict.de>
To: Ed Schouten <ed@fxq.nl>
Cc: John Baldwin <jhb@freebsd.org>,
   FreeBSD Hackers <freebsd-hackers@freebsd.org>, philippe.lang@attiksystem.ch,
   bug-followup@freebsd.org
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Tue, 16 Jan 2007 21:53:29 +0100

 Hi !
 
 I had this problem on my machines many times now.
 
 One additional thing I found:
 
 on my machines all jails have their own (virtual) disks mounted
 to the root-fs of the jail.
 In the case a zombie jail is left after stopping a jail 
 the entry in the jls output is still visible _and_ 
 it is not possible to umount the virtual disk of the jail
 anymore.  
 
 This is very independant of the version, it occured from my
 first productive 5.x versions up to 6.2 RELEASE where i got
 the problem again today.
 
 Maybe it leads us further if we try to investigate this.
 
 bye,
   Juergen
 
 -- 
 ENOSIG
Responsible-Changed-From-To: bz->freebsd-bugs 
Responsible-Changed-By: bz 
Responsible-Changed-When: Wed Feb 28 12:18:02 UTC 2007 
Responsible-Changed-Why:  
It seems whatever the correct solution will be, 
I won't be able to help with it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 

From: Nick Jones <nick@freebsd.cx>
To: bug-followup@FreeBSD.org, philippe.lang@attiksystem.ch
Cc:  
Subject: Re: kern/89528: [jail] impossible to kill a jail
Date: Tue, 1 May 2007 15:10:45 +0100

 Hi all.
 
 I'm seeing this behaviour on a fresh build (i.e this morning) of
 -STABLE.
 
 Jails shutdown cleanly if they aren't touched, but as soon as I login
 and out again via SSH they'll no longer die properly.  There's no open
 sockets and no processes running within the jail itself, for example:
 
 [nick@x330 ~]$ jls
 JID		IP Address			Hostname		Path
 2		192.168.1.66		dischord.org	/home/jails/dischord.org
 
 [nick@x330 ~]$ sudo jexec 2 ps auxwww
 USER   PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
 root  1468  0.0  0.1  1476   924  p0  R+J   2:02PM   0:00.01 ps auxwww
 
 [nick@x330 ~]$ sudo killall -j 2
 No matching processes were found
 
 [nick@x330 ~]$ sudo jkill 2
 
 [nick@x330 ~]$ jls
 JID		IP Address		Hostname		Path
 2		192.168.1.66	dischord.org	/home/jails/dischord.org
 
 The only thing that'll clear it is a reboot.
 
 -- 
 
 -Nick
 

From: Andrew Thompson <thompsa@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/89528: [jail] [patch] impossible to kill a jail
Date: Fri, 7 Dec 2007 20:21:09 +1300

 A workaround has been committed for 6.3 and 7.0, the problem is avoided
 unless MAC is compiled in to the kernel.

From: "Philip M. Gollucci" <philip@ridecharge.com>
To: bug-followup@FreeBSD.org,  philippe.lang@attiksystem.ch
Cc:  
Subject: Re: kern/89528: [jail] [patch] impossible to kill a jail
Date: Thu, 10 Jan 2008 20:01:45 -0500

 Hi Andrew Thompson,
 
 Thanks for the work, but I have to disagree.  I'll second that this is 
 _definitely_ a PITA.
 
 Everytime I end up in this state, I have to end up rebooting the server 
 to fix it luckly this is just our Stage environment and not production.
 
 I'm fairly sure the C/kernel related stuff is over my head, but if I can 
 help somehow, please let me know.
 
 $ ssh hobbes.tld
 % grep MAC /usr/src/sys/amd64/conf/HOBBES
 
 % uname -a
 FreeBSD hobbes.tld 7.0-BETA3 FreeBSD 7.0-BETA3 #0: Wed Dec  5 15:09:21 
 UTC 2007     root@elektra.rws:/usr/obj/usr/src/sys/HOBBES  amd64
 
 % jls
 JID  IP Address      Hostname                      Path
      20  192.168.1.34    obi.tld                    /jails/obi.tld
      19  192.168.1.33    yoda.tld               /jails/yoda.tld
      18  192.168.1.137   aquagirl.tld           /jails/aquagirl.tld
      17  192.168.1.18    olsen.tld              /jails/olsen.tld
      16  192.168.1.17    arthur.tld             /jails/arthur.tld
      15  192.168.1.79    c3po.tld               /jails/c3po.tld
      14  192.168.1.65    r2d2.tld               /jails/r2d2.tld
      13  192.168.1.166   chewie.tld             /jails/chewie.tld
      12  192.168.1.165   solo.tld               /jails/solo.tld
       4  192.168.0.137   abu.tld                /jails/abu.tld
       3  192.168.0.138   aphrodite.tld          /jails/aphrodite.tld
       2  192.168.0.139   flounder.tld           /jails/flounder.tld
       1  192.168.0.57    bert.tld               /jails/bert.tld
 
 % ssh solo.tld
 %% exit
 % sudo /etc/rc.d/jail stop solo
 
 % jls
 JID  IP Address      Hostname                      Path
      20  192.168.1.34    obi.tld                    /jails/obi.tld
      19  192.168.1.33    yoda.tld               /jails/yoda.tld
      18  192.168.1.137   aquagirl.tld           /jails/aquagirl.tld
      17  192.168.1.18    olsen.tld              /jails/olsen.tld
      16  192.168.1.17    arthur.tld             /jails/arthur.tld
      15  192.168.1.79    c3po.tld               /jails/c3po.tld
      14  192.168.1.65    r2d2.tld               /jails/r2d2.tld
      13  192.168.1.166   chewie.tld             /jails/chewie.tld
      12  192.168.1.165   solo.tld               /jails/solo.tld
       4  192.168.0.137   abu.tld                /jails/abu.tld
       3  192.168.0.138   aphrodite.tld          /jails/aphrodite.tld
       2  192.168.0.139   flounder.tld           /jails/flounder.tld
       1  192.168.0.57    bert.tld               /jails/bert.tld
 
 
 -- 
 ------------------------------------------------------------------------
 Philip M. Gollucci (philip@ridecharge.com)
 o:703.549.2050x206
 Senior System Admin - Riderway, Inc.
 http://riderway.com / http://ridecharge.com
 1024D/EC88A0BF 0DE5 C55C 6BF3 B235 2DAB  B89E 1324 9B4F EC88 A0BF
 
 Work like you don't need the money,
 love like you'll never get hurt,
 and dance like nobody's watching.
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-jail 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Jan 25 22:02:59 UTC 2008 
Responsible-Changed-Why:  
Reassign to appropriate mailing list. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 

From: Charles KOPROWSKI <charles.koprowski@klipix.org>
To: bug-followup@FreeBSD.org, philippe.lang@attiksystem.ch
Cc:  
Subject: Re: kern/89528: [jail] [patch] impossible to kill a jail
Date: Sat, 26 Jan 2008 20:48:36 +0100

 Hi everyone,
 
 I had the same issue on a 6.2-RELEASE. Upgrading to 6.3-RELEASE didn't 
 fix it.
 
 It seems that freshly built jails aren't affected by this problem.
 
 Let me know if I can help or if you need more information on my system.
 
 Regards.
 
 --
 Charles KOPROWSKI

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/89528: [jail] [patch] impossible to kill a jail
Date: Sat, 10 Jan 2009 21:11:01 +0000 (UTC)

 Before I am going to look it up another few times, this is the commit
 referenced by Andrew Thompson at Fri, 7 Dec 2007 20:21:09 +1300.
 
 Can also be looked up as SVN r174280 these days.
 
 -- 
 Bjoern A. Zeeb                      The greatest risk is not taking one.
 
 ---------- Forwarded message ----------
 Date: Wed, 5 Dec 2007 01:22:03 +0000 (UTC)
 From: Andrew Thompson <thompsa@FreeBSD.org>
 To: src-committers@FreeBSD.org, cvs-src@FreeBSD.org, cvs-all@FreeBSD.org
 Subject: cvs commit: src/sys/kern kern_conf.c
 
 thompsa     2007-12-05 01:22:03 UTC
 
    FreeBSD src repository
 
    Modified files:
      sys/kern             kern_conf.c
    Log:
    Apply a workaround for the unkillable jail problem where some devices created
    within the jail are never freed. si_cred is only used by the MAC framework so
    make the cred reference conditional on it being compiled in, this is not a fix
    and will need to be reviewed for any new consumers of si_cred.
 
    This will quell some user complaint when using jails with a default kernel.
 
    Reviewed by:    rwatson
    MFC after:      3 days
 
    Revision  Changes    Path
    1.209     +2 -0      src/sys/kern/kern_conf.c
 
 Index: sys/kern/kern_conf.c
 ===================================================================
 --- sys/kern/kern_conf.c        (revision 174279)
 +++ sys/kern/kern_conf.c        (revision 174280)
 @@ -608,9 +608,11 @@ make_dev_credv(int flags, struct cdevsw *devsw,
 in
          }
 
          dev->si_flags |= SI_NAMED;
 +#ifdef MAC
          if (cr != NULL)
                  dev->si_cred = crhold(cr);
          else
 +#endif
                  dev->si_cred = NULL;
          dev->si_uid = uid;
          dev->si_gid = gid;
Responsible-Changed-From-To: freebsd-jail->bz 
Responsible-Changed-By: bz 
Responsible-Changed-When: Sat Jan 10 23:15:03 UTC 2009 
Responsible-Changed-Why:  
Take again to track possible follow-ups. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 
State-Changed-From-To: suspended->patched 
State-Changed-By: bz 
State-Changed-When: Sat Jan 10 23:19:39 UTC 2009 
State-Changed-Why:  
With MPSAFE TTY, svn r181905, we have proper PTY dtor handling 
and this is gone. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 

From: "Bjoern A. Zeeb" <bz@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/89528: [jail] [patch] impossible to kill a jail
Date: Sat, 10 Jan 2009 23:14:47 +0000 (UTC)

 Hi,
 
 I just talked to Ed again about this.  Summary:
 
 - the workaround in 6/7 for !MAC is the best we can get there.
 
 - In freebsd 8 and later with MPSAFE TTY, SVN r181905, we have proper
    PTY dtor handling and thus the bug is gone there.
 
 As MPSAFE TTY will not be MFCed this bug is considered to be fixed the
 best we can.
 
 -- 
 Bjoern A. Zeeb                      The greatest risk is not taking one.
State-Changed-From-To: patched->closed 
State-Changed-By: bz 
State-Changed-When: Sat Jan 10 23:33:32 UTC 2009 
State-Changed-Why:  
Fixed in HEAD, not entirely fixable in 6/7. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=89528 
>Unformatted:
