From nobody@FreeBSD.org  Mon Apr 26 06:04:40 2010
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 95701106564A
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 26 Apr 2010 06:04:40 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 83C818FC16
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 26 Apr 2010 06:04:40 +0000 (UTC)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o3Q64eh6063177
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 26 Apr 2010 06:04:40 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id o3Q64eZw063176;
	Mon, 26 Apr 2010 06:04:40 GMT
	(envelope-from nobody)
Message-Id: <201004260604.o3Q64eZw063176@www.freebsd.org>
Date: Mon, 26 Apr 2010 06:04:40 GMT
From: Chad Milios <cropcirclesystems@yahoo.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: shutdown of jails breaks intra-jail dependencies
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         146053
>Category:       conf
>Synopsis:       [patch] [request] shutdown of jails breaks inter-jail dependencies
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-rc
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Mon Apr 26 06:10:00 UTC 2010
>Closed-Date:    
>Last-Modified:  Sun Jan 20 23:10:01 UTC 2013
>Originator:     Chad Milios
>Release:        7.3-RELEASE
>Organization:
Crop Circle Systems
>Environment:
FreeBSD sensei.ccsys.org 7.3-RELEASE FreeBSD 7.3-RELEASE #2: Sun Apr 11 14:55:27 UTC 2010     root@sarge.ccsys.org:/usr/obj/usr/src/sys/IPSEC  amd64
>Description:
jails shutdown in the same order they are started. this makes it impossible to have one jail depend on any of the services of another. (for instance, my first jail to start up provides a caching dns resolver that the other jails use) as implemented currently, that jail shuts down first, causing other jails to loose their ability to look up names. if any other jail needs that functionality (or anything provided by an earlier-started jail for that matter) in order to shut down gracefully, it will timeout and often timeout rc.shutdown as well.
>How-To-Repeat:
create a couple jails. boot multi-user, then shutdown. note the order.
>Fix:
add boolean rc.conf variable jail_reverse_shutdown. reverse jail_list when doing jail_stop in rc.d/jail if it is set to yes. i would argue that the default should be enabled (a la: rcorder in rc.shutdown) but to preserve existing behavior it probably ought to be off by default.

i have been using this small straightforward patch in production for years, and been meaning to kick it upstream for a long time. please get this into 7.4 and 8.1. (if not 7.3-STABLE or 7.3-RELEASE-p?)

Patch attached with submission follows:

diff -rNU3 ../src.dist/etc/defaults/rc.conf ./etc/defaults/rc.conf
--- ../src.dist/etc/defaults/rc.conf	2010-02-10 00:26:20.000000000 +0000
+++ ./etc/defaults/rc.conf	2010-04-25 21:20:23.000000000 +0000
@@ -621,6 +621,7 @@
 jail_set_hostname_allow="YES" # Allow root user in a jail to change its hostname
 jail_socket_unixiproute_only="YES" # Route only TCP/IP within a jail
 jail_sysvipc_allow="NO"	# Allow SystemV IPC use from within a jail
+jail_reverse_shutdown="NO"	# Shutdown jails in reverse order
 
 #
 # To use rc's built-in jail infrastructure create entries for
diff -rNU3 ../src.dist/etc/rc.d/jail ./etc/rc.d/jail
--- ../src.dist/etc/rc.d/jail	2010-02-10 00:26:20.000000000 +0000
+++ ./etc/rc.d/jail	2010-04-25 21:20:23.000000000 +0000
@@ -679,6 +679,9 @@
 jail_stop()
 {
 	echo -n 'Stopping jails:'
+	if checkyesno jail_reverse_shutdown; then
+		jail_list=`reverse_list $jail_list`
+	fi
 	for _jail in ${jail_list}
 	do
 		if [ -f "/var/run/jail_${_jail}.id" ]; then


>Release-Note:
>Audit-Trail:

From: Chad Milios <cropcirclesystems@yahoo.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: misc/146053: shutdown of jails breaks intra-jail dependencies
Date: Mon, 26 Apr 2010 17:13:55 -0700 (PDT)

 heres an additional patch to the rc.conf manpage to document this feature:
 
 diff -rNU3 ../src.dist/share/man/man5/rc.conf.5 ./share/man/man5/rc.conf.5
 --- ../src.dist/share/man/man5/rc.conf.52010-02-10 00:26:20.000000000 +0000
 +++ share/man/man5/rc.conf.52010-04-26 16:42:53.000000000 +0000
 @@ -3696,6 +3696,11 @@
  If set to
  .Dq Li YES ,
  allow applications within a jail to use System V IPC.
 +.It Va jail_reverse_shutdown
 +.Pq Vt bool
 +If set to
 +.Dq Li YES ,
 +use the reverse order when stopping multiple jails.
  .\" ----- ISDN settings ---------------------------------
  .It Va isdn_enable
  .Pq Vt bool
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-rc 
Responsible-Changed-By: jilles 
Responsible-Changed-When: Tue Apr 27 21:58:27 UTC 2010 
Responsible-Changed-Why:  
Assign to correct mailing list, fix category. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=146053 

From: Chad Milios <cropcirclesystems@yahoo.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: conf/146053: [patch] [request] shutdown of jails breaks inter-jail dependencies
Date: Wed, 2 Jun 2010 18:03:32 -0700 (PDT)

 I think my original change request may have been unclear and my example was certainly contrived.
 
 If a user of FreeBSD jails wishes to ensure that for jail A B and C, A and B are always running at any point in time in which C is, it is impossible to do so using the existing rc.conf system because `/etc/rc.d/jail stop` will stop them all, serially, in the same order that `./jail start` started them all.
 
 In the EXISTING implementation, for rc.d scripts "a b jail f" (as a list output by rcorder(8)), and jails "C D E":
 
 rc does basically something to the effect of: `./a start; ./b start; ./jail start C; ./jail stop D; ./jail stop E; ./f start`
 
 while rc.shutdown then later does something to the effect of: `./f stop; ./jail stop C; ./jail stop D; ./jail stop E; ./b stop; ./a stop`
 
 but I PROPOSE that the rc.shutdown process should instead result in: `./f stop; ./jail stop E; ./jail stop D; ./jail stop C; ./b stop; ./a stop`
 
 If no one seems to care and there is no one worrying about their jails' shutdown order, IMHO we should just quietly add the single important line of my patch to rc.d/jail:
 
  jail_stop()
  {
  echo -n 'Stopping jails:'
 +jail_list=`reverse_list $jail_list`
 
 and not clutter up rc.conf any more with this jail_reverse_shutdown boolean. But, the patch i originally provided will add this rc.conf variable while allowing existing behavior to remain the default.
 
 Just because we are not meta-data gathering and automatically dependency sorting the jails for the user (a la what rcorder does with the rc.d scripts) doesn't mean we shouldn't reverse that jail_list for them when it comes time to shutdown, (like rc.shutdown does to the dependency-sorted list of rc.d scripts that are output from rcorder,) since the user may have ordered their jail_list manually for dependency reasons, and some dependencies may be required all the way through shutdown for a particular servi ce.
 
 reversing the jail list in jail_stop also affects manual use of the jail rc.d script on the command line such as `./jail start A B C` and `./jail stop A B C`. the jails should still be listed in dependency order on the command line because jail_stop will reverse the list when stopping them.

From: "Chad J. Milios" <cropcirclesystems@yahoo.com>
To: bug-followup@FreeBSD.org,
 cropcirclesystems@yahoo.com
Cc:  
Subject: Re: conf/146053: [patch] [request] shutdown of jails breaks inter-jail dependencies
Date: Sun, 20 Jan 2013 16:03:58 -0700

 the cool new jail features in 9.1 release now include ACTUAL dependency =
 sorting using the 'depend' parameter.
 
 If someone wants to close this PR, that'd be fine.=
>Unformatted:
