From dmitry@atlantis.dp.ua  Sun Feb 18 14:33:24 2007
Return-Path: <dmitry@atlantis.dp.ua>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 0FD0616A407
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 18 Feb 2007 14:33:24 +0000 (UTC)
	(envelope-from dmitry@atlantis.dp.ua)
Received: from postman.atlantis.dp.ua (postman.atlantis.dp.ua [193.108.47.1])
	by mx1.freebsd.org (Postfix) with ESMTP id 6523513C467
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 18 Feb 2007 14:33:23 +0000 (UTC)
	(envelope-from dmitry@atlantis.dp.ua)
Received: from homelynx.homenet (p177.atlantis.dp.ua [193.19.241.177])
	by postman.atlantis.dp.ua (8.13.1/8.13.1) with ESMTP id l1IEHxp5069663
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 18 Feb 2007 16:18:00 +0200 (EET)
	(envelope-from dmitry@atlantis.dp.ua)
Received: from homelynx.homenet (localhost [127.0.0.1])
	by homelynx.homenet (8.13.8/8.13.8) with ESMTP id l1IEHtxB001880
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 18 Feb 2007 16:17:55 +0200 (EET)
	(envelope-from dmitry@homelynx.homenet)
Received: (from dmitry@localhost)
	by homelynx.homenet (8.13.8/8.13.8/Submit) id l1IEHsCJ001879;
	Sun, 18 Feb 2007 16:17:54 +0200 (EET)
	(envelope-from dmitry)
Message-Id: <200702181417.l1IEHsCJ001879@homelynx.homenet>
Date: Sun, 18 Feb 2007 16:17:54 +0200 (EET)
From: Dmitry Pryanishnikov <lynx.ripe@gmail.com>
Reply-To: Dmitry Pryanishnikov <lynx.ripe@gmail.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: kernel ppp(4) botches clist reservation in RELENG_6
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         109277
>Category:       kern
>Synopsis:       [pppd] [patch] : kernel ppp(4) botches clist reservation in RELENG_6
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Feb 18 14:40:05 GMT 2007
>Closed-Date:    
>Last-Modified:  Sun Apr 22 10:10:42 GMT 2007
>Originator:     Dmitry Pryanishnikov
>Release:        FreeBSD 6.2-STABLE i386
>Organization:
Atlantis ISP
>Environment:
System: FreeBSD homelynx.homenet 6.2-STABLE FreeBSD 6.2-STABLE #0: Sun Feb 18 05:55:06 EET 2007 root@homelynx.homenet:/usr/obj/usr/RELENG_6/src/sys/lynx i386
    Hardware: Intel D845EBG2 mainboard + Pentium(R) 4 CPU 2.80GHz +
    RAM 512Mb, ECC check+correction enabled.
    System is rock-stable when NOT using ppp(4).

>Description:
    Very rare (maybe, once a month) spontaneous crashes occur during the
    active simultaneous use of kernel ppp and system console. When console
    is in X.org mode, system just silently reboots. OTOH, there is a certain
    chance to get valid crash dump when system console is in text mode.
    Last such a crash was "panic: clist reservation botch" (see cblock_alloc()
    function in /sys/kern/tty_subr.c), this was RELENG_6 as of 1-Feb-2007,
    backtrace was:
    
panic(c05f55c8,0,c04cd3ee,20,38,...) at 0xc049a8a4 = panic+0xa8
b_to_q(c37fd6a8,24,c36d6838,c36d6838,0,...) at 0xc04cd60e = b_to_q+0xce
pppasyncstart(c62bfc00,c36cd50c,0,c05f9daf,3e3) at 0xc0508ff4 = pppasyncstart+0x
108
pppoutput(c36cd400,c37fd600,c39b7a70,c39debdc,0,...) at 0xc0506a36 = pppoutput+0
x326
ip_output(c37fd600,0,d9bc79b8,0,0,c3a7e654) at 0xc0526ab4 = ip_output+0xa64
tcp_output(c3a81cb0) at 0xc052eee5 = tcp_output+0xe05
tcp_input(c37fde00,14,d9bc7b80,0,0,...) at 0xc052d467 = tcp_input+0x28df
ip_input(c37fde00,c37fde74,0,8c,c37fde00,...) at 0xc05248ad = ip_input+0x75d
div_send(c3a826f4,0,c37fde00,c6a27120,0,...) at 0xc079bc1b = div_send+0x17b
sosend(c3a826f4,c6a27120,d9bc7c40,c37fde00,0,0,c382c000) at 0xc04d1fd3 = sosend+
0x5eb
kern_sendit(c382c000,3,d9bc7cbc,0,0,0) at 0xc04d71a4 = kern_sendit+0x104
sendit(c382c000,3,d9bc7cbc,0,bfbdebfc,...) at 0xc04d7077 = sendit+0x147
sendto(c382c000,d9bc7d04) at 0xc04d72d5 = sendto+0x4d
syscall(3b,3b,bfbe003b,1,8c,...) at 0xc05c62c7 = syscall+0x22f
Xint0x80_syscall() at 0xc05b495f = Xint0x80_syscall+0x1f

    I've decided to look thru closed PRs and found kern/25632, which
    describes a similar problem (yes, that was RELENG_4 kernel vs.
    USB stack interaction, but the result - bothched clist reservation - 
    was the same). So there's apparently a lack of proper
    locking during the operations with clist in kernel ppp within modern
    (at least RELENG_6) kernel.

>How-To-Repeat:
    I've shamelessly stolen the idea of cblock_alloc() recursion detection
    for the kern/25632:

--- tty_subr.c.orig	Fri Jan  7 01:35:40 2005
+++ tty_subr.c	Sun Feb 18 14:37:29 2007
@@ -94,17 +94,30 @@
  * Remove a cblock from the cfreelist queue and return a pointer
  * to it.
  */
+static int someone_here = 0;
+#define N1MAX 100000
 static __inline struct cblock *
 cblock_alloc()
 {
 	struct cblock *cblockp;
+	int n1;
 
+	for (n1=0; n1<N1MAX; n1++)
+	    if (someone_here != 0) panic("cblock_alloc recursion a");
+	someone_here++;
+	for (n1=0; n1<N1MAX; n1++)
+	    if (someone_here != 1) panic("cblock_alloc recursion b");
 	cblockp = cfreelist;
 	if (cblockp == NULL)
 		panic("clist reservation botch");
 	cfreelist = cblockp->c_next;
 	cblockp->c_next = NULL;
 	cfreecount -= CBSIZE;
+	for (n1=0; n1<N1MAX; n1++)
+	    if (someone_here != 1) panic("cblock_alloc recursion c");
+	someone_here--;
+	for (n1=0; n1<N1MAX; n1++)
+	    if (someone_here != 0) panic("cblock_alloc recursion d");
 	return (cblockp);
 }
 
    With the kernel patched this way I've got the "cblock_alloc recursion a"
    panic almost immediately after setting up ppp(4) connection and pinging
    remote peer with 'ping -f' and simultaneous trampling upon the keyboard:
    
#11 0xc049abd3 in panic (fmt=0xc05f6208 "cblock_alloc recursion a")
    at /usr/RELENG_6/src/sys/kern/kern_shutdown.c:549
#12 0xc04cd7b7 in putc (chr=39, clistp=0xc36ef000)
    at /usr/RELENG_6/src/sys/kern/tty_subr.c:106
#13 0xc04c6b6b in ttyinput (c=39, tp=0xc36ef000)
    at /usr/RELENG_6/src/sys/kern/tty.c:657
#14 0xc05a81e9 in sckbdevent (thiskbd=0xc064f440, event=0, arg=0xc0667000)
    at linedisc.h:122
#15 0xc05974ed in atkbd_intr (kbd=0xc064f440, arg=0x0)
    at /usr/RELENG_6/src/sys/dev/atkbdc/atkbd.c:503
#16 0xc059860a in atkbdintr (arg=0xc1015000)
    at /usr/RELENG_6/src/sys/dev/atkbdc/atkbd_atkbdc.c:174
#17 0xc0487712 in ithread_execute_handlers (p=0xc36aa000, ie=0xc35afc00)
    at /usr/RELENG_6/src/sys/kern/kern_intr.c:682
#18 0xc0487836 in ithread_loop (arg=0xc36e2660)
    at /usr/RELENG_6/src/sys/kern/kern_intr.c:765
#19 0xc0486980 in fork_exit (callout=0xc04877d0 <ithread_loop>,
    arg=0xc36e2660, frame=0xd5633d38)
    at /usr/RELENG_6/src/sys/kern/kern_fork.c:821
#20 0xc05b551c in fork_trampoline ()
    at /usr/RELENG_6/src/sys/i386/i386/exception.s:208

    Looks like ppp(4) enters cblock_alloc(), then gets preempted, then
    ttyinput() reenters cblock_alloc().

>Fix:
    I'm ready to provide further debugging information on this issue.
    Unfortunately, I'm not familiar enough with the locking concepts
    in modern FreeBSD kernels (and in tty subsystem particularly)
    in order to make the fix myself.
>Release-Note:
>Audit-Trail:

From: Bruce Evans <bde@zeta.org.au>
To: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
Cc: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Subject: Re: kern/109277: kernel ppp(4) botches clist reservation in RELENG_6
Date: Mon, 19 Feb 2007 11:18:34 +1100 (EST)

 On Sun, 18 Feb 2007, Dmitry Pryanishnikov wrote:
 
 >> Description:
 >    Very rare (maybe, once a month) spontaneous crashes occur during the
 >    active simultaneous use of kernel ppp and system console. When console
 >    is in X.org mode, system just silently reboots. OTOH, there is a certain
 >    chance to get valid crash dump when system console is in text mode.
 >    Last such a crash was "panic: clist reservation botch" (see cblock_alloc()
 >    function in /sys/kern/tty_subr.c), this was RELENG_6 as of 1-Feb-2007,
 >    backtrace was:
 > ...
 >    Looks like ppp(4) enters cblock_alloc(), then gets preempted, then
 >    ttyinput() reenters cblock_alloc().
 >
 >> Fix:
 >    I'm ready to provide further debugging information on this issue.
 >    Unfortunately, I'm not familiar enough with the locking concepts
 >    in modern FreeBSD kernels (and in tty subsystem particularly)
 >    in order to make the fix myself.
 
 Tty locking is especially simple and not very good -- everything must
 be Giant-locked to work.  However, the default for network drivers is
 now not to use Giant locking.  ppp doesn't seem to be aware of this.
 The only simple fix seems to be to pessimize all network drivers by
 configuring Giant locking for them all -- see netisr.c.  I'm not sure
 if this is enough -- it is probably necessary to Giant-lock all calls
 into ppp (especially ioctls), but things in netisr.c only logically
 affect isrs.
 
 Bruce

From: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
To: Bruce Evans <bde@zeta.org.au>
Cc: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Subject: Re: kern/109277: kernel ppp(4) botches clist reservation in RELENG_6
Date: Thu, 22 Feb 2007 00:30:16 +0200 (EET)

 Hello!
 
 On Mon, 19 Feb 2007, Bruce Evans wrote:
 > On Sun, 18 Feb 2007, Dmitry Pryanishnikov wrote:
 >>    Last such a crash was "panic: clist reservation botch" (see 
 >> cblock_alloc()
 >>    function in /sys/kern/tty_subr.c), this was RELENG_6 as of 1-Feb-2007,
 >>> Fix:
 >>    I'm ready to provide further debugging information on this issue.
 >>    Unfortunately, I'm not familiar enough with the locking concepts
 >>    in modern FreeBSD kernels (and in tty subsystem particularly)
 >>    in order to make the fix myself.
 >
 > Tty locking is especially simple and not very good -- everything must
 > be Giant-locked to work.  However, the default for network drivers is
 > now not to use Giant locking.  ppp doesn't seem to be aware of this.
 
    I was under the wrong impression that the addition of IFF_NEEDSGIANT to
 ppp's ifp->if_flags (made 30-Mar-2006 by rwatson@) cured exactly this problem. 
 But apparently it's not enough: simple addition of the GIANT_REQUIRED at 
 start of the cblock_alloc() reveals that Giant isn't held at this point.
 
 > The only simple fix seems to be to pessimize all network drivers by
 > configuring Giant locking for them all -- see netisr.c.  I'm not sure
 > if this is enough -- it is probably necessary to Giant-lock all calls
 > into ppp (especially ioctls), but things in netisr.c only logically
 > affect isrs.
 
    I've tried to solve the problem from the other end. Upon some analysis
 of the ppp(4) code I concluded that: a) all calls of the tty functions
 are concentrated in the single ppp(4) source file ppp_tty.c, and
 b) _almost_ all these calls are protected by the now dummy pair of function
 calls "s = spltty(); / splx(s);". So I went ahead and converted those
 pairs to the conditional Giant acquisition:
 
 --- ppp_tty.c.orig	Tue Dec 12 23:56:22 2006
 +++ ppp_tty.c	Wed Feb 21 21:39:50 2007
 @@ -86,6 +86,8 @@
   #include <sys/tty.h>
   #include <sys/conf.h>
   #include <sys/uio.h>
 +#include <sys/lock.h>
 +#include <sys/mutex.h>
 
   #ifdef PPP_FILTER
   #include <net/bpf.h>
 @@ -93,6 +95,12 @@
   #include <net/if_ppp.h>
   #include <net/if_pppvar.h>
 
 +#undef spltty
 +#undef splx
 +#define spltty()	mtx_owned(&Giant) ? 0 : (mtx_lock(&Giant), 1)
 +#define splx(relflg)	if (relflg) mtx_unlock(&Giant)
 +
 +
   static int	pppopen(struct cdev *dev, struct tty *tp);
   static int	pppclose(struct tty *tp, int flag);
   static int	pppread(struct tty *tp, struct uio *uio, int flag);
 @@ -556,7 +564,9 @@
   	    /* XXX as above. */
   	    if (CCOUNT(&tp->t_outq) == 0) {
   		++sc->sc_stats.ppp_obytes;
 +		s = spltty();
   		(void) putc(PPP_FLAG, &tp->t_outq);
 +		splx(s);
   	    }
 
   	    /* Calculate the FCS for the first mbuf's worth. */
 @@ -579,7 +589,9 @@
   		n = cp - start;
   		if (n) {
   		    /* NetBSD (0.9 or later), 4.3-Reno or similar. */
 +		    s = spltty();
   		    ndone = n - b_to_q(start, n, &tp->t_outq);
 +		    splx(s);
   		    len -= ndone;
   		    start += ndone;
   		    sc->sc_stats.ppp_obytes += ndone;
 
 This actually insures that Giant is held within tty layer (GIANT_REQUIRED
 check in cblock_alloc() confirms this), but this causes lock order reversals 
 like these:
 
 lock order reversal: (Giant after non-sleepable)
 1st 0xc39904c8 inp (rawinp) @ /usr/RELENG_6/src/sys/netinet/raw_ip.c:289
 2nd 0xc06488a0 Giant (Giant) @ /usr/RELENG_6/src/sys/net/ppp_tty.c:567
 KDB: stack backtrace:
 kdb_backtrace(0,ffffffff,c06556a0,c0657ba8,c0625fe4,...) at 0xc04b2f59 = kdb_backtrace+0x29
 witness_checkorder(c06488a0,9,c05fff6e,237) at 0xc04bdb00 = witness_checkorder+0x578
 _mtx_lock_flags(c06488a0,0,c05fff6e,237) at 0xc0493180 = _mtx_lock_flags+0x78
 pppasyncstart(c35ab800,c3692d0c,0,c05ffb7e,3e3) at 0xc050c6e4 = pppasyncstart+0x90
 pppoutput(c3692c00,c3804700,c39764d0,c3987bdc,0,...) at 0xc0509d7a = pppoutput+0x326
 ip_output(c3804700,0,dae08b24,20,0,...) at 0xc052a764 = ip_output+0xa64
 rip_output(c3804700,c3a8b000,12e6cc1,40,c3804700,...) at 0xc052cb3b = rip_output+0x29f
 rip_send(c3a8b000,0,c3804700,c3a45470,0,...) at 0xc052d62f = rip_send+0x93
 sosend(c3a8b000,c3a45470,dae08c3c,c3804700,0,0,c6895780) at 0xc04d4ff7 = sosend+0x5eb
 kern_sendit(c6895780,3,dae08cbc,0,0,0) at 0xc04da28c = kern_sendit+0x104
 sendit(c6895780,3,dae08cbc,0,804e0b4,...) at 0xc04da15f = sendit+0x163
 sendto(c6895780,dae08d04) at 0xc04da3bd = sendto+0x4d
 syscall(3b,3b,3b,804e074,40,...) at 0xc05ca317 = syscall+0x22f
 Xint0x80_syscall() at 0xc05b89af = Xint0x80_syscall+0x1f
 --- syscall (133, FreeBSD ELF32, sendto), eip = 0x481401eb, esp = 0xbfbee90c, ebp = 0xbfbee958 ---
 
 lock order reversal: (Giant after non-sleepable)
 1st 0xc398457c inp (udpinp) @ /usr/RELENG_6/src/sys/netinet/udp_usrreq.c:804
 2nd 0xc06488a0 Giant (Giant) @ /usr/RELENG_6/src/sys/net/ppp_tty.c:567
 KDB: stack backtrace:
 kdb_backtrace(0,ffffffff,c0657838,c0657ba8,c0625fe4,...) at 0xc04b2f59 = kdb_backtrace+0x29
 witness_checkorder(c06488a0,9,c05fff6e,237) at 0xc04bdb00 = witness_checkorder+0x578
 _mtx_lock_flags(c06488a0,0,c05fff6e,237) at 0xc0493180 = _mtx_lock_flags+0x78
 pppasyncstart(c35ab800,c3692d0c,0,c05ffb7e,3e3) at 0xc050c6e4 = pppasyncstart+0x90
 pppoutput(c3692c00,c3803900,c39764d0,c3987bdc,0,...) at 0xc0509d7a = pppoutput+0x326
 ip_output(c3803900,0,dae02b00,0,0,...) at 0xc052a764 = ip_output+0xa64
 udp_output(c39844ec,c3803900,c384a330,0,c6895480,...) at 0xc053a5b2 = udp_output+0x4b6
 udp_send(c3a8b000,0,c3803900,c384a330,0,...) at 0xc053ab0a = udp_send+0x1a
 sosend(c3a8b000,c384a330,dae02c3c,c3803900,0,0,c6895480) at 0xc04d4ff7 = sosend+0x5eb
 kern_sendit(c6895480,4,dae02cbc,0,0,0) at 0xc04da28c = kern_sendit+0x104
 sendit(c6895480,4,dae02cbc,0,bfbfe890,...) at 0xc04da15f = sendit+0x163
 sendto(c6895480,dae02d04) at 0xc04da3bd = sendto+0x4d
 syscall(3b,3b,3b,bfbfed4c,bfbfe860,...) at 0xc05ca317 = syscall+0x22f
 Xint0x80_syscall() at 0xc05b89af = Xint0x80_syscall+0x1f
 --- syscall (133, FreeBSD ELF32, sendto), eip = 0x481341eb, esp = 0xbfbfe80c, ebp = 0xbfbfe838 ---
 
 So, is it possible to avoid these LORs w/o massive ppp/tty/network code
 rewriting?
 
 > Bruce
 
 Sincerely, Dmitry
 -- 
 Atlantis ISP, System Administrator
 e-mail:  dmitry@atlantis.dp.ua
 nic-hdl: LYNX-RIPE

From: Bruce Evans <bde@zeta.org.au>
To: Dmitry Pryanishnikov <dmitry@atlantis.dp.ua>
Cc: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org, 
    rwatson@freebsd.org
Subject: Re: kern/109277: kernel ppp(4) botches clist reservation in RELENG_6
Date: Thu, 22 Feb 2007 17:02:23 +1100 (EST)

 On Thu, 22 Feb 2007, Dmitry Pryanishnikov wrote:
 > On Mon, 19 Feb 2007, Bruce Evans wrote:
 >> On Sun, 18 Feb 2007, Dmitry Pryanishnikov wrote:
 >>>    Last such a crash was "panic: clist reservation botch" (see 
 >>> cblock_alloc()
 >>>    function in /sys/kern/tty_subr.c), this was RELENG_6 as of 1-Feb-2007,
 >>>> Fix:
 >>>    I'm ready to provide further debugging information on this issue.
 >>>    Unfortunately, I'm not familiar enough with the locking concepts
 >>>    in modern FreeBSD kernels (and in tty subsystem particularly)
 >>>    in order to make the fix myself.
 >> 
 >> Tty locking is especially simple and not very good -- everything must
 >> be Giant-locked to work.  However, the default for network drivers is
 >> now not to use Giant locking.  ppp doesn't seem to be aware of this.
 >
 >  I was under the wrong impression that the addition of IFF_NEEDSGIANT to
 > ppp's ifp->if_flags (made 30-Mar-2006 by rwatson@) cured exactly this 
 > problem. But apparently it's not enough: simple addition of the 
 > GIANT_REQUIRED at start of the cblock_alloc() reveals that Giant isn't held 
 > at this point.
 
 Ah, I remembered that something was supposed to be fix but couldn't find
 the details since I only looked in ppp files.
 
 >> The only simple fix seems to be to pessimize all network drivers by
 >> configuring Giant locking for them all -- see netisr.c.  I'm not sure
 >> if this is enough -- it is probably necessary to Giant-lock all calls
 >> into ppp (especially ioctls), but things in netisr.c only logically
 >> affect isrs.
 
 Hopefully IFF_NEEDSGIANT for some interfaces makes pessimizing all of them
 unnecessary.  But IIRC the support for the MP-unsafe interfaces is supposed
 to go away soon.
 
 >  I've tried to solve the problem from the other end. Upon some analysis
 > of the ppp(4) code I concluded that: a) all calls of the tty functions
 > are concentrated in the single ppp(4) source file ppp_tty.c, and
 > b) _almost_ all these calls are protected by the now dummy pair of function
 > calls "s = spltty(); / splx(s);". So I went ahead and converted those
 > pairs to the conditional Giant acquisition:
 
 I think some protection was also provided by splimp(), but that is in the
 network-specific parts (if_ppp.c) and is now more likely than the tty-specific
 parts to be coverted by Giant/IFF_NEEDSGIANT.
 
 > --- ppp_tty.c.orig	Tue Dec 12 23:56:22 2006
 > +++ ppp_tty.c	Wed Feb 21 21:39:50 2007
 > @@ -86,6 +86,8 @@
 > #include <sys/tty.h>
 > #include <sys/conf.h>
 > #include <sys/uio.h>
 > +#include <sys/lock.h>
 > +#include <sys/mutex.h>
 >
 > #ifdef PPP_FILTER
 > #include <net/bpf.h>
 > @@ -93,6 +95,12 @@
 > #include <net/if_ppp.h>
 > #include <net/if_pppvar.h>
 >
 > +#undef spltty
 > +#undef splx
 > +#define spltty()	mtx_owned(&Giant) ? 0 : (mtx_lock(&Giant), 1)
 
 You could alos change this to GIANT_REQUIRED to find cases where
 IFF_NEEDSGIANT is helping.
 
 > +#define splx(relflg)	if (relflg) mtx_unlock(&Giant)
 > +
 > +
 > static int	pppopen(struct cdev *dev, struct tty *tp);
 > static int	pppclose(struct tty *tp, int flag);
 > static int	pppread(struct tty *tp, struct uio *uio, int flag);
 > @@ -556,7 +564,9 @@
 > 	    /* XXX as above. */
 > 	    if (CCOUNT(&tp->t_outq) == 0) {
 > 		++sc->sc_stats.ppp_obytes;
 > +		s = spltty();
 > 		(void) putc(PPP_FLAG, &tp->t_outq);
 > +		splx(s);
 > 	    }
 >
 > 	    /* Calculate the FCS for the first mbuf's worth. */
 > @@ -579,7 +589,9 @@
 > 		n = cp - start;
 > 		if (n) {
 > 		    /* NetBSD (0.9 or later), 4.3-Reno or similar. */
 > +		    s = spltty();
 > 		    ndone = n - b_to_q(start, n, &tp->t_outq);
 > +		    splx(s);
 > 		    len -= ndone;
 > 		    start += ndone;
 > 		    sc->sc_stats.ppp_obytes += ndone;
 
 These above is in pppasyncstart().  Apparently, IFF_NEEDSGIANT doesn't
 help for it (or ip_output()?).  Not having spls in the above at first
 looks a bug in the old code, but actually, putc() and b_to_q() call
 spltty() internally so spltty() was only needed here when more than
 one thing needs to be protected by it.  The fine-grained locking here
 might have been a bad method even with spls (it already gave the bugs
 documented in the XXX comments).
 
 > This actually insures that Giant is held within tty layer (GIANT_REQUIRED
 > check in cblock_alloc() confirms this), but this causes lock order reversals 
 > like these:
 >
 > lock order reversal: (Giant after non-sleepable)
 > 1st 0xc39904c8 inp (rawinp) @ /usr/RELENG_6/src/sys/netinet/raw_ip.c:289
 > 2nd 0xc06488a0 Giant (Giant) @ /usr/RELENG_6/src/sys/net/ppp_tty.c:567
 > KDB: stack backtrace:
 > kdb_backtrace(0,ffffffff,c06556a0,c0657ba8,c0625fe4,...) at 0xc04b2f59 = 
 > kdb_backtrace+0x29
 > witness_checkorder(c06488a0,9,c05fff6e,237) at 0xc04bdb00 = 
 > witness_checkorder+0x578
 > _mtx_lock_flags(c06488a0,0,c05fff6e,237) at 0xc0493180 = _mtx_lock_flags+0x78
 > pppasyncstart(c35ab800,c3692d0c,0,c05ffb7e,3e3) at 0xc050c6e4 = 
 > pppasyncstart+0x90
 > pppoutput(c3692c00,c3804700,c39764d0,c3987bdc,0,...) at 0xc0509d7a = 
 > pppoutput+0x326
 > ip_output(c3804700,0,dae08b24,20,0,...) at 0xc052a764 = ip_output+0xa64
 > rip_output(c3804700,c3a8b000,12e6cc1,40,c3804700,...) at 0xc052cb3b = 
 > rip_output+0x29f
 > rip_send(c3a8b000,0,c3804700,c3a45470,0,...) at 0xc052d62f = rip_send+0x93
 > sosend(c3a8b000,c3a45470,dae08c3c,c3804700,0,0,c6895780) at 0xc04d4ff7 = 
 > sosend+0x5eb
 > kern_sendit(c6895780,3,dae08cbc,0,0,0) at 0xc04da28c = kern_sendit+0x104
 > sendit(c6895780,3,dae08cbc,0,804e0b4,...) at 0xc04da15f = sendit+0x163
 > sendto(c6895780,dae08d04) at 0xc04da3bd = sendto+0x4d
 > syscall(3b,3b,3b,804e074,40,...) at 0xc05ca317 = syscall+0x22f
 > Xint0x80_syscall() at 0xc05b89af = Xint0x80_syscall+0x1f
 > --- syscall (133, FreeBSD ELF32, sendto), eip = 0x481401eb, esp = 0xbfbee90c, 
 > ebp = 0xbfbee958 ---
 >
 > lock order reversal: (Giant after non-sleepable)
 > 1st 0xc398457c inp (udpinp) @ /usr/RELENG_6/src/sys/netinet/udp_usrreq.c:804
 > 2nd 0xc06488a0 Giant (Giant) @ /usr/RELENG_6/src/sys/net/ppp_tty.c:567
 > KDB: stack backtrace:
 > kdb_backtrace(0,ffffffff,c0657838,c0657ba8,c0625fe4,...) at 0xc04b2f59 = 
 > kdb_backtrace+0x29
 > witness_checkorder(c06488a0,9,c05fff6e,237) at 0xc04bdb00 = 
 > witness_checkorder+0x578
 > _mtx_lock_flags(c06488a0,0,c05fff6e,237) at 0xc0493180 = _mtx_lock_flags+0x78
 > pppasyncstart(c35ab800,c3692d0c,0,c05ffb7e,3e3) at 0xc050c6e4 = 
 > pppasyncstart+0x90
 > pppoutput(c3692c00,c3803900,c39764d0,c3987bdc,0,...) at 0xc0509d7a = 
 > pppoutput+0x326
 > ip_output(c3803900,0,dae02b00,0,0,...) at 0xc052a764 = ip_output+0xa64
 > udp_output(c39844ec,c3803900,c384a330,0,c6895480,...) at 0xc053a5b2 = 
 > udp_output+0x4b6
 > udp_send(c3a8b000,0,c3803900,c384a330,0,...) at 0xc053ab0a = udp_send+0x1a
 > sosend(c3a8b000,c384a330,dae02c3c,c3803900,0,0,c6895480) at 0xc04d4ff7 = 
 > sosend+0x5eb
 > kern_sendit(c6895480,4,dae02cbc,0,0,0) at 0xc04da28c = kern_sendit+0x104
 > sendit(c6895480,4,dae02cbc,0,bfbfe890,...) at 0xc04da15f = sendit+0x163
 > sendto(c6895480,dae02d04) at 0xc04da3bd = sendto+0x4d
 > syscall(3b,3b,3b,bfbfed4c,bfbfe860,...) at 0xc05ca317 = syscall+0x22f
 > Xint0x80_syscall() at 0xc05b89af = Xint0x80_syscall+0x1f
 > --- syscall (133, FreeBSD ELF32, sendto), eip = 0x481341eb, esp = 0xbfbfe80c, 
 > ebp = 0xbfbfe838 ---
 >
 > So, is it possible to avoid these LORs w/o massive ppp/tty/network code
 > rewriting?
 
 I think Giant needs to be acquired early in the networking code for
 if_output like it is for if_start.  It seems to only be be acquired for
 if_start and if_ioctl, and not for if_output, if_input, if_watchdog,
 if_init or if_resolvemulti.  ppp doesn't use any of the others or
 if_start.  Slip uses if_start and seems to have the same problem with
 if_output.
 
 Bruce
>Unformatted:
