From ruben@helium.verweg.com  Sun Mar  8 10:49:21 2009
Return-Path: <ruben@helium.verweg.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 92937106564A
	for <FreeBSD-gnats-submit@freebsd.org>; Sun,  8 Mar 2009 10:49:21 +0000 (UTC)
	(envelope-from ruben@helium.verweg.com)
Received: from helium.verweg.com (tunnel41.ipv6.xs4all.nl [IPv6:2001:888:10:29::2])
	by mx1.freebsd.org (Postfix) with ESMTP id ECBFD8FC1E
	for <FreeBSD-gnats-submit@freebsd.org>; Sun,  8 Mar 2009 10:49:20 +0000 (UTC)
	(envelope-from ruben@helium.verweg.com)
Received: from helium.verweg.com (ruben@localhost.verweg.com [127.0.0.1])
	by helium.verweg.com (8.14.3/8.14.1) with ESMTP id n28AnEjF004730
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 8 Mar 2009 11:49:19 +0100 (CET)
	(envelope-from ruben@helium.verweg.com)
Received: (from ruben@localhost)
	by helium.verweg.com (8.14.3/8.14.1/Submit) id n28AnDuw004729;
	Sun, 8 Mar 2009 11:49:13 +0100 (CET)
	(envelope-from ruben)
Message-Id: <200903081049.n28AnDuw004729@helium.verweg.com>
Date: Sun, 8 Mar 2009 11:49:13 +0100 (CET)
From: Ruben van Staveren <ruben@verweg.com>
Reply-To: Ruben van Staveren <ruben@verweg.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: panic sleeping thread after 25th Feb src/sys/net commits
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         132404
>Category:       kern
>Synopsis:       panic sleeping thread after 25th Feb src/sys/net commits
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    rwatson
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Mar 08 10:50:04 UTC 2009
>Closed-Date:    Sun Mar 08 14:04:27 UTC 2009
>Last-Modified:  Sun Mar 08 14:04:27 UTC 2009
>Originator:     Ruben van Staveren
>Release:        FreeBSD 7.1-STABLE i386
>Organization:
>Environment:
System: FreeBSD helium.verweg.com 7.1-STABLE FreeBSD 7.1-STABLE #55: Sat Mar 7 20:49:56 CET 2009 root@helium.verweg.com:/usr/obj/usr/cvsup/7-stable/src/sys/HELIUM-SMP i386


	
>Description:

cvsupping to feb 25th makes the system panic during early boot, somewhere
around the time when /etc/rc.d/ppp and /etc/rc.d/routing is started. After
enabling witness it will panic with a witness_warn

Maybe some of it is related to kern/129846


>How-To-Repeat:

This happens while using IPv6, pf, ppp (during boot to connect ADSL) using the following versions

     $FreeBSD: src/sys/net/route.c,v 1.120.2.9 2009/02/25 11:18:18 rwatson Exp $
     $FreeBSD: src/sys/net/rtsock.c,v 1.143.2.8 2009/02/25 10:59:56 rwatson Exp $
     $FreeBSD: src/sys/net/route.h,v 1.65.2.5 2009/02/25 10:59:56 rwatson Exp $
     $FreeBSD: src/sys/netinet6/in6_rmx.c,v 1.18.2.4 2009/02/25 10:59:56 rwatson Exp $
     $FreeBSD: src/sys/netinet6/in6_ifattach.c,v 1.39.2.4 2009/02/25 10:59:56 rwatson Exp $
     $FreeBSD: src/sys/netinet/in_rmx.c,v 1.57.2.3 2009/02/25 10:59:56 rwatson Exp $

The following textdumps are available:

Without WITNESS enabled (so empty show locks/alllocks)

1. http://ruben.is.verweg.com/stuff/freebsd/panic-sleepable-thread-textdump.tar.301

After enabling WITNESS and WITNESS_KDB

2. http://ruben.is.verweg.com/stuff/freebsd/witness_checkorder-textdump.tar.302

After setting debug.witness.kdb=0, to get the actual panic again

3. http://ruben.is.verweg.com/stuff/freebsd/panic-witness_warn-textdump.tar.303


>Fix:

Backing out those changes prevents the system from panicing

These revisions are safe

     $FreeBSD: src/sys/net/route.c,v 1.120.2.7 2008/11/06 22:11:57 julian Exp $
     $FreeBSD: src/sys/net/rtsock.c,v 1.143.2.7 2009/02/18 20:12:08 jamie Exp $
     $FreeBSD: src/sys/net/route.h,v 1.65.2.4 2008/11/06 22:11:57 julian Exp $
     $FreeBSD: src/sys/netinet6/in6_rmx.c,v 1.18.2.3 2008/09/01 19:23:04 obrien Exp $
     $FreeBSD: src/sys/netinet6/in6_ifattach.c,v 1.39.2.3 2008/09/01 19:23:04 obrien Exp $
     $FreeBSD: src/sys/netinet/in_rmx.c,v 1.57.2.2 2008/11/06 22:11:57 julian Exp $
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->rwatson 
Responsible-Changed-By: rwatson 
Responsible-Changed-When: Sun Mar 8 11:25:49 UTC 2009 
Responsible-Changed-Why:  
Take ownership since I'm working on this bug. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=132404 
State-Changed-From-To: open->analyzed 
State-Changed-By: rwatson 
State-Changed-When: Sun Mar 8 11:28:04 UTC 2009 
State-Changed-Why:  
Transition to analyzed: this panic may be the result of a missed MFC of 
r186061, which corrected a leaked lock in routing socket error handling. 
I've merged the commit to stable/7; hopefully it will appear in this PR 
shortly.  Please let me know if it resolves the problem and follow up on 
this PR with that information.  Thanks! 


http://www.freebsd.org/cgi/query-pr.cgi?pr=132404 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/132404: commit references a PR
Date: Sun,  8 Mar 2009 11:21:09 +0000 (UTC)

 Author: rwatson
 Date: Sun Mar  8 11:20:54 2009
 New Revision: 189531
 URL: http://svn.freebsd.org/changeset/base/189531
 
 Log:
   Merge missed routing lock fix r186061 from head to stable/7:
   
     Dont leak the rnh lock on error.
   
   Original change was from thompsa.  This may correct routing-related panics
   seen by uses of ppp, including the following PRs:
   
   PR:		132215, 132222, 132404
   Reported by:	Ethan <hsiao.ethan at gmail.com>
 
 Modified:
   stable/7/sys/   (props changed)
   stable/7/sys/contrib/pf/   (props changed)
   stable/7/sys/dev/ath/ath_hal/   (props changed)
   stable/7/sys/dev/cxgb/   (props changed)
   stable/7/sys/net/rtsock.c
 
 Modified: stable/7/sys/net/rtsock.c
 ==============================================================================
 --- stable/7/sys/net/rtsock.c	Sun Mar  8 11:12:23 2009	(r189530)
 +++ stable/7/sys/net/rtsock.c	Sun Mar  8 11:20:54 2009	(r189531)
 @@ -629,10 +629,10 @@ route_output(struct mbuf *m, struct sock
  				       rt->rt_ifa->ifa_addr))) {
  				RT_UNLOCK(rt);
  				RADIX_NODE_HEAD_LOCK(rnh);
 -				if ((error = rt_getifa_fib(&info,
 -				    rt->rt_fibnum)) != 0)
 -					senderr(error);
 +				error = rt_getifa_fib(&info, rt->rt_fibnum);
  				RADIX_NODE_HEAD_UNLOCK(rnh);
 +				if (error != 0)
 +					senderr(error);
  				RT_LOCK(rt);
  			}
  			if (info.rti_ifa != NULL &&
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: analyzed->patched 
State-Changed-By: rwatson 
State-Changed-When: Sun Mar 8 12:11:05 UTC 2009 
State-Changed-Why:  
Change r189531 is believed to correct this problem, could you confirm? 


http://www.freebsd.org/cgi/query-pr.cgi?pr=132404 

From: Ruben van Staveren <ruben@verweg.com>
To: rwatson@FreeBSD.org
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: kern/132404: panic sleeping thread after 25th Feb src/sys/net commits
Date: Sun, 8 Mar 2009 14:22:43 +0100

 Hi Robert,
 
 On 8 Mar 2009, at 13:11, rwatson@FreeBSD.org wrote:
 
 > Synopsis: panic sleeping thread after 25th Feb src/sys/net commits
 >
 > State-Changed-From-To: analyzed->patched
 > State-Changed-By: rwatson
 > State-Changed-When: Sun Mar 8 12:11:05 UTC 2009
 > State-Changed-Why:
 > Change r189531 is believed to correct this problem, could you confirm?
 >
 >
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=132404
 
 Using rev 1.143.2.9 of src/sys/net/rtsock.c indeed solves the problem,  
 thanks!
 
 Regards,
 	Ruben
State-Changed-From-To: patched->closed 
State-Changed-By: rwatson 
State-Changed-When: Sun Mar 8 14:04:04 UTC 2009 
State-Changed-Why:  
Closed as now believed fixed; thanks for the report!  If you run into further 
problems, please let me know. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=132404 
>Unformatted:
