From nobody@FreeBSD.org  Thu Jan 18 14:03:02 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id AE8F716A415
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 18 Jan 2007 14:03:02 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [69.147.83.33])
	by mx1.freebsd.org (Postfix) with ESMTP id 92A0213C45E
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 18 Jan 2007 14:03:02 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id l0IE32lZ084953
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 18 Jan 2007 14:03:02 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id l0IE32ih084934;
	Thu, 18 Jan 2007 14:03:02 GMT
	(envelope-from nobody)
Message-Id: <200701181403.l0IE32ih084934@www.freebsd.org>
Date: Thu, 18 Jan 2007 14:03:02 GMT
From: SI<si@kbrod.net>
To: freebsd-gnats-submit@FreeBSD.org
Subject: PPPoE server machine kernel panic (maybe netgraph)
X-Send-Pr-Version: www-3.0

>Number:         108092
>Category:       kern
>Synopsis:       [panic] PPPoE server machine kernel panic (maybe netgraph)
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Jan 18 14:10:14 GMT 2007
>Closed-Date:    Sun May 18 16:33:08 UTC 2008
>Last-Modified:  Sun May 18 16:33:08 UTC 2008
>Originator:     SI
>Release:        5.5
>Organization:
Kbrod.net
>Environment:
FreeBSD ... 5.5-RELEASE FreeBSD 5.5-RELEASE #0: Wed Jan 17 15:40:05 EET 2007 ...:/usr/obj/usr/src/sys/MYKERNEL  i386
>Description:
Hello,
The problem machine is an IBM x3400 dual-core (with hyperthreading), which serves about 200 PPPoE users, acts as RADIUS and MySQL server. In the past the same work was done without any crashes by an old machine under FreeBSD 5.3.

So, the new machine crashes about 3 times a day. Here are the details:
Dump header from device /dev/ad0s1b
  Architecture: i386
  Architecture Version: 1
  Dump Length: 1073086464B (1023 MB)
  Blocksize: 512
  Dumptime: Thu Jan 18 14:16:46 2007
  Hostname: 
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 5.5-RELEASE #0: Wed Jan 17 15:40:05 EET
2007
    :/usr/obj/usr/src/sys/MYKERNEL
  Panic String: spin lock held too long
  Dump Parity: 743047805
  Bounds: 7
  Dump Status: good
ppoe1# kgdb kernel.debug /var/crash/vmcore.7 
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".

Unread portion of the kernel message buffer:
       t(   6$#
                                ( (
tp   1+$#T ( p
t 0 ( 1,$# ( 
t x
p
1M$$ ( 
               tH 

6n$ $, (  H
                 t   @6o$ $t (  
                                             t P H 6p$0$ (  
                                                                            t    6q$@$  (   
                                                                                                           th   6r$P$L ( h
    0t (
               
                .9$`$ (  
                                  @tp
                                        h
                                          H p$  
                                                       Pt@   N9$

#0  doadump () at pcpu.h:160
160             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) backtrace
#0  doadump () at pcpu.h:160
#1  0xc0507a65 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:412
#2  0xc0507d89 in panic (fmt=0xc06b353e "spin lock held too long") at /usr/src/sys/kern/kern_shutdown.c:568
#3  0xc04ff034 in _mtx_lock_spin (m=0xc2af0898, td=0xc2325300, opts=0, 
    file=0xc2aef4a8 "/usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c", line=1992)
    at /usr/src/sys/kern/kern_mutex.c:606
#4  0xc04fee18 in _mtx_lock_spin_flags (m=0xc2af0898, opts=0, 
    file=0xc2aef4a8 "/usr/src/sys/modules/netgraph/netgraph/../../../netgraph/ng_base.c", line=1992)
    at /usr/src/sys/kern/kern_mutex.c:377
#5  0xc2aeaf79 in ?? ()
#6  0xc2af0898 in ?? ()
#7  0x00000000 in ?? ()
#8  0xc2aef4a8 in ?? ()
#9  0x000007c8 in ?? ()
#10 0x00000040 in ?? ()
#11 0x00000000 in ?? ()
#12 0xc064fe3f in uma_zalloc_arg (zone=0xc2af0898, udata=0x0, flags=0) at /usr/src/sys/vm/uma_core.c:1856
#13 0xc2aec78b in ?? ()
#14 0xc31d0800 in ?? ()
#15 0x00000000 in ?? ()
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xc2aea4d4 in ?? ()
#19 0xc2915d00 in ?? ()
#20 0xc296f980 in ?? ()
#21 0x00000000 in ?? ()
#22 0xe4e58c88 in ?? ()
#23 0xc2aea5f9 in ?? ()
#24 0xc2af0840 in ?? ()
#25 0xc2af08e0 in ?? ()
#26 0xc2aea4d4 in ?? ()
#27 0x00000000 in ?? ()
#28 0x00000000 in ?? ()
#29 0xc296f980 in ?? ()
#30 0xc25eb180 in ?? ()
#31 0xe4e58cb0 in ?? ()
#32 0xc2aeb21f in ?? ()
#33 0xc2915d00 in ?? ()
#34 0xc296f980 in ?? ()
#35 0x00000000 in ?? ()
#36 0x00000000 in ?? ()
#37 0x00000000 in ?? ()
#38 0xc25eb180 in ?? ()
#39 0xc2915d00 in ?? ()
#40 0xc25eb180 in ?? ()
#41 0xe4e58ccc in ?? ()
#42 0xc2aec2ca in ?? ()
#43 0xc2915d00 in ?? ()
#44 0xc25eb180 in ?? ()
#45 0xc07455c8 in netisrs ()
---Type <return> to continue, or q <return> to quit--- 
#46 0x00000000 in ?? ()
#47 0x00000004 in ?? ()
#48 0xe4e58ce4 in ?? ()
#49 0xc057a4be in swi_net (dummy=0xc2af0840) at /usr/src/sys/net/netisr.c:346
Previous frame inner to this frame (corrupt stack?)
(kgdb)

We will appreciate your help to solve this problem.




>How-To-Repeat:

>Fix:

>Release-Note:
>Audit-Trail:

From: SI <si@kbrod.net>
To: bug-followup@FreeBSD.org, si@kbrod.net
Cc:  
Subject: Re: kern/108092: [panic] PPPoE server machine kernel panic (maybe netgraph)
Date: Fri, 19 Jan 2007 08:43:51 +0200

 Here are some additional info (excuse me for not providing details before).
 
 The system is running PPPOED in form: 
 /usr/libexec/pppoed -d -P /var/run/pppoed0.pid -a kbrod.ppoe0 -l pppoe-in bge0
 
 The installed software is the same as it was on the previous FreeBSD 5.3
 machine (of course with new versions).
 
 Thank you in advance
 _________________________________________________________
 This mail sent using V-webmail - http://www.v-webmail.org
 
State-Changed-From-To: open->closed 
State-Changed-By: mav 
State-Changed-When: Sun May 18 16:27:54 UTC 2008 
State-Changed-Why:  
This PR is probably outdated now. There was many changes in netgraph 
since that time. Present 6.x versions not use spinlocks there and HEAD 
lives completely without locks in that place. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=108092 
>Unformatted:
