From anholt@leguin.anholt.net  Mon Dec 12 05:02:30 2005
Return-Path: <anholt@leguin.anholt.net>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id A8BDE16A420
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 12 Dec 2005 05:02:30 +0000 (GMT)
	(envelope-from anholt@leguin.anholt.net)
Received: from leguin.anholt.net (69-30-77-85.dq1sn.easystreet.com [69.30.77.85])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 89E4843D4C
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 12 Dec 2005 05:02:29 +0000 (GMT)
	(envelope-from anholt@leguin.anholt.net)
Received: from leguin.anholt.net (localhost [127.0.0.1])
	by leguin.anholt.net (8.13.4/8.13.1) with ESMTP id jBC52SVl001399
	for <FreeBSD-gnats-submit@freebsd.org>; Sun, 11 Dec 2005 21:02:28 -0800 (PST)
	(envelope-from anholt@leguin.anholt.net)
Received: (from anholt@localhost)
	by leguin.anholt.net (8.13.4/8.13.1/Submit) id jBC52RUp001398;
	Sun, 11 Dec 2005 21:02:27 -0800 (PST)
	(envelope-from anholt)
Message-Id: <200512120502.jBC52RUp001398@leguin.anholt.net>
Date: Sun, 11 Dec 2005 21:02:27 -0800 (PST)
From: Eric Anholt <anholt@FreeBSD.org>
Reply-To: Eric Anholt <anholt@FreeBSD.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: panic in pmc sampling mode
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         90269
>Category:       kern
>Synopsis:       [pmc] [panic] panic in pmc sampling mode
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    jkoshy
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Dec 12 05:10:02 GMT 2005
>Closed-Date:    Tue Mar 28 01:13:28 GMT 2006
>Last-Modified:  Tue Mar 28 01:13:28 GMT 2006
>Originator:     Eric Anholt
>Release:        FreeBSD 7.0-CURRENT amd64
>Organization:
>Environment:
System: FreeBSD leguin.anholt.net 7.0-CURRENT FreeBSD 7.0-CURRENT #22: Thu Dec 1 01:48:14 PST 2005 anholt@leguin.anholt.net:/usr/obj/usr/src/current/sys/MYKERNEL amd64


	
>Description:
	Attempting to use pmc to profile a library I'm working on results in
	panics.  While I couldn't get a dump, here's the backtrace:

	kdb_enter() at kdb_enter+0x2f
	panic() at panic+0x249
	pmc_release_pmc_descriptor() at pmc_release_pmc_descriptor+0x2eb
	pmc_syscall_handler() at pmc_syscall_handler+0x41a
	syscall() at syscall+0x217
	Xfast_syscall() at Xfast_syscall+0xa8

	I didn't get the panic str, it looks like.  Started my minicom log too
	late.  I'll get that later by reproducing, but wanted to get the bug
	filed for now.
>How-To-Repeat:
	Install liboil from ports
	Download and build libpcg from my website:
		http://www.anholt.net/libpcg/release/libpcg-0.2.1.tar.bz2
	Try to run a shell script containing:
		pmcstat -S instructions -O samples.out ./tests/timing
		pmcstat -R samples.out -g
	Watch your serial console.  It'll run for a couple of seconds, but then
	die (perhaps around the time that the program should be ending).

	I suspect the actual list of steps is not so specific, but it's what I'm
	using (modulo that I'm using current libpcg sources that I'm working on
	making faster :)
>Fix:

	


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->jkoshy 
Responsible-Changed-By: jkoshy 
Responsible-Changed-When: Mon Dec 12 14:17:30 UTC 2005 
Responsible-Changed-Why:  
Take ownership. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=90269 

From: Joseph Koshy <jkoshy@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/90269: [pmc] [panic] panic in pmc sampling mode
Date: Wed, 14 Dec 2005 13:53:04 GMT

 I'm unable to reproduce this panic on a UP amd64 box running 6.0-RC1.
 I will upgrade to -current and retry.
 
 Is this panic happening on a multi-cpu machine?  Could you post the
 output of dmesg?

From: Eric Anholt <eta@lclark.edu>
To: gnats <freebsd-gnats-submit@FreeBSD.ORG>
Cc:  
Subject: Re: kern/90269
Date: Mon, 19 Dec 2005 01:05:37 -0800

 Sorry, somehow I managed to send-pr without my email address set.  Boy,
 I hate this tool.  Hopefully it's fixed now.
 
 I've put my dmesg up at:
 http://people.freebsd.org/~anholt/leguin-dmesg.txt
 
 It's a UP machine, hwpmc says:
 hwpmc: TSC/1/0x20<REA> K8/4/0x1ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA>
 
 And I found the panic str in a /var/log/messages:
 Dec 11 18:43:13 leguin kernel: panic: Bad link elm 0xffffff000a081b00 prev->next != elm
 
 -- 
 Eric Anholt                                     eta@lclark.edu
 http://people.freebsd.org/~anholt/              anholt@FreeBSD.org

From: Joseph Koshy <jkoshy@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/90269: [pmc] [panic] panic in pmc sampling mode
Date: Mon, 19 Dec 2005 14:08:58 GMT

 > Dec 11 18:43:13 leguin kernel: panic: Bad link elm 0xffffff000a081b00 prev->next != elm
 
 This is a message from <sys/queue.h>, from QMD_LIST_CHECK_PREV(). We seem to have a race
 here-abouts:
 
    "sys/dev/hwpmc/hwpmc_mod.c":
    1961         /* unlink from the owner's list */
    1962         if (pm->pm_owner) {
    1963                 LIST_REMOVE(pm, pm_next);
    1964                 pm->pm_owner = NULL;
    1965         }
  
 

From: Eric Anholt <eta@lclark.edu>
To: gnats <freebsd-gnats-submit@FreeBSD.ORG>
Cc:  
Subject: Re: kern/90269
Date: Fri, 23 Dec 2005 23:25:24 -0800

 I tried dropping an SX_ASSERT(&pmc_sx, SX_XLOCKED) above the
 LIST_INSERT_HEAD and the LIST_REMOVE for that list (yes, it's that
 LIST_REMOVE you mentioned that's triggering it, according to objdump).
 No effect on the problem (INVARIANTS and _SUPPORT are both in the
 kernel).
 
 -- 
 Eric Anholt                                     eta@lclark.edu
 http://people.freebsd.org/~anholt/              anholt@FreeBSD.org
State-Changed-From-To: open->feedback 
State-Changed-By: jkoshy 
State-Changed-When: Thu Mar 9 02:08:35 UTC 2006 
State-Changed-Why:  
Revs 1.5 hwpmc_logging.c and 1.19 hwpmc_mod.c should fix this bug. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=90269 
State-Changed-From-To: feedback->closed 
State-Changed-By: jkoshy 
State-Changed-When: Tue Mar 28 01:11:15 UTC 2006 
State-Changed-Why:  
MFCed (r1.10.2.6 hwpmc_mod.c, r1.3.2.1 hwpmc_logging.c). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=90269 
>Unformatted:
