From nobody@FreeBSD.org  Wed Feb 29 01:14:34 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id E91331065679
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 29 Feb 2012 01:14:33 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id D508B8FC17
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 29 Feb 2012 01:14:33 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q1T1EX2K009934
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 29 Feb 2012 01:14:33 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q1T1EXZ2009933;
	Wed, 29 Feb 2012 01:14:33 GMT
	(envelope-from nobody)
Message-Id: <201202290114.q1T1EXZ2009933@red.freebsd.org>
Date: Wed, 29 Feb 2012 01:14:33 GMT
From: Marc Abramowitz <marc@marc-abramowitz.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Kernel panic when dtracing userland
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         165541
>Category:       kern
>Synopsis:       [dtrace] [panic] Kernel panic when dtracing userland
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    gnn
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Feb 29 01:20:11 UTC 2012
>Closed-Date:    Fri Apr 13 20:54:39 UTC 2012
>Last-Modified:  Tue Jul  3 16:20:04 UTC 2012
>Originator:     Marc Abramowitz
>Release:        9.0
>Organization:
>Environment:
FreeBSD freebsd9-0.localdomain 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Mon Feb 27 00:16:24 PST 2012    root@freebsd9-0.localdomain:/usr/obj/usr/src/sys/DTRACE amd64
>Description:
When I dtrace even a trivial program in certain ways, the kernel panics when the traced process exits.

I'm using FreeBSD 9.0 on amd64 (I also later reproduced the problem with i386 as well) in VMware Fusion 4.1.1 on OS X 10.6.8 and trying to DTrace userland programs.

I recompiled my kernel and world, following the instructions at:

http://wiki.freebsd.org/DTrace
http://wiki.freebsd.org/DTrace/userland

The test.c pid provider example worked fine for me:

$ sudo dtrace -s pid.d -c ./test
dtrace: script 'pid.d' matched 2 probes
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry
  0  43031                      sleep:entry
  0  43031                      sleep:entry
  0  43031                      sleep:entry

As does a simple probe of test.c specified with the -n option:

[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test
dtrace: description 'pid$target:test:main:entry' matched 1 probe
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry

When I start trying to dtrace other programs, things don't go so well.

DTracing very simple programs causes a kernel panic when the process exits. For example:

[marca at freebsd9-0 ~]$ sudo kldload dtraceall
[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt'
(kernel panic!)

According to the core.txt file, it was a "Fatal trap 10: trace trap while
in kernel mode" and here's the KDB backtrace:

KDB: stack backtrace:
#0 0xffffffff8089025e at kdb_backtrace+0x5e
#1 0xffffffff80858ce7 at panic+0x187
#2 0xffffffff80b4bf20 at trap_fatal+0x290
#3 0xffffffff80b4c540 at trap+0x180
#4 0xffffffff80b36963 at calltrap+0x8
#5 0xffffffff8162583d at dtrace_assfail+0x2d
#6 0xffffffff8188aa2e at fasttrap_provider_free+0x1de
#7 0xffffffff8188ad13 at fasttrap_pid_cleanup_cb+0x1c3
#8 0xffffffff8086dfa1 at softclock+0x3a1
#9 0xffffffff8082d724 at intr_event_execute_handlers+0x104
#10 0xffffffff8082eee4 at ithread_loop+0xa4
#11 0xffffffff8082a34f at fork_exit+0x11f
#12 0xffffffff80b36e8e at fork_trampoline+0xe
>How-To-Repeat:
[marca at freebsd9-0 ~]$ sudo kldload dtraceall
[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:cat:main:entry' -c '/bin/cat hello_world.txt'

- or -

[marca at freebsd9-0 ~]$ sudo kldload dtraceall
[marca at freebsd9-0 ~]$ cat -n test.c
     1 #include <stdio.h>
     2
     3 int main()
     4 {
     5    sleep(15);
     6
     7    FILE *fp = fopen("hello.txt", "w");
     8    fprintf(fp, "Here I am at %s:%d.\n", __FILE__, __LINE__);
     9    fclose(fp);
    10 }
[marca at freebsd9-0 ~]$ gcc test.c -o test
[marca at freebsd9-0 ~]$ sudo dtrace -n 'pid$target:test:main:entry' -c ./test
dtrace: description 'pid$target:test:main:entry' matched 1 probe
dtrace: buffer size lowered to 1m
CPU     ID                    FUNCTION:NAME
  0  43030                       main:entry
(Kernel panic!  After reboot....)
[marca at freebsd9-0 ~]$ cat hello.txt
Here I am at test.c:8.

Interestingly, the crash doesn't occur until after the sleep and the
fprintf call, so it looks the kernel panic happens as a result of the
traced process _exiting_...

I reproduced this on both the amd64 and i386 architectures.
>Fix:


>Release-Note:
>Audit-Trail:

From: Michael Gmelin <freebsd@grem.de>
To: bug-followup@FreeBSD.org,
 marc@marc-abramowitz.com
Cc:  
Subject: Re: kern/165541: [dtrace] [panic] Kernel panic when dtracing userland
Date: Thu, 8 Mar 2012 18:11:23 +0100

 I can confirm this behavior running FBSD 9.0 amd64 in a non-virtual =
 environment (intel). No kernel crashes so far, but every program I try =
 to dtrace using the pid provider simply exits (mostly signal 9).=
State-Changed-From-To: open->closed 
State-Changed-By: gnn 
State-Changed-When: Fri Apr 13 20:54:08 UTC 2012 
State-Changed-Why:  

Could not reproduce on HEAD as of 13 April 2012 


Responsible-Changed-From-To: freebsd-bugs->gnn 
Responsible-Changed-By: gnn 
Responsible-Changed-When: Fri Apr 13 20:54:08 UTC 2012 
Responsible-Changed-Why:  
Could not reproduce on HEAD as of 13 April 2012 

http://www.freebsd.org/cgi/query-pr.cgi?pr=165541 

From: Marc Abramowitz <msabramo@gmail.com>
To: "bug-followup@FreeBSD.org" <bug-followup@FreeBSD.org>,
 "marc@marc-abramowitz.com" <marc@marc-abramowitz.com>
Cc:  
Subject: Re: kern/165541: [dtrace] [panic] Kernel panic when dtracing userland
Date: Tue, 3 Jul 2012 09:18:41 -0700

 Wanted to test this again. Couldn't find a FreeBSD 9.0 snapshot -- do I need=
  to cvsup and build from source?
 
 -Marc
 http://marc-abramowitz.com
 Sent from my iPhone 4
 
>Unformatted:
