From nobody@FreeBSD.org  Wed Jan 30 16:59:00 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	by hub.freebsd.org (Postfix) with ESMTP id 69642D9E
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 30 Jan 2013 16:59:00 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 3D9D08CD
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 30 Jan 2013 16:59:00 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.5/8.14.5) with ESMTP id r0UGwxer074813
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 30 Jan 2013 16:58:59 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.5/8.14.5/Submit) id r0UGwx6n074812;
	Wed, 30 Jan 2013 16:58:59 GMT
	(envelope-from nobody)
Message-Id: <201301301658.r0UGwx6n074812@red.freebsd.org>
Date: Wed, 30 Jan 2013 16:58:59 GMT
From: Alan Somers <asomers@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: "panic: process not held" when invoking userland dtrace with INVARIANTS enabled
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         175705
>Category:       kern
>Synopsis:       [dtrace] "panic: process not held" when invoking userland dtrace with INVARIANTS enabled
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    markj
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jan 30 17:00:00 UTC 2013
>Closed-Date:    Thu Mar 27 00:35:39 UTC 2014
>Last-Modified:  Thu Mar 27 00:35:39 UTC 2014
>Originator:     Alan Somers
>Release:        9.1
>Organization:
Spectralogic
>Environment:
FreeBSD testy 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #2: Thu Jan 24 19:34:34 MST 2013     root@****:/tb/3a9569b9/tinderbuild/jails/9/obj/net/nexus/scratch/cruise/projects/Strata-debug/SpectraBSD/sys/STRATA_DBG  amd64

>Description:
I tried using userland dtrace exactly according to the instructions at https://wiki.freebsd.org/DTrace/userland.  The following panic resulted:

panic: process not held
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff8036266a = db_trace_self_wrapper+0x2a
kdb_backtrace() at 0xffffffff8090c4b7 = kdb_backtrace+0x37
panic() at 0xffffffff808d4a48 = panic+0x1d8
proc_ops() at 0xffffffff8232a03a = proc_ops+0x8a
fasttrap_pid_probe() at 0xffffffff8232b319 = fasttrap_pid_probe+0x9a9
trap() at 0xffffffff80bd2e85 = trap+0x2c5
calltrap() at 0xffffffff80bbcec3 = calltrap+0x8
--- trap 0x3, rip = 0x80086a911, rsp = 0x7fffffffdb98, rbp = 0x7fffffffdba0 ---
KDB: enter: panic
[ thread pid 40555 tid 114882 ]
Stopped at      0xffffffff8090c17b = kdb_enter+0x3b:    movq    $0,0xb4d1a2(%rip)


Looking at the source, it seems that fasttrap_isa.c acquires a process lock at line 1036, drops it at line 1184, then calls uwrite at line 1737.  uwrite calls proc_ops() which panics because it expects a process lock to be held.  The subversion history shows that this bug was introduced in change 227291, which replaced fasttrap_copyout (which does not require a process lock) with uwrite (which does).  It probably wasn't caught at the time because nobody tested it with INVARIANTS.
>How-To-Repeat:
1) Load a kernel with INVARIANTS enabled. 
2) kldload dtraceall
3) Try to dtrace a user process according to the instructions at https://wiki.freebsd.org/DTrace/userland
>Fix:
Possibly could be fixed be reacquiring the process lock before calling uwrite().

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->markj 
Responsible-Changed-By: markj 
Responsible-Changed-When: Sun Dec 29 18:27:19 UTC 2013 
Responsible-Changed-Why:  
Track. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175705 
State-Changed-From-To: open->feedback 
State-Changed-By: markj 
State-Changed-When: Sun Dec 29 18:28:02 UTC 2013 
State-Changed-Why:  
Cannot reproduce on CURRENT. Waiting for the submitter to confirm/deny 
that it's still an issue (see the followup). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175705 

From: Mark Johnston <markj@freebsd.org>
To: bug-followup@FreeBSD.org, Alan Somers <asomers@FreeBSD.org>
Cc:  
Subject: Re: kern/175705: [dtrace] &quot;panic: process not held&quot; when
 invoking userland dtrace with INVARIANTS enabled
Date: Sun, 29 Dec 2013 13:26:54 -0500

 Is this crash still occurring for you? I cannot reproduce it on
 CURRENT; the uwrite is executed when fasttrap needs to execute an
 instruction that it doesn't know how to emulate, so it copies it into
 scratch space (the FASTTRAP_T_COMMON case). On amd64, memcpy() is a
 function that is traced this way, and I cannot trigger a panic by
 tracing it. (It currently does not work properly, but that's a
 separate issue which I'm working on.)
 
 It seems that proc_rwmem is asserting that p->p_lock > 0, not that the
 process lock is held. proc_ops() calls PHOLD before the proc_rwmem()
 call, so I don't see how this assertion could fail.
 
 -Mark

From: Alan Somers <asomers@freebsd.org>
To: Mark Johnston <markj@freebsd.org>
Cc: bug-followup@freebsd.org, Alan Somers <asomers@freebsd.org>
Subject: Re: kern/175705: [dtrace] &quot;panic: process not held&quot; when
 invoking userland dtrace with INVARIANTS enabled
Date: Sun, 29 Dec 2013 19:51:14 -0700

 On Sun, Dec 29, 2013 at 11:26 AM, Mark Johnston <markj@freebsd.org> wrote:
 > Is this crash still occurring for you? I cannot reproduce it on
 > CURRENT; the uwrite is executed when fasttrap needs to execute an
 > instruction that it doesn't know how to emulate, so it copies it into
 > scratch space (the FASTTRAP_T_COMMON case). On amd64, memcpy() is a
 > function that is traced this way, and I cannot trigger a panic by
 > tracing it. (It currently does not work properly, but that's a
 > separate issue which I'm working on.)
 
 I haven't tried since I posted the bug.  I'll test again when I get
 back to work.  That will be on Friday, but it may have to wait for the
 following week if I'm too backlogged.
 
 -Alan
 
 >
 > It seems that proc_rwmem is asserting that p->p_lock > 0, not that the
 > process lock is held. proc_ops() calls PHOLD before the proc_rwmem()
 > call, so I don't see how this assertion could fail.
 >
 > -Mark

From: Alan Somers <asomers@freebsd.org>
To: Cc: Mark Johnston <markj@freebsd.org>, bug-followup@freebsd.org
Subject: Re: kern/175705: [dtrace] &quot;panic: process not held&quot; when
 invoking userland dtrace with INVARIANTS enabled
Date: Fri, 3 Jan 2014 16:59:22 -0700

 On Sun, Dec 29, 2013 at 7:51 PM, Alan Somers <asomers@freebsd.org> wrote:
 > On Sun, Dec 29, 2013 at 11:26 AM, Mark Johnston <markj@freebsd.org> wrote:
 >> Is this crash still occurring for you? I cannot reproduce it on
 >> CURRENT; the uwrite is executed when fasttrap needs to execute an
 >> instruction that it doesn't know how to emulate, so it copies it into
 >> scratch space (the FASTTRAP_T_COMMON case). On amd64, memcpy() is a
 >> function that is traced this way, and I cannot trigger a panic by
 >> tracing it. (It currently does not work properly, but that's a
 >> separate issue which I'm working on.)
 
 I can no longer reproduce it on head.  I haven't tried on stable/10 or
 stable/9 yet.
State-Changed-From-To: feedback->patched 
State-Changed-By: markj 
State-Changed-When: Sun Jan 5 03:38:27 UTC 2014 
State-Changed-Why:  
No longer reproducible on CURRENT. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175705 
State-Changed-From-To: patched->closed 
State-Changed-By: markj 
State-Changed-When: Thu Mar 27 00:35:39 UTC 2014 
State-Changed-Why:  
Alan has reported that the problem is no longer reproducible on stable/9 
or stable/10. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=175705 
>Unformatted:
