From nobody@FreeBSD.org  Mon Dec 30 12:00:12 2013
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1])
	(using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by hub.freebsd.org (Postfix) with ESMTPS id 27140AA6
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 30 Dec 2013 12:00:12 +0000 (UTC)
Received: from oldred.freebsd.org (oldred.freebsd.org [IPv6:2001:1900:2254:206a::50:4])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.freebsd.org (Postfix) with ESMTPS id 140CB160C
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 30 Dec 2013 12:00:12 +0000 (UTC)
Received: from oldred.freebsd.org ([127.0.1.6])
	by oldred.freebsd.org (8.14.5/8.14.7) with ESMTP id rBUC0BH6015018
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 30 Dec 2013 12:00:11 GMT
	(envelope-from nobody@oldred.freebsd.org)
Received: (from nobody@localhost)
	by oldred.freebsd.org (8.14.5/8.14.5/Submit) id rBUC0B5X015000;
	Mon, 30 Dec 2013 12:00:11 GMT
	(envelope-from nobody)
Message-Id: <201312301200.rBUC0B5X015000@oldred.freebsd.org>
Date: Mon, 30 Dec 2013 12:00:11 GMT
From: Robert David <robert.david.public@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: Dtrace does not work on -stable/10
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         185290
>Category:       kern
>Synopsis:       [dtrace] Dtrace does not work on -stable/10
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    markj
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Dec 30 12:10:00 UTC 2013
>Closed-Date:    Thu Mar 27 02:17:47 UTC 2014
>Last-Modified:  Thu Mar 27 02:17:47 UTC 2014
>Originator:     Robert David
>Release:        stable/10 (10.0-PRERELEASE)
>Organization:
none
>Environment:
FreeBSD notebook.linsystem.net 10.0-PRERELEASE FreeBSD 10.0-PRERELEASE #4 53624fa(HEAD): Mon Dec 30 12:34:25 CET 2013     root@notebook.linsystem.net:/usr/obj/rpool/FREEBSD/src/sys/thinkpad_02  amd64

>Description:
All simple dtrace oneliners I tryed has the same output:

root@notebook ~ # dtrace -n 'syscall:::entry { @num[execname] = count(); }'
dtrace: invalid probe specifier syscall:::entry { @num[execname] = count(); }: "/usr/lib/dtrace/io.d", line 43: operator -> cannot be applied to a forward declaration: no struct devstat definition is available


>How-To-Repeat:
Try any dtrace oneliner on actual stable/10 branch.
>Fix:


>Release-Note:
>Audit-Trail:

From: John-Mark Gurney <jmg@funkthat.com>
To: Robert David <robert.david.public@gmail.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Mon, 30 Dec 2013 09:28:49 -0800

 Robert David wrote this message on Mon, Dec 30, 2013 at 12:00 +0000:
 > >Description:
 > All simple dtrace oneliners I tryed has the same output:
 > 
 > root@notebook ~ # dtrace -n 'syscall:::entry { @num[execname] = count(); }'
 > dtrace: invalid probe specifier syscall:::entry { @num[execname] = count(); }: "/usr/lib/dtrace/io.d", line 43: operator -> cannot be applied to a forward declaration: no struct devstat definition is available
 
 Did you do a:
 kldload dtraceall
 
 on the system?  Just dtrace.ko isn't enough to have dtrace work..  What
 does dtrace -l say?  If you just have dtrace.ko loaded, you'll see
 something like:
 # dtrace -l
    ID   PROVIDER            MODULE                          FUNCTION NAME
     1     dtrace                                                     BEGIN
     2     dtrace                                                     END
     3     dtrace                                                     ERROR
 
 if you have dtraceall.ko loaded, you should see many pages of lines..
 
 -- 
   John-Mark Gurney				Voice: +1 415 225 5579
 
      "All that I will do, has been done, All that I have, has not."

From: Robert David <robert.david.public@gmail.com>
To: John-Mark Gurney <jmg@funkthat.com>
Cc: freebsd-gnats-submit@freebsd.org
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Mon, 30 Dec 2013 18:52:18 +0100

 Hi John-Mark,
 
 thanks for fast reply. Yes I got all modules loaded:
 
 root@notebook ~src (git)-[53624fa...] # dtrace -l | wc -l
    42596
 root@notebook ~src (git)-[53624fa...] # kldstat
 Id Refs Address            Size     Name
  1  109 0xffffffff80200000 b1b000   kernel
  2   15 0xffffffff80eda000 594d     opensolaris.ko
  3    1 0xffffffff80ee0000 223a51   zfs.ko
  4    1 0xffffffff81104000 4632     acpi_dock.ko
  5    1 0xffffffff81109000 6df3     acpi_ibm.ko
  6    1 0xffffffff81110000 149d5    ehci.ko
  7    1 0xffffffff81125000 fd38     uhci.ko
  8    1 0xffffffff81135000 efc3     ukbd.ko
  9    1 0xffffffff81212000 9bb4     linprocfs.ko
 10    2 0xffffffff8121c000 42d5a    linux.ko
 11    1 0xffffffff8125f000 5219     fdescfs.ko
 12    1 0xffffffff81265000 13050    ext2fs.ko
 13    1 0xffffffff81279000 a07d     tmpfs.ko
 14    1 0xffffffff81284000 d863     fuse.ko
 15    1 0xffffffff81292000 b41a     aio.ko
 16    3 0xffffffff8129e000 3f5a     libiconv.ko
 17    1 0xffffffff812a2000 153f     libmchain.ko
 18    1 0xffffffff812a4000 791      cd9660_iconv.ko
 19    1 0xffffffff812a5000 799      msdosfs_iconv.ko
 20    1 0xffffffff812a6000 2915     coretemp.ko
 21    1 0xffffffff812a9000 5556     if_tap.ko
 22    1 0xffffffff812af000 5499     linsysfs.ko
 23    1 0xffffffff812b5000 2ecf0    vboxdrv.ko
 24    1 0xffffffff812e4000 10769    ipfw.ko
 25    1 0xffffffff812f5000 aeb0     i915.ko
 26    1 0xffffffff81300000 1478c    drm.ko
 27    1 0xffffffff81315000 8bf      dtraceall.ko
 28    3 0xffffffff81316000 1da9     cyclic.ko
 29   11 0xffffffff81318000 35e47    dtrace.ko
 30    1 0xffffffff8134e000 edf      dtio.ko
 31    1 0xffffffff8134f000 44b3     dtmalloc.ko
 32    1 0xffffffff81354000 2203     dtnfscl.ko
 33    1 0xffffffff81357000 6254     fbt.ko
 34    1 0xffffffff8135e000 a5a3     fasttrap.ko
 35    1 0xffffffff81369000 482f     lockstat.ko
 36    1 0xffffffff8136e000 4af9     sdt.ko
 37    1 0xffffffff81373000 d4c4     systrace.ko
 38    1 0xffffffff81381000 d138     systrace_freebsd32.ko
 39    1 0xffffffff8138f000 4b49     profile.ko
 40    1 0xffffffff81394000 34a6     ums.ko
 41    1 0xffffffff81398000 2a08     uhid.ko
 
 
 I just tried to play with kernel and anything here is enabled (dtrace stuff). On plain GENERIC kernel it is the same.
 
 Robert.
 
From: Mark Johnston <markj@freebsd.org>
To: bug-followup@FreeBSD.org, robert.david.public@gmail.com
Cc:  
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Mon, 30 Dec 2013 14:17:47 -0500

 Hm, I've never seen this on CURRENT.
 
 Are you running in a jail by any chance? Do things work if you remove
 /usr/lib/dtrace/io.d?

From: Robert David <robert.david.public@gmail.com>
To: Mark Johnston <markj@freebsd.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Mon, 30 Dec 2013 21:43:06 +0100

 Hi,
 
 I got the system installed on my notebook and now does not have any
 jail set up. 
 
 Tried to move io.d out:
 
 root@notebook ~src (git)-[53624fa...] #
 mv /usr/lib/dtrace/io.d /usr/lib/dtrace/io.d.bu
 
 root@notebook ~src (git)-[53624fa...] # dtrace -n 'syscall:::entry
 { @num[execname] = count(); }' dtrace: invalid probe specifier
 syscall:::entry { @num[execname] = count(); }:
 "/usr/lib/dtrace/psinfo.d", line 90: failed to resolve type
 kernel`struct thread * for identifier curthread: Module is no longer
 loaded 1 
 root@notebook ~src (git)-[53624fa...] #
 
 Serching through internet I found this:
 http://lists.freebsd.org/pipermail/freebsd-dtrace/2013-October/000110.html
 
 Maybe something not backported from CURRENT.
 
 Robert.
 
 
 
 On Mon, 30 Dec 2013 14:17:47 -0500
 Mark Johnston <markj@freebsd.org> wrote:
 
 > Hm, I've never seen this on CURRENT.
 > 
 > Are you running in a jail by any chance? Do things work if you remove
 > /usr/lib/dtrace/io.d?
 

From: Mark Johnston <markj@freebsd.org>
To: Robert David <robert.david.public@gmail.com>, bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Sat, 4 Jan 2014 22:34:55 -0500

 >  Serching through internet I found this:
 >  http://lists.freebsd.org/pipermail/freebsd-dtrace/2013-October/000110.html
 >  
 >  Maybe something not backported from CURRENT.
 
 I'm testing 10.0-RC4 in a bhyve instance, and DTrace seems to be working
 properly. I will upgrade to stable/10 and test some more, but pretty much
 nothing has changed DTrace-wise between releng/10 and stable/10 so far.
 Moreover, the problems you're seeing tend to be the result of corrupted
 or absent CTF data, rather than bugs in the code.
 
 So here are a few more questions that'll help us pinpoint what's going
 on:
 - What happens when you run "ctfdump /boot/kernel/kernel"? Could you
   send the output? If it's correct, it should contain many thousands of
   lines.
 - Did you build the GENERIC kernel that you reported as having the
   issue? Do you have anything in make.conf or src.conf?
 - You mentioned that you have all of the probes (dtrace -l); do they
   have type info? For example, the command below gives the types for the
   arguments to read(2). Are they present when you run the same command?
 
 # dtrace -lv -n syscall:freebsd:read:entry
    ID   PROVIDER            MODULE                          FUNCTION NAME
 38182    syscall           freebsd                              read entry
 
 	Probe Description Attributes
 		Identifier Names: Private
 		Data Semantics:   Private
 		Dependency Class: Unknown
 
 	Argument Attributes
 		Identifier Names: Private
 		Data Semantics:   Private
 		Dependency Class: ISA
 
 	Argument Types
 		args[0]: int
 		args[1]: void *
 		args[2]: size_t
 
 Thanks!
 -Mark
Responsible-Changed-From-To: freebsd-amd64->markj 
Responsible-Changed-By: markj 
Responsible-Changed-When: Sun Jan 5 03:40:49 UTC 2014 
Responsible-Changed-Why:  
Track. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=185290 

From: Robert David <robert.david.public@gmail.com>
To: Mark Johnston <markj@freebsd.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Thu, 30 Jan 2014 15:05:18 +0100

 Hi Mark,
 
 I dont know if the previous mail was send ok, so I just resend it:
 
 for me it seems wired also. Now I only updated my src tree and rebuild
 everything. I installed another machine to test it up. On plain binary
 FreeBSD 10 it works ok. 
 
 Maybe something bad with kernel compilation and CTF. I will try
 download binary GENERIC kernel an test that.
 
 
 # dtrace -lv -n syscall:freebsd:read:entry
 dtrace: invalid probe specifier syscall:freebsd:read:entry:
 "/usr/lib/dtrace/io.d", line 43: operator -> cannot be applied to a
 forward declaration: no struct devstat definition is available
 
 produce the same errer as reported, so no luck to print it out
 
 I share ctfdump of kernel and make.conf. I think I dont
 have any special settings here. I do not have anything in src.conf. And
 yes I build all the stuff myself.
 
 make.conf:
 http://linsystem.net/make.conf
 ctfdump:
 http://linsystem.net/bug_ctfdump
 
 
 Robert.
 
From: Robert David <robert.david.public@gmail.com>
To: Mark Johnston <markj@freebsd.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Thu, 30 Jan 2014 16:24:04 +0100

 Hi Mark,
 
 I also tried to grab binary kernel form FBSD 10.0 which work on another
 machine. And have the same problem after booting it. So it seems to be
 something outside kernel. Maybe I will try to create new zfs root
 and place there binary release 10.0 to check that out.
 
 Robert.

From: Robert David <robert.david.public@gmail.com>
To: Mark Johnston <markj@freebsd.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: amd64/185290: Dtrace does not work on -stable/10
Date: Mon, 24 Feb 2014 22:11:44 +0100

 --MP_/4zjxZwLY=XAfIrDFK68zQk5
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline
 
 Hi Mark,
 
 I finaly solve that, but just only for me. I commented out some
 functions in .d scripts. Nothing usable for all but it works for me.
 
 Attached a patch for those who got the same problem.
 
 Robert.
 
 On Thu, 30 Jan 2014 16:24:04 +0100
 Robert David <robert.david.public@gmail.com> wrote:
 
 > Hi Mark,
 > 
 > I also tried to grab binary kernel form FBSD 10.0 which work on
 > another machine. And have the same problem after booting it. So it
 > seems to be something outside kernel. Maybe I will try to create new
 > zfs root and place there binary release 10.0 to check that out.
 > 
 > Robert.
 > 
 > 
 > On Sat, 4 Jan 2014 22:34:55 -0500
 > Mark Johnston <markj@freebsd.org> wrote:
 > 
 > > On Mon, Dec 30, 2013 at 08:50:01PM +0000, Robert David wrote:
 > > > The following reply was made to PR amd64/185290; it has been noted
 > > > by GNATS.
 > > > 
 > > > From: Robert David <robert.david.public@gmail.com>
 > > > To: Mark Johnston <markj@freebsd.org>
 > > > Cc: bug-followup@FreeBSD.org
 > > > Subject: Re: amd64/185290: Dtrace does not work on -stable/10
 > > > Date: Mon, 30 Dec 2013 21:43:06 +0100
 > > > 
 > > >  Hi,
 > > >  
 > > >  I got the system installed on my notebook and now does not have
 > > > any jail set up. 
 > > >  
 > > >  Tried to move io.d out:
 > > >  
 > > >  root@notebook ~src (git)-[53624fa...] #
 > > >  mv /usr/lib/dtrace/io.d /usr/lib/dtrace/io.d.bu
 > > >  
 > > >  root@notebook ~src (git)-[53624fa...] # dtrace -n
 > > > 'syscall:::entry { @num[execname] = count(); }' dtrace: invalid
 > > > probe specifier syscall:::entry { @num[execname] = count(); }:
 > > >  "/usr/lib/dtrace/psinfo.d", line 90: failed to resolve type
 > > >  kernel`struct thread * for identifier curthread: Module is no
 > > > longer loaded 1 
 > > >  root@notebook ~src (git)-[53624fa...] #
 > > >  
 > > >  Serching through internet I found this:
 > > >  http://lists.freebsd.org/pipermail/freebsd-dtrace/2013-October/000110.html
 > > >  
 > > >  Maybe something not backported from CURRENT.
 > > 
 > > I'm testing 10.0-RC4 in a bhyve instance, and DTrace seems to be
 > > working properly. I will upgrade to stable/10 and test some more,
 > > but pretty much nothing has changed DTrace-wise between releng/10
 > > and stable/10 so far. Moreover, the problems you're seeing tend to
 > > be the result of corrupted or absent CTF data, rather than bugs in
 > > the code.
 > > 
 > > So here are a few more questions that'll help us pinpoint what's
 > > going on:
 > > - What happens when you run "ctfdump /boot/kernel/kernel"? Could you
 > >   send the output? If it's correct, it should contain many thousands
 > > of lines.
 > > - Did you build the GENERIC kernel that you reported as having the
 > >   issue? Do you have anything in make.conf or src.conf?
 > > - You mentioned that you have all of the probes (dtrace -l); do they
 > >   have type info? For example, the command below gives the types for
 > > the arguments to read(2). Are they present when you run the same
 > > command?
 > > 
 > > # dtrace -lv -n syscall:freebsd:read:entry
 > >    ID   PROVIDER            MODULE                          FUNCTION
 > > NAME 38182    syscall           freebsd
 > > read entry
 > > 
 > > 	Probe Description Attributes
 > > 		Identifier Names: Private
 > > 		Data Semantics:   Private
 > > 		Dependency Class: Unknown
 > > 
 > > 	Argument Attributes
 > > 		Identifier Names: Private
 > > 		Data Semantics:   Private
 > > 		Dependency Class: ISA
 > > 
 > > 	Argument Types
 > > 		args[0]: int
 > > 		args[1]: void *
 > > 		args[2]: size_t
 > > 
 > > Thanks!
 > > -Mark
 > 
 
 
 --MP_/4zjxZwLY=XAfIrDFK68zQk5
 Content-Type: text/x-patch
 Content-Transfer-Encoding: 7bit
 Content-Disposition: attachment; filename=dtrace.patch
 
 diff --git a/io.d b/io.d
 index 18a54af..7494e82 100644
 --- a/io.d
 +++ b/io.d
 @@ -38,6 +38,7 @@ typedef struct devinfo {
          string dev_pathname;            /* pathname of device */
  } devinfo_t;
  
 +/*
  #pragma D binding "1.0" translator
  translator devinfo_t < struct devstat *D > {
             dev_major = D->device_number;
 @@ -47,6 +48,7 @@ translator devinfo_t < struct devstat *D > {
             dev_statname = stringof(D->device_name);
             dev_pathname = stringof(D->device_name);
  };
 +*/
  
  typedef struct bufinfo {
          int b_flags;                    /* flags */
 @@ -61,6 +63,7 @@ typedef struct bufinfo {
  /*        dev_t b_edev;                  extended device */
  } bufinfo_t;
  
 +/*
  #pragma D binding "1.0" translator
  translator bufinfo_t < struct bio *B > {
             b_flags = B->bio_flags;
 @@ -69,9 +72,10 @@ translator bufinfo_t < struct bio *B > {
             b_blkno = 0;
             b_lblkno = 0;
             b_resid = B->bio_resid;
 -           b_bufsize = 0; /* XXX gnn */
 +           b_bufsize = 0; 
             b_error = B->bio_error;
  };
 +*/
  
  /*
   * The following inline constants can be used to examine fi_oflags when using
 diff --git a/psinfo.d b/psinfo.d
 index c2219f7..dcb9be8 100644
 --- a/psinfo.d
 +++ b/psinfo.d
 @@ -88,6 +87,7 @@ translator lwpsinfo_t < struct thread *T > {
  	pr_wchan = (uintptr_t)T->td_wchan;
  };
  
 +/*
  inline psinfo_t *curpsinfo = xlate <psinfo_t *> (curthread->td_proc);
  #pragma D attributes Stable/Stable/Common curpsinfo
  #pragma D binding "1.0" curpsinfo
 @@ -95,4 +95,5 @@ inline psinfo_t *curpsinfo = xlate <psinfo_t *> (curthread->td_proc);
  inline lwpsinfo_t *curlwpsinfo = xlate <lwpsinfo_t *> (curthread);
  #pragma D attributes Stable/Stable/Common curlwpsinfo
  #pragma D binding "1.0" curlwpsinfo
 +*/
  
 diff --git a/tcp.d b/tcp.d
 index 4204507..c01e670 100644
 --- a/tcp.d
 +++ b/tcp.d
 @@ -160,6 +160,7 @@ typedef struct tcpinfoh {
  	struct tcphdr *tcp_hdr;		/* raw TCP header */
  } tcpinfoh_t;
  
 +/*
  #pragma D binding "1.0" translator
  translator csinfo_t < struct tcpcb *p > {
  	cs_addr =	NULL;
 @@ -167,12 +168,14 @@ translator csinfo_t < struct tcpcb *p > {
  	cs_pid =	0;
  	cs_zoneid =	0;
  };
 +*/
  
 +/*
  #pragma D binding "1.0" translator
  translator tcpsinfo_t < struct tcpcb *p > {
  	tcps_addr =		(uintptr_t)p;
 -	tcps_local =		-1; /* XXX */
 -	tcps_active =		-1; /* XXX */
 +	tcps_local =		-1;
 +	tcps_active =		-1; 
  	tcps_lport =		p == NULL ? 0 : ntohs(p->t_inpcb->inp_inc.inc_ie.ie_lport);
  	tcps_rport =		p == NULL ? 0 : ntohs(p->t_inpcb->inp_inc.inc_ie.ie_fport);
  	tcps_laddr =		p == NULL ? 0 :
 @@ -197,10 +200,11 @@ translator tcpsinfo_t < struct tcpcb *p > {
  	tcps_cwnd_ssthresh =	p == NULL ? -1  : p->snd_ssthresh;
  	tcps_sack_fack =	p == NULL ? 0  : p->snd_fack;
  	tcps_sack_snxt =	p == NULL ? 0  : p->sack_newdata;
 -	tcps_rto =		p == NULL ? -1  : p->t_rxtcur / 1000; /* XXX */
 +	tcps_rto =		p == NULL ? -1  : p->t_rxtcur / 1000;
  	tcps_mss =		p == NULL ? -1  : p->t_maxseg;
  	tcps_retransmit =	p == NULL ? -1 : p->t_rxtshift > 0 ? 1 : 0;
  };
 +*/
  
  #pragma D binding "1.0" translator
  translator tcpinfo_t < struct tcphdr *p > {
 
 --MP_/4zjxZwLY=XAfIrDFK68zQk5--

From: Mark Johnston <markj@freebsd.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/185290: [dtrace] Dtrace does not work on -stable/10
Date: Wed, 26 Mar 2014 21:18:37 -0400

 Just a follow-up to this PR before I close it.
 
 Based on some private emails, it looks like the problem had to do with
 booting the FreeBSD kernel from GRUB. In this case, the module and
 bootfile paths are not made available, and DTrace is unable to locate
 the kernel. Then it cannot process the scripts in /usr/lib/dtrace
 since kernel CTF data is unavailable, and it aborts.
 
 There's not much we can do in this case, and other tools besides
 DTrace may not work properly anyway. We agreed that the PR can be
 closed, and the submitter will see if the problem can be resolved from
 the GRUB side of things.
 
 -Mark
State-Changed-From-To: open->closed 
State-Changed-By: markj 
State-Changed-When: Thu Mar 27 02:17:46 UTC 2014 
State-Changed-Why:  
The problem was caused by an interaction with GRUB. See the final 
followup eamil. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=185290 
>Unformatted:
