From nobody@FreeBSD.org  Sun Jun 24 17:30:12 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 02CCC1065691
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 24 Jun 2012 17:30:12 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id E23C08FC0C
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 24 Jun 2012 17:30:11 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q5OHUBti048261
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 24 Jun 2012 17:30:11 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q5OHUBNm048260;
	Sun, 24 Jun 2012 17:30:11 GMT
	(envelope-from nobody)
Message-Id: <201206241730.q5OHUBNm048260@red.freebsd.org>
Date: Sun, 24 Jun 2012 17:30:11 GMT
From: Fabian Keil <fk@fabiankeil.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [PATCH] dtrace's timestamp variable is unreliable when the cpu goes into C3 state
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         169379
>Category:       kern
>Synopsis:       [dtrace] [patch] dtrace's timestamp variable is unreliable when the cpu goes into C3 state
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    markj
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Jun 24 17:40:06 UTC 2012
>Closed-Date:    
>Last-Modified:  Mon Apr 21 17:14:46 UTC 2014
>Originator:     Fabian Keil
>Release:        HEAD
>Organization:
>Environment:
FreeBSD r500.local 10.0-CURRENT FreeBSD 10.0-CURRENT #444 r+46680b7: Sat Jun 23 18:58:48 CEST 2012     fk@r500.local:/usr/obj/usr/src/sys/ZOEY  amd64

>Description:
On my amd64 system dtrace's timestamp variable is unreliable
if the cpu is allowed to go into C3 state and the best
kern.timecounter.hardware available (HPET) is used.

The timestamp counter seems to tick about ten times slower than
expected, causing things that are timed with timestamp deltas to
appear ten times faster than they are.

This does not happen if TSC is used as timecounter,
but it's rated as -1000 and using it is undesirable.

Looking at the code, I suspect the same issue exists on i386
but I didn't test this.
>How-To-Repeat:
Set hw.acpi.cpu.cx_lowest to C3 and execute the dtrace
script below while calling date in a loop with:

while date; do sleep 1; done

I get averages around 90 ms instead of 1000 ms + overhead as expected.

-------

#!/usr/sbin/dtrace -s

#pragma D option quiet

dtrace:::BEGIN
{
        cached_timestamp = (int64_t)0;
}

syscall::exec*:return
/(execname == "date") && cached_timestamp/
{
        self->elapsed = (timestamp - cached_timestamp) / 1000000;
        printf("ms since the last date execution: %d\n", self->elapsed);
        @elapsed_time["elapsed"] = lquantize(self->elapsed, 0, 1200, 10);
        @elapsed_avg["elapsed avg"] = avg(self->elapsed);
        self->elapsed = 0;
}

syscall::exec*:return
/execname == "date"/
{
        cached_timestamp = timestamp;
        printf("Date execution detected at %d\n", cached_timestamp);
}

tick-60sec
{
        exit(1);
}
>Fix:
The attached patch replaces the kernel timestamp code on
amd64 and i386 with the much simpler code already used on
mips.

This fixes the issue for me on amd64, I now reliably
get an average of 1007 ms which seems reasonable to me.

Patch attached with submission follows:

From 802f15e0c4a8ae946fc96370e484eabf2dfda091 Mon Sep 17 00:00:00 2001
From: Fabian Keil <fk@fabiankeil.de>
Date: Sun, 24 Jun 2012 15:50:28 +0200
Subject: [PATCH 2/2] dtrace: Fix timestamp unreliability on x86 when the cpu
 goes into C3 state

This replaces the code on amd64 and i386 with the much
simpler one used on mips.
---
 sys/cddl/dev/dtrace/amd64/dtrace_subr.c |   94 ++-----------------------------
 sys/cddl/dev/dtrace/i386/dtrace_subr.c  |   94 ++-----------------------------
 2 files changed, 8 insertions(+), 180 deletions(-)

diff --git a/sys/cddl/dev/dtrace/amd64/dtrace_subr.c b/sys/cddl/dev/dtrace/amd64/dtrace_subr.c
index 2d42ce2..0557f1e 100644
--- a/sys/cddl/dev/dtrace/amd64/dtrace_subr.c
+++ b/sys/cddl/dev/dtrace/amd64/dtrace_subr.c
@@ -355,82 +355,6 @@ dtrace_safe_defer_signal(void)
 }
 #endif
 
-static int64_t	tgt_cpu_tsc;
-static int64_t	hst_cpu_tsc;
-static int64_t	tsc_skew[MAXCPU];
-static uint64_t	nsec_scale;
-
-/* See below for the explanation of this macro. */
-#define SCALE_SHIFT	28
-
-static void
-dtrace_gethrtime_init_cpu(void *arg)
-{
-	uintptr_t cpu = (uintptr_t) arg;
-
-	if (cpu == curcpu)
-		tgt_cpu_tsc = rdtsc();
-	else
-		hst_cpu_tsc = rdtsc();
-}
-
-static void
-dtrace_gethrtime_init(void *arg)
-{
-	struct pcpu *pc;
-	uint64_t tsc_f;
-	cpuset_t map;
-	int i;
-
-	/*
-	 * Get TSC frequency known at this moment.
-	 * This should be constant if TSC is invariant.
-	 * Otherwise tick->time conversion will be inaccurate, but
-	 * will preserve monotonic property of TSC.
-	 */
-	tsc_f = atomic_load_acq_64(&tsc_freq);
-
-	/*
-	 * The following line checks that nsec_scale calculated below
-	 * doesn't overflow 32-bit unsigned integer, so that it can multiply
-	 * another 32-bit integer without overflowing 64-bit.
-	 * Thus minimum supported TSC frequency is 62.5MHz.
-	 */
-	KASSERT(tsc_f > (NANOSEC >> (32 - SCALE_SHIFT)), ("TSC frequency is too low"));
-
-	/*
-	 * We scale up NANOSEC/tsc_f ratio to preserve as much precision
-	 * as possible.
-	 * 2^28 factor was chosen quite arbitrarily from practical
-	 * considerations:
-	 * - it supports TSC frequencies as low as 62.5MHz (see above);
-	 * - it provides quite good precision (e < 0.01%) up to THz
-	 *   (terahertz) values;
-	 */
-	nsec_scale = ((uint64_t)NANOSEC << SCALE_SHIFT) / tsc_f;
-
-	/* The current CPU is the reference one. */
-	sched_pin();
-	tsc_skew[curcpu] = 0;
-	CPU_FOREACH(i) {
-		if (i == curcpu)
-			continue;
-
-		pc = pcpu_find(i);
-		CPU_SETOF(PCPU_GET(cpuid), &map);
-		CPU_SET(pc->pc_cpuid, &map);
-
-		smp_rendezvous_cpus(map, NULL,
-		    dtrace_gethrtime_init_cpu,
-		    smp_no_rendevous_barrier, (void *)(uintptr_t) i);
-
-		tsc_skew[i] = tgt_cpu_tsc - hst_cpu_tsc;
-	}
-	sched_unpin();
-}
-
-SYSINIT(dtrace_gethrtime_init, SI_SUB_SMP, SI_ORDER_ANY, dtrace_gethrtime_init, NULL);
-
 /*
  * DTrace needs a high resolution time function which can
  * be called from a probe context and guaranteed not to have
@@ -441,21 +365,11 @@ SYSINIT(dtrace_gethrtime_init, SI_SUB_SMP, SI_ORDER_ANY, dtrace_gethrtime_init,
 uint64_t
 dtrace_gethrtime()
 {
-	uint64_t tsc;
-	uint32_t lo;
-	uint32_t hi;
+	struct      timespec curtime;
 
-	/*
-	 * We split TSC value into lower and higher 32-bit halves and separately
-	 * scale them with nsec_scale, then we scale them down by 2^28
-	 * (see nsec_scale calculations) taking into account 32-bit shift of
-	 * the higher half and finally add.
-	 */
-	tsc = rdtsc() - tsc_skew[curcpu];
-	lo = tsc;
-	hi = tsc >> 32;
-	return (((lo * nsec_scale) >> SCALE_SHIFT) +
-	    ((hi * nsec_scale) << (32 - SCALE_SHIFT)));
+	nanouptime(&curtime);
+
+	return (curtime.tv_sec * 1000000000UL + curtime.tv_nsec);
 }
 
 uint64_t
diff --git a/sys/cddl/dev/dtrace/i386/dtrace_subr.c b/sys/cddl/dev/dtrace/i386/dtrace_subr.c
index 3a1677a..4b6ae24 100644
--- a/sys/cddl/dev/dtrace/i386/dtrace_subr.c
+++ b/sys/cddl/dev/dtrace/i386/dtrace_subr.c
@@ -356,82 +356,6 @@ dtrace_safe_defer_signal(void)
 }
 #endif
 
-static int64_t	tgt_cpu_tsc;
-static int64_t	hst_cpu_tsc;
-static int64_t	tsc_skew[MAXCPU];
-static uint64_t	nsec_scale;
-
-/* See below for the explanation of this macro. */
-#define SCALE_SHIFT	28
-
-static void
-dtrace_gethrtime_init_cpu(void *arg)
-{
-	uintptr_t cpu = (uintptr_t) arg;
-
-	if (cpu == curcpu)
-		tgt_cpu_tsc = rdtsc();
-	else
-		hst_cpu_tsc = rdtsc();
-}
-
-static void
-dtrace_gethrtime_init(void *arg)
-{
-	cpuset_t map;
-	struct pcpu *pc;
-	uint64_t tsc_f;
-	int i;
-
-	/*
-	 * Get TSC frequency known at this moment.
-	 * This should be constant if TSC is invariant.
-	 * Otherwise tick->time conversion will be inaccurate, but
-	 * will preserve monotonic property of TSC.
-	 */
-	tsc_f = atomic_load_acq_64(&tsc_freq);
-
-	/*
-	 * The following line checks that nsec_scale calculated below
-	 * doesn't overflow 32-bit unsigned integer, so that it can multiply
-	 * another 32-bit integer without overflowing 64-bit.
-	 * Thus minimum supported TSC frequency is 62.5MHz.
-	 */
-	KASSERT(tsc_f > (NANOSEC >> (32 - SCALE_SHIFT)), ("TSC frequency is too low"));
-
-	/*
-	 * We scale up NANOSEC/tsc_f ratio to preserve as much precision
-	 * as possible.
-	 * 2^28 factor was chosen quite arbitrarily from practical
-	 * considerations:
-	 * - it supports TSC frequencies as low as 62.5MHz (see above);
-	 * - it provides quite good precision (e < 0.01%) up to THz
-	 *   (terahertz) values;
-	 */
-	nsec_scale = ((uint64_t)NANOSEC << SCALE_SHIFT) / tsc_f;
-
-	/* The current CPU is the reference one. */
-	sched_pin();
-	tsc_skew[curcpu] = 0;
-	CPU_FOREACH(i) {
-		if (i == curcpu)
-			continue;
-
-		pc = pcpu_find(i);
-		CPU_SETOF(PCPU_GET(cpuid), &map);
-		CPU_SET(pc->pc_cpuid, &map);
-
-		smp_rendezvous_cpus(map, NULL,
-		    dtrace_gethrtime_init_cpu,
-		    smp_no_rendevous_barrier, (void *)(uintptr_t) i);
-
-		tsc_skew[i] = tgt_cpu_tsc - hst_cpu_tsc;
-	}
-	sched_unpin();
-}
-
-SYSINIT(dtrace_gethrtime_init, SI_SUB_SMP, SI_ORDER_ANY, dtrace_gethrtime_init, NULL);
-
 /*
  * DTrace needs a high resolution time function which can
  * be called from a probe context and guaranteed not to have
@@ -442,21 +366,11 @@ SYSINIT(dtrace_gethrtime_init, SI_SUB_SMP, SI_ORDER_ANY, dtrace_gethrtime_init,
 uint64_t
 dtrace_gethrtime()
 {
-	uint64_t tsc;
-	uint32_t lo;
-	uint32_t hi;
+	struct      timespec curtime;
 
-	/*
-	 * We split TSC value into lower and higher 32-bit halves and separately
-	 * scale them with nsec_scale, then we scale them down by 2^28
-	 * (see nsec_scale calculations) taking into account 32-bit shift of
-	 * the higher half and finally add.
-	 */
-	tsc = rdtsc() - tsc_skew[curcpu];
-	lo = tsc;
-	hi = tsc >> 32;
-	return (((lo * nsec_scale) >> SCALE_SHIFT) +
-	    ((hi * nsec_scale) << (32 - SCALE_SHIFT)));
+	nanouptime(&curtime);
+
+	return (curtime.tv_sec * 1000000000UL + curtime.tv_nsec);
 }
 
 uint64_t
-- 
1.7.10.3



>Release-Note:
>Audit-Trail:

From: Fabian Keil <fk@fabiankeil.de>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: amd64/169379: [PATCH] dtrace's timestamp variable is unreliable
 when the cpu goes into C3 state
Date: Sun, 24 Jun 2012 23:04:25 +0200

 --Sig_/6qgpQuCO_xqGoQhyYL1wP_k
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 
 Unfortunately the patch has the same issue as the one I proposed in:
 http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/159612
 
 In this case it causes the system to panic when using timestamp
 inside probes for "fbt::nanouptime:".
 
 Fabian
 
 --Sig_/6qgpQuCO_xqGoQhyYL1wP_k
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iEYEARECAAYFAk/ngN8ACgkQSMVSH78upWPUPgCfcHz9ZYyLUVxonGrtQTPXQgrd
 HGYAniuhJ9rdSE9lE7PQls3Ar1WZubfD
 =gSXG
 -----END PGP SIGNATURE-----
 
 --Sig_/6qgpQuCO_xqGoQhyYL1wP_k--
Responsible-Changed-From-To: freebsd-amd64->gnn 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Sun Jun 24 23:51:09 UTC 2012 
Responsible-Changed-Why:  
possibly another one for gnn to mull over. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=169379 
State-Changed-From-To: open->patched 
State-Changed-By: gnn 
State-Changed-When: Thu Jul 26 16:19:37 UTC 2012 
State-Changed-Why:  
This should be fixed by: 

http://svn.freebsd.org/changeset/base/238537 


http://www.freebsd.org/cgi/query-pr.cgi?pr=169379 

From: Fabian Keil <fk@fabiankeil.de>
To: gnn@FreeBSD.org
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/169379: [dtrace] [patch] dtrace's timestamp variable is
 unreliable when the cpu goes into C3 state
Date: Thu, 26 Jul 2012 19:09:45 +0200

 --Sig_/oWYLLjjbOefr3gKJPSQtzlI
 Content-Type: multipart/mixed; boundary="MP_/CjEege9cEUS=5pWokTqRx0S"
 
 --MP_/CjEege9cEUS=5pWokTqRx0S
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: inline
 
 gnn@FreeBSD.org wrote:
 
 > State-Changed-Why:=20
 > This should be fixed by:
 >=20
 > http://svn.freebsd.org/changeset/base/238537
 
 I assume this wasn't meant for this PR, but for:
 http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dkern/159612
 
 Anyway, I realised that using a shell loop to reproduce the
 problem is unnecessary. The attached script tries to measure
 the time between tick-1sec probes instead, which makes it
 self-contained.
 
 fk@r500 ~ $sysctl hw.acpi.cpu.cx_lowest
 hw.acpi.cpu.cx_lowest: C3
 fk@r500 ~ $sudo ~/scripts/timestamp-test.d
 Probe fired at 2012 Jul 26 19:06:36,  44 ms after the previous one.
 Probe fired at 2012 Jul 26 19:06:37,  63 ms after the previous one.
 [...]
 Probe fired at 2012 Jul 26 19:07:33,  55 ms after the previous one.
 Probe fired at 2012 Jul 26 19:07:34,  52 ms after the previous one.
 
   elapsed
            value  ------------- Distribution ------------- count
               30 |                                         0
               40 |@@@@@@@@@@@@@@@@@@@@@@@@@                37
               50 |@@@@@@@@@                                14
               60 |@                                        2
               70 |@                                        1
               80 |@                                        1
               90 |                                         0
              100 |                                         0
              110 |@                                        1
              120 |                                         0
              130 |@                                        2
              140 |                                         0
              150 |                                         0
              160 |                                         0
              170 |                                         0
              180 |                                         0
              190 |                                         0
              200 |                                         0
              210 |@                                        1
              220 |                                         0
 
   elapsed avg                                                      55
 
 
 The results depend on the load of course. The above "measurement"
 was made while the system was mostly idle.
 
 Fabian
 
 --MP_/CjEege9cEUS=5pWokTqRx0S
 Content-Type: text/x-dsrc
 Content-Transfer-Encoding: quoted-printable
 Content-Disposition: attachment; filename=timestamp-test.d
 
 #!/usr/sbin/dtrace -s
 
 #pragma D option quiet
 
 dtrace:::BEGIN
 {
         cached_timestamp =3D (int64_t)0;
 }
 
 tick-1sec
 /cached_timestamp/
 {
         self->elapsed =3D (timestamp - cached_timestamp) / 1000000;
         printf("Probe fired at %Y, %3d ms after the previous one.\n",
                walltimestamp, self->elapsed);
         @elapsed_time["elapsed"] =3D lquantize(self->elapsed, 0, 1200, 10);
         @elapsed_avg["elapsed avg"] =3D avg(self->elapsed);
         self->elapsed =3D 0;
 }
 
 tick-1sec
 {
         cached_timestamp =3D timestamp;
 }
 
 tick-60sec
 {
         exit(1);
 }
 
 --MP_/CjEege9cEUS=5pWokTqRx0S--
 
 --Sig_/oWYLLjjbOefr3gKJPSQtzlI
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.19 (FreeBSD)
 
 iEYEARECAAYFAlARedwACgkQSMVSH78upWNGPwCgg+XaNwE5j6xARAkBmg5b2GWJ
 hgwAn2fs1BS8YYUKbIRHFijPBWkkuGtk
 =glvL
 -----END PGP SIGNATURE-----
 
 --Sig_/oWYLLjjbOefr3gKJPSQtzlI--
State-Changed-From-To: patched->open 
State-Changed-By: gnn 
State-Changed-When: Thu Jul 26 18:15:01 UTC 2012 
State-Changed-Why:  
Updated the wrong PR, this is still an issue. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=169379 

From: Davide Italiano <davide@freebsd.org>
To: bug-followup@FreeBSD.org, fk@fabiankeil.de, 
	George Neville-Neil <gnn@freebsd.org>
Cc:  
Subject: Re: kern/169379: [dtrace] [patch] dtrace&#39;s timestamp variable is
 unreliable when the cpu goes into C3 state
Date: Wed, 29 May 2013 13:11:48 +0200

 Hi you both.
 I'm not able to reproduce this as per r251026.
 root@dirichlet:/home/davide # sysctl hw.acpi.cpu.cx_lowest
 hw.acpi.cpu.cx_lowest: C3
 
 root@dirichlet:/home/davide/debug_dtrace # ./timestamp.d
 Probe fired at 2013 May 28 17:19:01, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:02, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:03, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:04, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:05, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:06, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:07, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:08, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:09, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:10, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:11, 999 ms after the previous one.
 Probe fired at 2013 May 28 17:19:12, 999 ms after the previous one.
 ^C
 Probe fired at 2013 May 28 17:19:13, 999 ms after the previous one.
 
   elapsed
            value  ------------- Distribution ------------- count
              980 |                                         0
              990 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 13
             1000 |                                         0
 
   elapsed avg                                                     999
 
 root@dirichlet:/home/davide/debug_dtrace # ./timestamp2.d
 Date execution detected at 582088745137
 ms since the last date execution: 1012
 Date execution detected at 583101069460
 ms since the last date execution: 1065
 Date execution detected at 584166933763
 ms since the last date execution: 1034
 Date execution detected at 585201455739
 ms since the last date execution: 1010
 Date execution detected at 586212097290
 ms since the last date execution: 1020
 Date execution detected at 587233056523
 ms since the last date execution: 1003
 Date execution detected at 588237041013
 ms since the last date execution: 1043
 Date execution detected at 589280876859
 ms since the last date execution: 1008
 Date execution detected at 590289153541
 ms since the last date execution: 1065
 Date execution detected at 591355033619
 ms since the last date execution: 1003
 Date execution detected at 592359053134
 
 Can you please see if you can reproduce it on a recent HEAD?

From: Fabian Keil <fk@fabiankeil.de>
To: Davide Italiano <davide@freebsd.org>
Cc: bug-followup@FreeBSD.org, George Neville-Neil <gnn@freebsd.org>
Subject: Re: kern/169379: [dtrace] [patch] dtrace's timestamp variable is
 unreliable when the cpu goes into C3 state
Date: Wed, 29 May 2013 19:27:05 +0200

 --Sig_/5ewo3FF3gEy=h=p0jI5QE3z
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 
 Davide Italiano <davide@freebsd.org> wrote:
 
 > I'm not able to reproduce this as per r251026.
 
 Were you ever able to reproduce it with your current hardware?
 
 My understanding is that whether or not the various sleep states
 affect the TSC depends on the hardware.
 
 This is already reflected in the TSC timecounter quality which I
 assume isn't -1000 on your system, but DTrace currently doesn't
 leverage this information.
 
 > Can you please see if you can reproduce it on a recent HEAD?
 
 I can still reproduce it with a kernel based on r251099:
 
 fk@r500 ~ $sudo scripts/timestamp-test.d=20
 Probe fired at 2013 May 29 19:09:42,  28 ms after the previous one.
 Probe fired at 2013 May 29 19:09:43,  33 ms after the previous one.
 Probe fired at 2013 May 29 19:09:44, 320 ms after the previous one.
 [...]
 Probe fired at 2013 May 29 19:10:37, 126 ms after the previous one.
 Probe fired at 2013 May 29 19:10:38, 149 ms after the previous one.
 Probe fired at 2013 May 29 19:10:39, 204 ms after the previous one.
 Probe fired at 2013 May 29 19:10:40, 136 ms after the previous one.
 
   elapsed                                          =20
            value  ------------- Distribution ------------- count   =20
               10 |                                         0       =20
               20 |@                                        2       =20
               30 |@@@@@                                    7       =20
               40 |@                                        2       =20
               50 |                                         0       =20
               60 |                                         0       =20
               70 |@@@                                      4       =20
               80 |@@                                       3       =20
               90 |@                                        2       =20
 [...]
              680 |                                         0       =20
              690 |@                                        1       =20
              700 |                                         0       =20
 
   elapsed avg                                                     177
 
 Fabian
 
 --Sig_/5ewo3FF3gEy=h=p0jI5QE3z
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.20 (FreeBSD)
 
 iQIcBAEBCAAGBQJRpjppAAoJEGkYIpGLojcccK0P/3CluVbxDe4AbiciAZeAxE1x
 cexkOqyrClikiHFfIFnR64W00A4K88D8AWC2tsDb3vcZsT+Ls5CmKKT1ln/+JD4I
 VqZ5L3Qll/GQrUaXnM5IWOj7+qWF6iWOasj/jHnxtvEzjOVhJ0xgvev+4vfeHoBC
 GiflJYLc3XRqe7IbeR71gSbjY7g0SWh0fN8TSUhEoRL1FHkegy4N7QRFCTsAbcOc
 CMkxvDtKYPhroCBg8UXAiihvKmcA1j5dQnN3YuJvU3eMDsHJHxVWvoYcQtJXplHN
 Uy2yr5K5jnyiqyachpBECjaing7xRgGEcB6B5fQ5RxZdBF9XCd1b7EVybQ7ooumZ
 2/FtPofeteT9KcRgN07dhZ5y0SG/7IY7jPNt2z1F/Ledqx1g/VQ9qquCvo3y47aO
 P7nQlW9aR04yShKzTQrJ6Ozdj7BturclaQ86X3IxGbeEB3krh39BFmqzSXodVF0R
 osJkrKgrM5jpUfx9UwHph22Og5r2YtCOu8ANInDi1vryo9qa4LUKTai1XWhKB7cy
 wh4mXnoPWUaIS/SIAgsbI0n3QRWlehCB39yN8+ZJw3stAdu08aIZ0hO+n0zMlwxE
 m/cJCpgM0zws80sIU5nRhL0SwRjRJHRpund9sUl0L3Tvu78qgIw3GESQqwiSgvt0
 sj/18AN9L9SfVU/5X7wk
 =Vp0u
 -----END PGP SIGNATURE-----
 
 --Sig_/5ewo3FF3gEy=h=p0jI5QE3z--

From: Davide Italiano <davide@freebsd.org>
To: Fabian Keil <fk@fabiankeil.de>
Cc: bug-followup@freebsd.org, George Neville-Neil <gnn@freebsd.org>, 
	Alfred Perlstein <alfred@ixsystems.com>
Subject: Re: kern/169379: [dtrace] [patch] dtrace's timestamp variable is
 unreliable when the cpu goes into C3 state
Date: Wed, 29 May 2013 19:36:25 +0200

 On Wed, May 29, 2013 at 7:27 PM, Fabian Keil <fk@fabiankeil.de> wrote:
 > Davide Italiano <davide@freebsd.org> wrote:
 >
 >> I'm not able to reproduce this as per r251026.
 >
 > Were you ever able to reproduce it with your current hardware?
 >
 > My understanding is that whether or not the various sleep states
 > affect the TSC depends on the hardware.
 >
 > This is already reflected in the TSC timecounter quality which I
 > assume isn't -1000 on your system, but DTrace currently doesn't
 > leverage this information.
 >
 >> Can you please see if you can reproduce it on a recent HEAD?
 >
 > I can still reproduce it with a kernel based on r251099:
 >
 > fk@r500 ~ $sudo scripts/timestamp-test.d
 > Probe fired at 2013 May 29 19:09:42,  28 ms after the previous one.
 > Probe fired at 2013 May 29 19:09:43,  33 ms after the previous one.
 > Probe fired at 2013 May 29 19:09:44, 320 ms after the previous one.
 > [...]
 > Probe fired at 2013 May 29 19:10:37, 126 ms after the previous one.
 > Probe fired at 2013 May 29 19:10:38, 149 ms after the previous one.
 > Probe fired at 2013 May 29 19:10:39, 204 ms after the previous one.
 > Probe fired at 2013 May 29 19:10:40, 136 ms after the previous one.
 >
 >   elapsed
 >            value  ------------- Distribution ------------- count
 >               10 |                                         0
 >               20 |@                                        2
 >               30 |@@@@@                                    7
 >               40 |@                                        2
 >               50 |                                         0
 >               60 |                                         0
 >               70 |@@@                                      4
 >               80 |@@                                       3
 >               90 |@                                        2
 > [...]
 >              680 |                                         0
 >              690 |@                                        1
 >              700 |                                         0
 >
 >   elapsed avg                                                     177
 >
 > Fabian
 
 hmm, OK, I trust you.
 Do you mind to give me the specific hardware configuration you're
 running on (e.g. CPU, the values of sysctl related to timecounter)? In
 particular, this problem is visible only if you rely on TSC or also
 with other hardware counters (e.g. HPET)?
 
 -- 
 Davide
 
 "There are no solved problems; there are only problems that are more
 or less solved" -- Henri Poincare

From: Fabian Keil <fk@fabiankeil.de>
To: Davide Italiano <davide@freebsd.org>
Cc: bug-followup@freebsd.org, George Neville-Neil <gnn@freebsd.org>, Alfred
 Perlstein <alfred@ixsystems.com>
Subject: Re: kern/169379: [dtrace] [patch] dtrace's timestamp variable is
 unreliable when the cpu goes into C3 state
Date: Wed, 29 May 2013 21:05:53 +0200

 --Sig_/SQNM0YZ3wbiJgh5p9b/1FZo
 Content-Type: text/plain; charset=US-ASCII
 Content-Transfer-Encoding: quoted-printable
 
 Davide Italiano <davide@freebsd.org> wrote:
 
 > On Wed, May 29, 2013 at 7:27 PM, Fabian Keil <fk@fabiankeil.de> wrote:
 > > Davide Italiano <davide@freebsd.org> wrote:
 
 > >> Can you please see if you can reproduce it on a recent HEAD?
 > >
 > > I can still reproduce it with a kernel based on r251099:
 > >
 > > fk@r500 ~ $sudo scripts/timestamp-test.d
 [...]
 
 > hmm, OK, I trust you.
 
 While you can trust me that I can still reproduce the problem with a kernel
 based on r251099, what I wrote about timecounters should probably be run
 by jkim@, mav@ or someone else with actual experience in this area.
 
 I was mainly parroting what I remembered from threads like
 "Time keeping Issues with the low-resolution TSC timecounter":
 http://lists.freebsd.org/pipermail/freebsd-current/2011-June/thread.html#25=
 175
 
 > Do you mind to give me the specific hardware configuration you're
 > running on (e.g. CPU, the values of sysctl related to timecounter)? In
 > particular, this problem is visible only if you rely on TSC or also
 > with other hardware counters (e.g. HPET)?
 
 I'm using a Lenove ThinkPad R500 2714-9VG:
 http://www.nycbug.org/?action=3Ddmesgd&dmesgid=3D2449
 
 fk@r500 ~ $sysctl -a | grep counter
 kern.timecounter.tc.i8254.mask: 65535
 kern.timecounter.tc.i8254.counter: 28801
 kern.timecounter.tc.i8254.frequency: 1193182
 kern.timecounter.tc.i8254.quality: 0
 kern.timecounter.tc.HPET.mask: 4294967295
 kern.timecounter.tc.HPET.counter: 1207831391
 kern.timecounter.tc.HPET.frequency: 14318180
 kern.timecounter.tc.HPET.quality: 950
 kern.timecounter.tc.ACPI-fast.mask: 16777215
 kern.timecounter.tc.ACPI-fast.counter: 4714057
 kern.timecounter.tc.ACPI-fast.frequency: 3579545
 kern.timecounter.tc.ACPI-fast.quality: 900
 kern.timecounter.tc.TSC.mask: 4294967295
 kern.timecounter.tc.TSC.counter: 106194638
 kern.timecounter.tc.TSC.frequency: 1995043070
 kern.timecounter.tc.TSC.quality: -1000
 kern.timecounter.stepwarnings: 0
 kern.timecounter.alloweddeviation: 5
 kern.timecounter.hardware: HPET
 kern.timecounter.choice: TSC(-1000) ACPI-fast(900) HPET(950) i8254(0) dummy=
 (-1000000)
 kern.timecounter.tick: 1
 kern.timecounter.fast_gettime: 1
 kern.timecounter.invariant_tsc: 1
 kern.timecounter.smp_tsc: 0
 kern.timecounter.smp_tsc_adjust: 0
 kern.timecounter.tsc_shift: 1
 
 With hw.acpi.cpu.cx_lowest set to C3, the DTrace timestamps are unreliable
 if kern.timecounter.hardware is set to ACPI-fast, HPET or i8254.
 
 With kern.timecounter.hardware=3DTSC the DTrace timestamp values are
 reliable but due to the poor rating using TSC seems undesirable to me:
 
 fk@r500 ~ $sudo scripts/timestamp-test.d=20
 Probe fired at 2013 May 29 20:20:13, 999 ms after the previous one.
 [...]
 Probe fired at 2013 May 29 20:21:11, 999 ms after the previous one.
 
   elapsed                                          =20
            value  ------------- Distribution ------------- count   =20
              980 |                                         0       =20
              990 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 59      =20
             1000 |                                         0       =20
 
   elapsed avg                                                     999
 
 Fabian
 
 --Sig_/SQNM0YZ3wbiJgh5p9b/1FZo
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Disposition: attachment; filename=signature.asc
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v2.0.20 (FreeBSD)
 
 iQIcBAEBCAAGBQJRplGQAAoJEGkYIpGLojccw9cP/0MTHiah1ZoTJY1FGCN1d575
 P5nDBIeNhTmWnfC7VKL1eHfFEx/5dhfAqgwSBW2gnh2e0w+gf5pD42mw2+aCY3GY
 fSQobr1GsnjUIIFdY76jKn8Lcc3lzqfU9Klj6VpGI5DONm9L6nWgqsiybfHoDEUt
 z0B9UFla5K+c4tyIp+qNE+vHEYME3MrdkH7mjunZYxE+gysZwO8LPCtl8/e1W+eA
 eQopCCyuRNNnYiTjtSisrpA6fysuEoTpM3LbX86FgeE/Fg8vo+OE8ImxLY7FB4Kd
 Djg+L7yLC7ENaiOK/2vcO+ehGL1+HOOxkNN+QAcwl6kSaAMW+eQ0XEcgmIuTT4UA
 xnfb5IrW+mENCTucbJhPpVj3NK9yh8qD04zJ2JhjfqeEfjVpMkfmUS9uNCxx6bTk
 NPnIgwaha6P7FZVAZu0TY8/95OHSnMCjKexshiwShxfEdUUnVxWCb1MFW/oT4mCI
 /KKKSUsc/zJWlPkO5lvvi47HWXzKAoa8UlnMyQgTSAuRDd+6Xuwkem9fCLG8s0V5
 GCafGRt9J4Tlv24bB6q+aLeIA7fUJWXlBsc853bXYljOY3uvRaIfnttcPzTClkK5
 vTC/qGhgQeNI1dTrOP4cpda4qmkVIO7vXo85jgIVqsSRH5OvB7Nq/JBn/R6dxT1W
 5fIR4NXhtVOOuc/olCty
 =oTgb
 -----END PGP SIGNATURE-----
 
 --Sig_/SQNM0YZ3wbiJgh5p9b/1FZo--
Responsible-Changed-From-To: gnn->markj 
Responsible-Changed-By: gnn 
Responsible-Changed-When: Mon Apr 21 17:14:39 UTC 2014 
Responsible-Changed-Why:  
Hand over to the person doing most of the DTrace work. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=169379 
>Unformatted:
