From nobody@FreeBSD.org  Wed May 27 20:26:55 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id DE45510658DF
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 27 May 2009 20:26:55 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id CC97B8FC14
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 27 May 2009 20:26:55 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n4RKQtlA014825
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 27 May 2009 20:26:55 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n4RKQt03014824;
	Wed, 27 May 2009 20:26:55 GMT
	(envelope-from nobody)
Message-Id: <200905272026.n4RKQt03014824@www.freebsd.org>
Date: Wed, 27 May 2009 20:26:55 GMT
From: Adrian Chadd <adrian@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: FreeBSD-current/Xen timecounter jumps
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         135008
>Category:       kern
>Synopsis:       [xen] FreeBSD-current/Xen timecounter jumps
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    brucec
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed May 27 20:30:02 UTC 2009
>Closed-Date:    Sat Mar 06 00:12:17 UTC 2010
>Last-Modified:  Sat Mar 06 00:12:17 UTC 2010
>Originator:     Adrian Chadd
>Release:        
>Organization:
>Environment:
>Description:
The Xen DomU clock seems "off". Its not sync'ed to the value of the Dom0 clock (at least by "date" in both VMs.) "xen.independent_wallclock" is 0 in the DomU.

I triggered a weird clock drift by running ntpdate in the dom0 by running ntpdate.

>How-To-Repeat:
In the Dom0:

[root@hosting-2 adrian]# ntpdate 203.56.14.1
28 May 04:03:03 ntpdate[30489]: step time server 203.56.14.1 offset 50.619242 sec
[root@hosting-2 adrian]# date
Thu May 28 04:03:07 WST 2009

In the DomU:

freebsd1# date
Mon May 25 04:04:39 UTC 2009
freebsd1# date
Sun May 24 07:03:31 UTC 2009


>Fix:


>Release-Note:
>Audit-Trail:

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135008: commit references a PR
Date: Thu, 28 May 2009 04:17:16 +0000 (UTC)

 Author: adrian
 Date: Thu May 28 04:17:05 2009
 New Revision: 192953
 URL: http://svn.freebsd.org/changeset/base/192953
 
 Log:
   Say hello to a very basic, read-only, Xen Hypervisor RTC.
   
   The hypervisor doesn't provide a single "TOD" - it instead provides a
   "start time" and a "running time". These are added together to form
   the current TOD. The TOD is in UTC.
   
   This RTC is only (initially) designed to be read at startup. There's
   some further poking that needs to happen to pick up hypervisor time
   changes (ie, by the Dom0 time being adjusted by something). This
   time adjustment currently can cause "weird stuff" in the DomU clock;
   I'll begin investigating and repairing that in subsequent commits.
   
   PR:		135008
 
 Added:
   head/sys/i386/xen/xen_rtc.c   (contents, props changed)
 Modified:
   head/sys/conf/files.i386
 
 Modified: head/sys/conf/files.i386
 ==============================================================================
 --- head/sys/conf/files.i386	Thu May 28 04:08:07 2009	(r192952)
 +++ head/sys/conf/files.i386	Thu May 28 04:17:05 2009	(r192953)
 @@ -327,6 +327,7 @@ i386/isa/atpic.c		optional atpic	
  #i386/isa/atpic_vector.s		standard
  i386/isa/clock.c		optional native
  i386/xen/clock.c		optional xen
 +i386/xen/xen_rtc.c		optional xen
  i386/isa/dpms.c			optional dpms
  i386/isa/elcr.c			standard
  i386/isa/elink.c		optional ep | ie
 
 Added: head/sys/i386/xen/xen_rtc.c
 ==============================================================================
 --- /dev/null	00:00:00 1970	(empty, because file is newly added)
 +++ head/sys/i386/xen/xen_rtc.c	Thu May 28 04:17:05 2009	(r192953)
 @@ -0,0 +1,191 @@
 +/*-
 + * Copyright (c) 2009 Adrian Chadd
 + * All rights reserved.
 + *
 + * Redistribution and use in source and binary forms, with or without
 + * modification, are permitted provided that the following conditions
 + * are met:
 + * 1. Redistributions of source code must retain the above copyright
 + *    notice, this list of conditions and the following disclaimer.
 + * 2. Redistributions in binary form must reproduce the above copyright
 + *    notice, this list of conditions and the following disclaimer in the
 + *    documentation and/or other materials provided with the distribution.
 + *
 + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
 + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
 + * ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
 + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
 + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
 + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
 + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 + * SUCH DAMAGE.
 + *
 + * $FreeBSD$
 + */
 +
 +#include <sys/cdefs.h>
 +__FBSDID("$FreeBSD$");
 +
 +#include <sys/param.h>
 +#include <sys/systm.h>
 +#include <sys/bus.h>
 +#include <sys/clock.h>
 +#include <sys/lock.h>
 +#include <sys/mutex.h>
 +#include <sys/kernel.h>
 +#include <sys/module.h>
 +#include <sys/time.h>
 +
 +#include <xen/xen_intr.h>
 +#include <vm/vm.h>
 +#include <vm/pmap.h>
 +#include <machine/pmap.h>
 +#include <xen/hypervisor.h>
 +#include <machine/xen/xen-os.h>
 +#include <machine/xen/xenfunc.h>
 +#include <xen/interface/io/xenbus.h>
 +#include <xen/interface/vcpu.h>
 +#include <machine/cpu.h>
 +
 +#include "clock_if.h"
 +
 +/*
 + * Read the current hypervisor start time (wall clock) from Xen.
 + */
 +static void
 +xen_fetch_wallclock(struct timespec *ts)
 +{ 
 +        shared_info_t *s = HYPERVISOR_shared_info;
 +        uint32_t ts_version;
 +   
 +        do {
 +                ts_version = s->wc_version;
 +                rmb();
 +                ts->tv_sec  = s->wc_sec;
 +                ts->tv_nsec = s->wc_nsec;
 +                rmb();
 +        }
 +        while ((s->wc_version & 1) | (ts_version ^ s->wc_version));
 +}
 +
 +/*
 + * Read the current hypervisor system uptime value from Xen.
 + */
 +static void
 +xen_fetch_uptime(struct timespec *ts)
 +{
 +        shared_info_t           *s = HYPERVISOR_shared_info;
 +        struct vcpu_time_info   *src;
 +	struct shadow_time_info	dst;
 +        uint32_t pre_version, post_version;
 +        
 +        src = &s->vcpu_info[smp_processor_id()].time;
 +
 +        spinlock_enter();
 +        do {
 +                pre_version = dst.version = src->version;
 +                rmb();
 +                dst.system_timestamp  = src->system_time;
 +                rmb();
 +                post_version = src->version;
 +        }
 +        while ((pre_version & 1) | (pre_version ^ post_version));
 +
 +        spinlock_exit();
 +
 +	ts->tv_sec = dst.system_timestamp / 1000000000;
 +	ts->tv_nsec = dst.system_timestamp % 1000000000;
 +}
 +
 +
 +static int
 +xen_rtc_probe(device_t dev)
 +{
 +	device_set_desc(dev, "Xen Hypervisor Clock");
 +	printf("[XEN] xen_rtc_probe: probing Hypervisor RTC clock\n");
 +	if (! HYPERVISOR_shared_info) {
 +		device_printf(dev, "No hypervisor shared page found; RTC can not start.\n");
 +		return (EINVAL);
 +	}
 +	return (0);
 +}
 +
 +static int
 +xen_rtc_attach(device_t dev)
 +{
 +	printf("[XEN] xen_rtc_attach: attaching Hypervisor RTC clock\n");
 +	clock_register(dev, 1000000);
 +	return(0);
 +}
 +
 +static int
 +xen_rtc_settime(device_t dev __unused, struct timespec *ts)
 +{
 +	device_printf(dev, "[XEN] xen_rtc_settime\n");
 +	/*
 +	 * Don't return EINVAL here; just silently fail if the domain isn't privileged enough
 +	 * to set the TOD.
 +	 */
 +	return(0);
 +}
 +
 +/*
 + * The Xen time structures document the hypervisor start time and the
 + * uptime-since-hypervisor-start (in nsec.) They need to be combined
 + * in order to calculate a TOD clock.
 + */
 +static int
 +xen_rtc_gettime(device_t dev, struct timespec *ts)
 +{
 +	struct timespec w_ts, u_ts;
 +
 +	device_printf(dev, "[XEN] xen_rtc_gettime\n");
 +	xen_fetch_wallclock(&w_ts);
 +	device_printf(dev, "[XEN] xen_rtc_gettime: wallclock %ld sec; %ld nsec\n", (long int) w_ts.tv_sec, (long int) w_ts.tv_nsec);
 +	xen_fetch_uptime(&u_ts);
 +	device_printf(dev, "[XEN] xen_rtc_gettime: uptime %ld sec; %ld nsec\n", (long int) u_ts.tv_sec, (long int) u_ts.tv_nsec);
 +
 +	timespecclear(ts);
 +	timespecadd(ts, &w_ts);
 +	timespecadd(ts, &u_ts);
 +
 +	device_printf(dev, "[XEN] xen_rtc_gettime: TOD %ld sec; %ld nsec\n", (long int) ts->tv_sec, (long int) ts->tv_nsec);
 +
 +	return(0);
 +}
 +
 +static void
 +xen_rtc_identify(driver_t *drv, device_t parent)
 +{
 +        BUS_ADD_CHILD(parent, 0, "rtc", 0);
 +}
 +
 +static device_method_t xen_rtc_methods[] = {
 +	/* Device interface */
 +	DEVMETHOD(device_probe,		xen_rtc_probe),
 +	DEVMETHOD(device_attach,	xen_rtc_attach),
 +	DEVMETHOD(device_identify,	xen_rtc_identify),
 +
 +	DEVMETHOD(device_detach,	bus_generic_detach),
 +	DEVMETHOD(device_shutdown,	bus_generic_shutdown),
 +
 +	/* clock interface */
 +	DEVMETHOD(clock_gettime,	xen_rtc_gettime),
 +	DEVMETHOD(clock_settime,	xen_rtc_settime),
 +
 +	{ 0, 0 }
 +};
 +
 +
 +static driver_t xen_rtc_driver = {
 +	"rtc",
 +	xen_rtc_methods,
 +	0
 +};
 +
 +static devclass_t xen_rtc_devclass;
 +
 +DRIVER_MODULE(rtc, nexus, xen_rtc_driver, xen_rtc_devclass, 0, 0);
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-xen 
Responsible-Changed-By: adrian 
Responsible-Changed-When: Thu May 28 04:21:40 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=135008 

From: Adrian Chadd <adrian@freebsd.org>
To: bug-followup@freebsd.org
Cc:  
Subject: Re: kern/135008: FreeBSD-current/Xen timecounter jumps
Date: Thu, 28 May 2009 12:25:45 +0800

 .. now that the RTC code is in and I've figured out whats going on
 there, the problem has become obvious.
 
 Changing the dom0 clock by > 1 second seems to trigger a re-read of
 the wall clock in the DomU. The DomU sets the TOD to the wall clock -
 but this unfortunately sets it to the hypervisor start time rather
 than including the start time + uptime.
 
 The fix is to figure out the code path(s) in the xen clock code and
 fix the TOD update to be correct.

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135008: commit references a PR
Date: Fri, 29 May 2009 13:43:32 +0000 (UTC)

 Author: adrian
 Date: Fri May 29 13:43:21 2009
 New Revision: 193033
 URL: http://svn.freebsd.org/changeset/base/193033
 
 Log:
   Fix the Xen TOD update when the hypervisor wall clock is nudged.
   
   The "wall clock" in the current code is actually the hypervisor start time.
   The time of day is the "start time" plus the hypervisor "uptime".
   
   Large enough bumps in the dom0 clock lead to a hypervisor "bump" which is
   implemented as a bump in the start time, not the uptime. The clock.c routines
   were reading in the hypervisor start time and then using this as the TOD.
   This meant that any hypervisor time bump would cause the FreeBSD DomU to
   set its TOD to the hypervisor start time, rather than the actual TOD.
   
   This fix is a bit hacky and some reshuffling should be done later on
   to clarify what is going on. I've left the wall clock code alone.
   (The code which updates shadow_tv and shadow_tv_version.)
   A new routine adds the uptime to the shadow_tv, which is then used to
   update the TOD.
   
   I've included some debugging so it is obvious when the clock is nudged.
   
   PR:	135008
 
 Modified:
   head/sys/i386/xen/clock.c
 
 Modified: head/sys/i386/xen/clock.c
 ==============================================================================
 --- head/sys/i386/xen/clock.c	Fri May 29 13:36:06 2009	(r193032)
 +++ head/sys/i386/xen/clock.c	Fri May 29 13:43:21 2009	(r193033)
 @@ -87,6 +87,7 @@ __FBSDID("$FreeBSD$");
  #include <machine/xen/xenfunc.h>
  #include <xen/interface/vcpu.h>
  #include <machine/cpu.h>
 +#include <machine/xen/xen_clock_util.h>
  
  /*
   * 32-bit time_t's can't reach leap years before 1904 or after 2036, so we
 @@ -236,6 +237,15 @@ static void update_wallclock(void)
  
  }
  
 +static void
 +add_uptime_to_wallclock(void)
 +{
 +	struct timespec ut;
 +
 +	xen_fetch_uptime(&ut);
 +	timespecadd(&shadow_tv, &ut);
 +}
 +
  /*
   * Reads a consistent set of time-base values from Xen, into a shadow data
   * area. Must be called with the xtime_lock held for writing.
 @@ -331,7 +341,9 @@ clkintr(void *arg)
  	 */
  	
  	if (shadow_tv_version != HYPERVISOR_shared_info->wc_version) {
 +		printf("[XEN] hypervisor wallclock nudged; nudging TOD.\n");
  		update_wallclock();
 +		add_uptime_to_wallclock();
  		tc_setclock(&shadow_tv);
  	}
  	
 @@ -542,6 +554,7 @@ domu_inittodr(time_t base)
  	struct timespec ts;
  
  	update_wallclock();
 +	add_uptime_to_wallclock();
  	
  	RTC_LOCK;
  	
 @@ -591,6 +604,7 @@ domu_resettodr(void)
  		op.u.settime.system_time = shadow->system_timestamp;
  		HYPERVISOR_dom0_op(&op);
  		update_wallclock();
 +		add_uptime_to_wallclock();
  	} else if (independent_wallclock) {
  		/* notyet */
  		;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->closed 
State-Changed-By: brucec 
State-Changed-When: Sat Mar 6 00:11:03 UTC 2010 
State-Changed-Why:  
Fixed in r193033. 


Responsible-Changed-From-To: freebsd-xen->brucec  
Responsible-Changed-By: brucec 
Responsible-Changed-When: Sat Mar 6 00:11:03 UTC 2010 
Responsible-Changed-Why:  
Track. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135008 
>Unformatted:
