From nobody@FreeBSD.org  Thu Apr 19 19:03:31 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 5A44C106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 19 Apr 2012 19:03:31 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 4418A8FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 19 Apr 2012 19:03:31 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3JJ3ULe047127
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 19 Apr 2012 19:03:30 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q3JJ3U6x047122;
	Thu, 19 Apr 2012 19:03:30 GMT
	(envelope-from nobody)
Message-Id: <201204191903.q3JJ3U6x047122@red.freebsd.org>
Date: Thu, 19 Apr 2012 19:03:30 GMT
From: Claudius Herder <claudius@ambtec.de>
To: freebsd-gnats-submit@FreeBSD.org
Subject: eventtimer tsc smp_test unreliable
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         167106
>Category:       kern
>Synopsis:       [eventtimers] eventtimer tsc smp_test unreliable
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    jimharris
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 19 19:10:10 UTC 2012
>Closed-Date:    Wed Aug 29 04:27:51 UTC 2012
>Last-Modified:  Wed Aug 29 04:27:51 UTC 2012
>Originator:     Claudius Herder
>Release:        FreeBSD 9.0-RELEASE
>Organization:
>Environment:
FreeBSD server.ambtec.de 9.0-RELEASE FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012     claudius@server.ambtec.de:/usr/obj/usr/src/sys/CUSTOM  amd64
>Description:
kernel switches timecounter after reboots from TSC-low to HPET and vice versa

Apr 19 19:38:07 server kernel: FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012
Apr 19 19:38:07 server kernel: SMP: passed TSC synchronization test

Apr 19 20:06:37 server kernel: FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012
Apr 19 20:06:37 server kernel: SMP: failed TSC synchronization test

Base Board Information
        Manufacturer: ASUSTeK COMPUTER INC.
        Product Name: P8H67-M PRO

BIOS Information
        Vendor: American Megatrends Inc.
        Version: 3604
        Release Date: 02/23/2012

dmesg:

Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012
    claudius@server.ambtec.de:/usr/obj/usr/src/sys/CUSTOM amd64
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (3411.56-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x17bae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16436305920 (15674 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
acpi0: <ALASKA A M I> on motherboard
ACPI Error: [RAMB] Namespace lookup failure, AE_NOT_FOUND (20110527/psargs-392)
ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20110527/nsinit-380)
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
pci0: <simple comms> at device 22.0 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xf7d04000-0xf7d043ff irq 23 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0: <EHCI (generic) USB 2.0 controller> on ehci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci3: <ACPI PCI bus> on pcib3
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xf7c00000-0xf7c07fff irq 17 at device 0.0 on pci3
xhci0: 32 byte context size.
usbus1 on xhci0
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.6 on pci0
pci4: <ACPI PCI bus> on pcib4
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 18 at device 0.0 on pci4
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 54:04:a6:7e:eb:34
pcib5: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 19 at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xf7d03000-0xf7d033ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2: <EHCI (generic) USB 2.0 controller> on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xf7d02000-0xf7d027ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 5.0Gbps Super Speed USB v3.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <0x1b21> at usbus1
uhub1: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ST3000DM001-9YN166 CC47> ATA-8 SATA 3.x device
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST3000DM001-9YN166 CC47> ATA-8 SATA 3.x device
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #7 Launched!
Timecounter "TSC-low" frequency 13326397 Hz quality 1000
GEOM_MIRROR: Device mirror/swap launched (2/2).
uhub1: 4 ports with 4 removable, self powered
Trying to mount root from zfs:system/rootfs [rw,noatime]...
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen2.2: <vendor 0x8087> at usbus2
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
uhub3: 6 ports with 6 removable, self powered
uhub4: 8 ports with 8 removable, self powered
GEOM_ELI: Device mirror/swap.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware

>How-To-Repeat:
For my system it is sufficient to reboot, I first noticed this after a
power failure because the ntp drift changed from -3ppm to +20ppm but it
does not happen every time, in 10 reboots I got 4 timecounter changes.
>Fix:


>Release-Note:
>Audit-Trail:

From: Jim Harris <jimharris@freebsd.org>
To: bug-followup@FreeBSD.org, claudius@ambtec.de
Cc:  
Subject: Re: kern/167106: [eventtimers] eventtimer tsc smp_test unreliable
Date: Tue, 24 Jul 2012 22:30:31 -0700

 I am almost certain kern/167106 is fixed by r238755, committed today.
 kib@ has already approved this to be MFC'd to stable/9 for upcoming
 9.1 release.  MFC will happen on Friday (27th July) barring any
 unforeseen issues with the patch.
 
 Full commit message below.
 
 Regards,
 
 -Jim
 
 
 Author: jimharris
 Date: Tue Jul 24 22:10:11 2012
 New Revision: 238755
 URL: http://svn.freebsd.org/changeset/base/238755
 
 Log:
   Add rmb() to tsc_read_##x to enforce serialization of rdtsc captures.
 
   Intel Architecture Manual specifies that rdtsc instruction is not serialized,
   so without this change, TSC synchronization test would periodically fail,
   resulting in use of HPET timecounter instead of TSC-low.  This caused
   severe performance degradation (40-50%) when running high IO/s
 workloads due to
   HPET MMIO reads and GEOM stat collection.
 
   Tests on Xeon E5-2600 (Sandy Bridge) 8C systems were seeing TSC
 synchronization
   fail approximately 20% of the time.
 
   Sponsored by: Intel
   Reviewed by: kib
   MFC after: 3 days
 
 Modified:
   head/sys/x86/x86/tsc.c
 
 Modified: head/sys/x86/x86/tsc.c
 ==============================================================================
 --- head/sys/x86/x86/tsc.c      Tue Jul 24 20:15:41 2012        (r238754)
 +++ head/sys/x86/x86/tsc.c      Tue Jul 24 22:10:11 2012        (r238755)
 @@ -328,6 +328,7 @@ init_TSC(void)
 
  #ifdef SMP
 
 +/* rmb is required here because rdtsc is not a serializing instruction. */
  #define        TSC_READ(x)                     \
  static void                            \
  tsc_read_##x(void *arg)                        \
 @@ -335,6 +336,7 @@ tsc_read_##x(void *arg)                     \
         uint32_t *tsc = arg;            \
         u_int cpu = PCPU_GET(cpuid);    \
                                         \
 +       rmb();                          \
         tsc[cpu * 3 + x] = rdtsc32();   \
  }
  TSC_READ(0)
State-Changed-From-To: open->closed 
State-Changed-By: jimharris 
State-Changed-When: Wed Aug 29 04:26:14 UTC 2012 
State-Changed-Why:  
Fixed in head in r238755. 
MFC to stable/9 in r238910. 


Responsible-Changed-From-To: freebsd-bugs->jimharris 
Responsible-Changed-By: jimharris 
Responsible-Changed-When: Wed Aug 29 04:26:14 UTC 2012 
Responsible-Changed-Why:  
Fixed in head in r238755. 
MFC to stable/9 in r238910. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=167106 
>Unformatted:
