From nobody@FreeBSD.org  Mon Feb 27 06:42:09 2006
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 8CD5216A420
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 27 Feb 2006 06:42:09 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [216.136.204.117])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 33C1643D48
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 27 Feb 2006 06:42:09 +0000 (GMT)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.13.1/8.13.1) with ESMTP id k1R6g8IS025892
	for <freebsd-gnats-submit@FreeBSD.org>; Mon, 27 Feb 2006 06:42:08 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.13.1/8.13.1/Submit) id k1R6g8VT025891;
	Mon, 27 Feb 2006 06:42:08 GMT
	(envelope-from nobody)
Message-Id: <200602270642.k1R6g8VT025891@www.freebsd.org>
Date: Mon, 27 Feb 2006 06:42:08 GMT
From: Arthur Hartwig <Arthur.Hartwig@nokia.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: cpu_spinwait calls missing
X-Send-Pr-Version: www-2.3

>Number:         93887
>Category:       kern
>Synopsis:       [kernel] cpu_spinwait calls missing in subr_smp.c
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Feb 27 06:50:04 GMT 2006
>Closed-Date:    
>Last-Modified:  Tue Feb 28 09:25:05 GMT 2006
>Originator:     Arthur Hartwig
>Release:        6.1-BETA1
>Organization:
Nokia
>Environment:
oz-net-11# uname -a
FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006     root@oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11  amd64


>Description:
There are five busy wait loops in kern/subr_smp.c which would be more
"multi-processor friendly" on i386 and amd64 architectures if they included
a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes
a pause instruction which the Intel IA32 architecture manual says "improves
the performance of spin wait loops."

Loops which could include the call to cpu_spinwait() occur in stop_cpus(),
stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action()
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:

From: Kris Kennaway <kris@obsecurity.org>
To: Arthur Hartwig <Arthur.Hartwig@nokia.com>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/93887: cpu_spinwait calls missing
Date: Mon, 27 Feb 2006 18:29:44 -0500

 Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
 restart_cpus()?  They're used e.g. as preparation to break into/out of
 the debugger, and AFAICT they're not in the critical path for
 anything.
 
 Not sure about smp_rendezvous_action(), maybe that one is OK.
 
 Kris

From: Arthur Hartwig <arthur.hartwig@nokia.com>
To: ext Kris Kennaway <kris@obsecurity.org>
Cc: freebsd-gnats-submit@FreeBSD.org
Subject: Re: misc/93887: cpu_spinwait calls missing
Date: Tue, 28 Feb 2006 12:32:08 +1000

 >Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
 >restart_cpus()?  They're used e.g. as preparation to break into/out of
 >the debugger, and AFAICT they're not in the critical path for
 >anything.
 >
 >Not sure about smp_rendezvous_action(), maybe that one is OK.
 >
 >Kris
 >
 Good question which didn't occur to me because I was coming from another 
 context. I was looking at using these functions to support a particular 
 piece of "hotswap" hardware. The hardware generates an interrupt to say 
 a PCI device has been removed (or plugged in). In the removal case some 
 work needs to be done to remap to main memory the memory address space 
 assigned to the device being removed so that if a device driver is 
 accessing the device's memory space it won't cause the system to die. On 
 a UP system this is "easy" in that because the hotswap interrupt is 
 running we know the driver for the removed device is not currently 
 running and even if it was running before the interrupt by the time it 
 runs again the memory remapping will have occurred and it will be safe 
 for the driver to access the memory space previously assigned to the 
 removed device.However on a MP system the driver could be executing 
 concurrently on another CPU and cause the system to die by accessing 
 memory that is no longer present.
 
 I thought of using these functions (stop_cpus() etc) to momentarily 
 "pause" all other CPUs while adjusting the memory mapping. It would be 
 of advantage to cause other CPUs to pause as quickly as possible to 
 minimise the likelihood that the driver for the removed device will 
 attempt to access memory on the removed device. I have not been able to 
 find any data on how long I have between the interrupt to say a device 
 is being removed and the actual removal of the device. Nor have I any 
 data on how long a busy CPU of a hyperthreaded pair (for example) is 
 able to "lock out" its hyperthreaded mate.
 
 The referenced functions are clearly adequate for their existing use by 
 the debugger so I accept that I might be better off making a customised 
 copy.
 
 I still have doubts about the effectiveness of the approach using 
 stop_cpus() and co and I'm inclined to see if I can get away with saying 
 "you can't pull out a board untill you have disabled all the devices on 
 that board" but this is somewhat restricted compared with what users 
 have become accustomed to.
 
 Arthur
>Unformatted:
