From nobody@FreeBSD.org  Sat May 30 06:52:38 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D4631106566C
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 30 May 2009 06:52:38 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id C2CD38FC0A
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 30 May 2009 06:52:38 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n4U6qcds035960
	for <freebsd-gnats-submit@FreeBSD.org>; Sat, 30 May 2009 06:52:38 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n4U6qcE3035959;
	Sat, 30 May 2009 06:52:38 GMT
	(envelope-from nobody)
Message-Id: <200905300652.n4U6qcE3035959@www.freebsd.org>
Date: Sat, 30 May 2009 06:52:38 GMT
From: Adrian Chadd <adrian@FreeBSD.org>
To: freebsd-gnats-submit@FreeBSD.org
Subject: FreeBSD-current/Xen SMP doesn't function at all after the AP's are loaded
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         135069
>Category:       kern
>Synopsis:       [xen] FreeBSD-current/Xen SMP doesn't function at all after the AP's are loaded
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-xen
>State:          patched
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat May 30 07:00:09 UTC 2009
>Closed-Date:    
>Last-Modified:  Tue Mar 01 10:15:58 EST 2011
>Originator:     Adrian Chadd
>Release:        FreeBSD-current i386 r182895
>Organization:
>Environment:
FreeBSD freebsd1 8.0-CURRENT FreeBSD 8.0-CURRENT #63: Sat May 30 06:33:21 UTC 2009     adrian@agnus.home.cacheboy.net:/local/AGNUS_DATA_1/adrian/xen/obj-head_r192895/home/adrian/work/freebsd/xen/svn/head/sys/XEN  i38
>Description:
FreeBSD-current/Xen doesn't function with more than one CPU configured.
>How-To-Repeat:
Run freebsd-current/xen with more than one virtual CPU.
>Fix:


>Release-Note:
>Audit-Trail:

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135069: commit references a PR
Date: Sat, 30 May 2009 08:53:26 +0000 (UTC)

 Author: adrian
 Date: Sat May 30 08:53:13 2009
 New Revision: 193085
 URL: http://svn.freebsd.org/changeset/base/193085
 
 Log:
   Make ipi_cpu() function as intended.
   
   IPI's in Xen are implemented through hypervisor event channels.
   The MP code creates a pair of IRQs for each base IPI per CPU
   (one for IPI function dispatch calls, one for IPI bitmap dispatch calls.)
   Using PCPU_GET() was returning the IRQ of the IPI handler for the
   current CPU; thus calls to ipi_cpu() were sending itself a message.
   Instead, looking up the IPI in the target CPU ipi-to-irq map is needed.
   
   Note: This doesn't fix Xen SMP (far from it!) but it at least
   sends IPI's to the right places. Next - sending IPIs..
   
   PR:	135069
 
 Modified:
   head/sys/xen/evtchn/evtchn.c
 
 Modified: head/sys/xen/evtchn/evtchn.c
 ==============================================================================
 --- head/sys/xen/evtchn/evtchn.c	Sat May 30 07:33:32 2009	(r193084)
 +++ head/sys/xen/evtchn/evtchn.c	Sat May 30 08:53:13 2009	(r193085)
 @@ -225,12 +225,15 @@ evtchn_do_upcall(struct trapframe *frame
  	}
  }
  
 +/*
 + * Send an IPI from the current CPU to the destination CPU.
 + */
  void
  ipi_pcpu(unsigned int cpu, int vector) 
  { 
          int irq;
  
 -	irq = PCPU_GET(ipi_to_irq[vector]);
 +	irq = pcpu_find(cpu)->pc_ipi_to_irq[vector];
  	
          notify_remote_via_irq(irq); 
  } 
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/135069: commit references a PR
Date: Sun, 31 May 2009 08:11:53 +0000 (UTC)

 Author: adrian
 Date: Sun May 31 08:11:39 2009
 New Revision: 193154
 URL: http://svn.freebsd.org/changeset/base/193154
 
 Log:
   Fix the MP IPI code to differentiate between bitmapped IPIs and function IPIs.
   
   This attempts to fix the IPI handling code to correctly differentiate
   between bitmapped IPIs and function IPIs. The Xen IPIs were on low numbers
   which clashed with the bitmapped IPIs.
   
   This commit bumps those IPI numbers up to 240 and above (just like in the i386
   code) and fiddles with the ipi_vectors[] logic to call the correct function.
   
   This still isn't "right". Specifically, the IPI code may work fine for TLB
   shootdown events but the rendezvous/lazypmap IPIs are thrown by calling ipi_*()
   routines which don't set the call_func stuff (function id, addr1, addr2) that
   the TLB shootdown events are. So the Xen SMP support is still broken.
   
   PR:		135069
 
 Modified:
   head/sys/i386/include/apicvar.h
   head/sys/i386/xen/mp_machdep.c
 
 Modified: head/sys/i386/include/apicvar.h
 ==============================================================================
 --- head/sys/i386/include/apicvar.h	Sun May 31 07:25:24 2009	(r193153)
 +++ head/sys/i386/include/apicvar.h	Sun May 31 08:11:39 2009	(r193154)
 @@ -114,14 +114,14 @@
  
  #define	APIC_IPI_INTS	(APIC_LOCAL_INTS + 2)
  #ifdef XEN
 -#define	IPI_RENDEZVOUS		(2)	/* Inter-CPU rendezvous. */
 -#define	IPI_INVLTLB		(3)	/* TLB Shootdown IPIs */
 -#define	IPI_INVLPG		(4)
 -#define	IPI_INVLRNG		(5)
 -#define	IPI_INVLCACHE		(6)
 -#define	IPI_LAZYPMAP		(7)	/* Lazy pmap release. */
 +#define	IPI_RENDEZVOUS		(APIC_IPI_INTS)	/* Inter-CPU rendezvous. */
 +#define	IPI_INVLTLB		(APIC_IPI_INTS + 1)	/* TLB Shootdown IPIs */
 +#define	IPI_INVLPG		(APIC_IPI_INTS + 2)
 +#define	IPI_INVLRNG		(APIC_IPI_INTS + 3)
 +#define	IPI_INVLCACHE		(APIC_IPI_INTS + 4)
 +#define	IPI_LAZYPMAP		(APIC_IPI_INTS + 5)	/* Lazy pmap release. */
  /* Vector to handle bitmap based IPIs */
 -#define	IPI_BITMAP_VECTOR	(8)
 +#define	IPI_BITMAP_VECTOR	(APIC_IPI_INTS + 6)
  
  #else
  #define	IPI_RENDEZVOUS	(APIC_IPI_INTS)		/* Inter-CPU rendezvous. */
 
 Modified: head/sys/i386/xen/mp_machdep.c
 ==============================================================================
 --- head/sys/i386/xen/mp_machdep.c	Sun May 31 07:25:24 2009	(r193153)
 +++ head/sys/i386/xen/mp_machdep.c	Sun May 31 08:11:39 2009	(r193154)
 @@ -350,17 +350,11 @@ iv_lazypmap(uintptr_t a, uintptr_t b)
  	atomic_add_int(&smp_tlb_wait, 1);
  }
  
 -
 -static void
 -iv_noop(uintptr_t a, uintptr_t b)
 -{
 -	atomic_add_int(&smp_tlb_wait, 1);
 -}
 -
 -static call_data_func_t *ipi_vectors[IPI_BITMAP_VECTOR] = 
 +/*
 + * These start from "IPI offset" APIC_IPI_INTS
 + */
 +static call_data_func_t *ipi_vectors[6] = 
  {
 -  iv_noop,
 -  iv_noop,
    iv_rendezvous,
    iv_invltlb,
    iv_invlpg,
 @@ -419,10 +413,11 @@ smp_call_function_interrupt(void *unused
  	atomic_t *started = &call_data->started;
  	atomic_t *finished = &call_data->finished;
  
 -	if (call_data->func_id > IPI_BITMAP_VECTOR)
 +	/* We only handle function IPIs, not bitmap IPIs */
 +	if (call_data->func_id < APIC_IPI_INTS || call_data->func_id > IPI_BITMAP_VECTOR)
  		panic("invalid function id %u", call_data->func_id);
  	
 -	func = ipi_vectors[call_data->func_id];
 +	func = ipi_vectors[call_data->func_id - APIC_IPI_INTS];
  	/*
  	 * Notify initiating CPU that I've grabbed the data and am
  	 * about to execute the function
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
Responsible-Changed-From-To: freebsd-bugs->freebsd-emulation 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Jun 2 02:14:37 UTC 2009 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135069 
Responsible-Changed-From-To: freebsd-emulation->freebsd-xen 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Tue Jun 2 17:38:04 UTC 2009 
Responsible-Changed-Why:  
more specific assignment. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135069 
State-Changed-From-To: open->patched 
State-Changed-By: eadler 
State-Changed-When: Tue Mar 1 10:15:57 EST 2011 
State-Changed-Why:  
committed in head (r193154) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=135069 
>Unformatted:
