From steve@sse0691.bri.hp.com  Mon Jul 31 04:58:31 2000
Return-Path: <steve@sse0691.bri.hp.com>
Received: from bbnrel4.net.external.hp.com (bbnrel4.net.external.hp.com [155.208.254.68])
	by hub.freebsd.org (Postfix) with ESMTP id 9314E37BC08
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Jul 2000 04:58:24 -0700 (PDT)
	(envelope-from steve@sse0691.bri.hp.com)
Received: from hpcpbla.bri.hp.com (hpcpbla.bri.hp.com [15.144.112.65])
	by bbnrel4.net.external.hp.com (Postfix) with ESMTP id CACD011BA7
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Jul 2000 13:58:19 +0200 (METDST)
Received: from sse0691.bri.hp.com (sse0691.bri.hp.com [15.144.0.53])
	by hpcpbla.bri.hp.com (8.9.3 (PHNE_18979)/8.9.3 SMKit7.0) with ESMTP id MAA23587
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 31 Jul 2000 12:58:18 +0100 (BST)
Received: (from steve@localhost)
	by sse0691.bri.hp.com (8.9.3/8.9.3) id NAA09865;
	Mon, 31 Jul 2000 13:00:55 +0100 (BST)
	(envelope-from steve)
Message-Id: <200007311200.NAA09865@sse0691.bri.hp.com>
Date: Mon, 31 Jul 2000 13:00:55 +0100 (BST)
From: Steve Roome <steve@sse0691.bri.hp.com>
Reply-To: steve@sse0691.bri.hp.com
To: FreeBSD-gnats-submit@freebsd.org
Subject: IO APIC problems
X-Send-Pr-Version: 3.2

>Number:         20312
>Category:       kern
>Synopsis:       IO APIC problems
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    tegge
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jul 31 05:00:00 PDT 2000
>Closed-Date:    Mon Oct 30 23:18:47 PST 2000
>Last-Modified:  Mon Oct 30 23:20:58 PST 2000
>Originator:     Steve Roome
>Release:        FreeBSD 4.1-RC i386
>Organization:
>Environment:

MPTABLE output :

===============================================================================

MPTable, version 2.0.15

 looking for EBDA pointer @ 0x040e, found, searching EBDA @ 0x0009f800
 searching CMOS 'top of mem' @ 0x0009f400 (637K)
 searching default 'top of mem' @ 0x0009fc00 (639K)
 searching BIOS @ 0x000f0000

 MP FPS found in BIOS @ physical addr: 0x000f99a0

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:			BIOS
  physical address:		0x000f99a0
  signature:			'_MP_'
  length:			16 bytes
  version:			1.1
  checksum:			0x4b
  mode:				Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:		0x000f99b0
  signature:			'PCMP'
  base table length:		236
  version:			1.1
  checksum:			0xe0
  OEM ID:			'HP      '
  Product ID:			'XX          '
  OEM table pointer:		0x00000000
  OEM table size:		0
  entry count:			21
  local APIC address:		0xfee00000
  extended table length:	0
  extended table checksum:	0

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
Processors:	APIC ID	Version	State		Family	Model	Step	Flags
		 1	 0x11	 BSP, usable	 6	 1	 0	 0xfbff
		 0	 0x11	 AP, usable	 6	 1	 0	 0xfbff
--
Bus:		Bus ID	Type
		 1	 ISA   
		 0	 PCI   
--
I/O APICs:	APIC ID	Version	State		Address
		16	 0x11	 usable		 0xfec00000
--
I/O Ints:	Type	Polarity    Trigger	Bus ID	 IRQ	APIC ID	PIN#
		INT	 conforms    conforms	     1	   1	    255	   1
		INT	 conforms    conforms	     1	   0	    255	   2
		INT	 conforms    conforms	     1	   3	    255	   3
		INT	 conforms    conforms	     1	   4	    255	   4
		INT	 conforms    conforms	     1	   5	    255	   5
		INT	 conforms    conforms	     1	   6	    255	   6
		INT	 conforms    conforms	     1	   7	    255	   7
		INT	 conforms    conforms	     1	   8	    255	   8
		INT	 conforms    conforms	     1	   9	    255	   9
		INT	 conforms    conforms	     1	  10	    255	  10
		INT	 conforms    conforms	     1	  11	    255	  11
		INT	 conforms    conforms	     1	  12	    255	  12
		INT	 conforms    conforms	     1	  13	    255	  13
		INT	 conforms    conforms	     1	  14	    255	  14
		INT	 conforms    conforms	     1	  15	    255	  15
--
Local Ints:	Type	Polarity    Trigger	Bus ID	 IRQ	APIC ID	PIN#
		NMI	 conforms    conforms	     1	   2	    255	   1

-------------------------------------------------------------------------------

# SMP kernel config file options:


# Required:
options		SMP			# Symmetric MultiProcessor Kernel
options		APIC_IO			# Symmetric (APIC) I/O

# Optional (built-in defaults will work in most cases):
#options		NCPU=2			# number of CPUs
#options		NBUS=2			# number of busses
#options		NAPIC=1			# number of IO APICs
#options		NINTR=24		# number of INTs

===============================================================================

-----

Kernel Config :

machine		i386
cpu		I686_CPU		# aka Pentium Pro(tm)
ident		MOOSE
maxusers	64

## SMP Stuff
options 	SMP			# Symmetric MultiProcessor Kernel
options 	APIC_IO			# Symmetric (APIC) I/O

## General options
options 	NO_F00F_HACK
options 	COMPAT_43
options 	SYSVSHM
options 	SYSVSEM
options 	SYSVMSG
options 	KTRACE			#kernel tracing
options 	PERFMON
options 	UCONSOLE
options 	INET			#Internet communications protocols
options 	ICMP_BANDLIM
options 	FFS			#Fast filesystem
options 	MFS			#Memory File System
options 	NFS			#Network File System
options 	CD9660			#ISO 9660 filesystem
options 	KERNFS			#Kernel filesystem
options 	PROCFS			#Process filesystem
options 	FFS_ROOT		#FFS usable as root device
options 	SOFTUPDATES
options 	P1003_1B
options 	_KPOSIX_PRIORITY_SCHEDULING
options 	_KPOSIX_VERSION=199309L
options 	VESA

pseudo-device	ether			#Generic Ethernet
pseudo-device	loop			#Network loopback device
pseudo-device	bpf			#Berkeley packet filter

device		scbus			#base SCSI code
device		ch			#SCSI media changers
device		da			#SCSI direct access devices (aka disks)
device		sa			#SCSI tapes
device		cd			#SCSI CD-ROMs
device		pass			#CAM passthrough driver

pseudo-device	pty		#Pseudo ttys
pseudo-device	gzip		#Exec gzipped a.out's
pseudo-device	vn		#Vnode driver (turns a file into a device)
pseudo-device	md		#Memory/malloc disk

device		isa
device		pci

device		atkbdc0	at isa? port IO_KBD
device		atkbd0	at atkbdc? irq 1
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
device		psm0	at atkbdc? irq 12

device		vga0	at isa?

# Splash screen at start up!  Screen savers require this too.
pseudo-device	splash

device		sc0	at isa?
options 	MAXCONS=16		# number of virtual consoles
options 	SC_DISABLE_REBOOT	# disable reboot key sequence
options 	SC_HISTORY_SIZE=200	# number of history buffer lines

device		npx0	at nexus? port IO_NPX flags 0x0 irq 13

device		ata
device		atadisk		# ATA disk drives
device		atapicd		# ATAPI CDROM drives

device		fdc0	at isa? port IO_FD1 irq 6 drq 2
device		fd0	at fdc0 drive 0
device lnc

device		pcm
device		sbc0	at isa? port 0x220 irq 5 drq 1 flags 0x15

device		ahc		# AHA2940 and onboard AIC7xxx devices
options 	AHC_ALLOW_MEMIO

device		miibus

device		sio0	at isa? port IO_COM1 flags 0x10 irq 4
device		sio1	at isa? port IO_COM2 irq 3



>Description:

Panic's like this :

Features = 0xfbff <.....CPU features of a PPro.....>
real memory = 201326592 (196608K bytes)
avail memory = 192376832 (187868K bytes)
panic: can't control IO APIC #0 ID, reg : 0x00000000
mp_lock = 0000000a; cpuid = 0; lapic.id = 01000000
Uptime : 0s

>How-To-Repeat:

	Boot a kernel built with the config file supplied on FreeBSD 4.1-RC.
	Maybe be hardware related, so to repeat you'd need this hardware which
	is a Dual Pentium Pro at 200MHz in a HP Vectra XU 6/200 (old machine
	which has worked fine with SMP since 3-0 current or thereabouts.)

>Fix:

>Release-Note:
>Audit-Trail:

From: Tor.Egge@fast.no
To: steve@sse0691.bri.hp.com
Cc: FreeBSD-gnats-submit@FreeBSD.ORG, msmith@FreeBSD.ORG
Subject: Re: kern/20312: IO APIC problems
Date: Sat, 05 Aug 2000 08:09:39 +0200

 The problem is due to a incomplete fix for panics on machines 
 where the BIOS doesn't set the physical IO APIC ids, cf. PR 18919.
 
 msmith      2000/05/31 14:37:29 PDT
 
   Revision  Changes    Path
   1.117     +3 -8      src/sys/i386/i386/mp_machdep.c
   1.39      +19 -1     src/sys/i386/i386/mpapic.c
   1.52      +2 -1      src/sys/i386/include/smp.h
 
 
 Section 3.6.6 of the Intel MP spec describes the operating system as
 responsible for verifying that the IO APIC ids are unique and
 acceptable.  If also contains a statement restricting the acceptable
 ids to the lowest possible ids left over after local apic id assignment.
 Thus the following patch might be applicable.
 
 - Tor Egge
 
 Index: mp_machdep.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/i386/i386/mp_machdep.c,v
 retrieving revision 1.117
 diff -u -r1.117 mp_machdep.c
 --- mp_machdep.c	2000/05/31 21:37:24	1.117
 +++ mp_machdep.c	2000/08/05 05:45:50
 @@ -1013,6 +1029,42 @@
  	}
  }
  
 +
 +static void
 +swap_apic_id(int oldid, int newid)
 +{
 +	int x;
 +	int apic;
 +	int oapic;
 +
 +
 +	if (oldid == newid)
 +		return;
 +
 +	printf("IOAPIC#%d assigned physical APIC ID %d\n",
 +	       ID_TO_IO(oldid), newid);
 +
 +	/* Swap IOAPIC physical IDs in interrupt entries */
 +	for (x = 0; x < nintrs; x++) {
 +		if (io_apic_ints[x].dst_apic_id == oldid)
 +			io_apic_ints[x].dst_apic_id = newid;
 +		else if (io_apic_ints[x].dst_apic_id == newid)
 +			io_apic_ints[x].dst_apic_id = oldid;
 +	}
 +
 +	/* Swap IOAPIC physical IDs in IO_TO_ID and ID_TO_IO mappings */
 +	apic = ID_TO_IO(oldid);
 +	oapic = ID_TO_IO(newid);
 +	if (oapic >= 0 && IO_TO_ID(oapic) == newid) {
 +		ID_TO_IO(oldid) = oapic;
 +		IO_TO_ID(oapic) = oldid;
 +	} else {
 +		ID_TO_IO(oldid) = -1;
 +	}
 +	ID_TO_IO(newid) = apic;
 +	IO_TO_ID(apic) = newid;
 +}
 +
  /*
   * parse an Intel MP specification table
   */
 @@ -1024,6 +1076,9 @@
  	int	bus_0 = 0;	/* Stop GCC warning */
  	int	bus_pci = 0;	/* Stop GCC warning */
  	int	num_pci_bus;
 +	int	apic;		/* logical IO APIC ID */
 +	int     new_physid;	/* Free physical IO APIC ID */
 +	int	max_physid;	/* Max physical IO APIC ID */
  
  	/*
  	 * Fix mis-numbering of the PCI bus and its INT entries if the BIOS
 @@ -1076,6 +1131,21 @@
  			}
  		}
  	}
 +	/* Assign IOAPIC ids */
 +	new_physid = 0;		/* Start with first possible APIC ID */
 +	max_physid = 15;	/* physical APIC IDs are in range 0..15 */
 +	for (apic = 0; apic < mp_napics; ++apic) { /* For all IO APICs */
 +		/* Find next free APIC ID (not used by any CPU) */
 +		while (new_physid <= max_physid &&
 +		       ID_TO_CPU(new_physid) >= 0 &&
 +		       CPU_TO_ID(ID_TO_CPU(new_physid)) == new_physid)
 +			new_physid++;
 +		
 +		if (new_physid < max_physid) {
 +			swap_apic_id(IO_TO_ID(apic), new_physid);
 +			new_physid++;
 +		}
 +	}
  }
  
  
 @@ -1602,7 +1672,6 @@
  {
  	int     ap_cpu_id;
  #if defined(APIC_IO)
 -	u_int32_t ux;
  	int     io_apic_id;
  	int     pin;
  #endif	/* APIC_IO */
 
State-Changed-From-To: open->analyzed 
State-Changed-By: sheldonh 
State-Changed-When: Mon Aug 7 01:41:37 PDT 2000 
State-Changed-Why:  
Fixed in HEAD by tegge: 
rev 1.53 of src/sys/i386/include/smp.h 
rev 1.118 of src/sys/i386/i386/mp_machdep.c 
rev 1.40 of src/sys/i386/i386/mpapic.c 


Responsible-Changed-From-To: freebsd-bugs->tegge 
Responsible-Changed-By: sheldonh 
Responsible-Changed-When: Mon Aug 7 01:41:37 PDT 2000 
Responsible-Changed-Why:  
Tor's reminder. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20312 

From: Steve Roome <steve@sse0691.bri.hp.com>
To: sheldonh@FreeBSD.org
Cc: freebsd-bugs@FreeBSD.org, tegge@FreeBSD.org
Subject: Re: kern/20312: IO APIC problems
Date: Thu, 10 Aug 2000 15:50:17 +0100

 On Mon, Aug 07, 2000 at 01:43:17AM -0700, sheldonh@FreeBSD.org wrote:
 > Synopsis: IO APIC problems
 > 
 > State-Changed-From-To: open->analyzed
 > State-Changed-By: sheldonh
 > State-Changed-When: Mon Aug 7 01:41:37 PDT 2000
 > State-Changed-Why: 
 > Fixed in HEAD by tegge:
 > rev 1.53 of src/sys/i386/include/smp.h
 > rev 1.118 of src/sys/i386/i386/mp_machdep.c
 > rev 1.40 of src/sys/i386/i386/mpapic.c
 > 
 > 
 > Responsible-Changed-From-To: freebsd-bugs->tegge
 > Responsible-Changed-By: sheldonh
 > Responsible-Changed-When: Mon Aug 7 01:41:37 PDT 2000
 > Responsible-Changed-Why: 
 > Tor's reminder.
 > 
 > http://www.freebsd.org/cgi/query-pr.cgi?pr=20312
 
 Firstly, thanks again for taking a look at this, still doesn't work
 though =(
 
 I grabbed the newer versions from these files and stuck them in my
 source tree, recompiled and it still panics - so I'm back to a non-smp
 kernel again.
 
 Just to check the versions.
 
 steve@moose 5 0 % fgrep "\$FreeBSD" /sys/i386/include/smp.h /sys/i386/i386/mp_machdep.c /sys/i386/i386/mpapic.c    ~/  3:26pm
 /sys/i386/include/smp.h: * $FreeBSD: src/sys/i386/include/smp.h,v 1.53 2000/08/06 00:04:02 tegge Exp $
 /sys/i386/i386/mp_machdep.c: * $FreeBSD: src/sys/i386/i386/mp_machdep.c,v 1.118 2000/08/06 00:04:02 tegge Exp $
 /sys/i386/i386/mpapic.c: * $FreeBSD: src/sys/i386/i386/mpapic.c,v 1.40 2000/08/06 00:04:03 tegge Exp $
 
 I've just copied the smp.h into /usr/include/machine as well, but now I can't seem to
 build anything at all, so it's possible I screwed up somewhere (although the newer
 panic below does indicate I managed to build a kernel with the new versions at
 least because the Changing APIC ID line only appears in the newer files I think..)
 
 =-= Panic Information :
 
 Changing APIC ID for IO APIC #0 from 0 to 16 on chip
 panic: can't control IO APIC #0 ID, reg: 0x00000000
 mp_lock = 0000000a; cpuid = 0; lapic.id = 01000000
 Uptime: 0s
 
 [ Machine freezed ]
 
 Anyway, I'm just about to cvsup down a completely clean source tree. Any ideas
 which I should go for - i.e. which might give me working SMP ? 
 
 So... which should I try: HEAD, RELENG_4 (again) or wherever RELENG_3 is now ?
 
 [I'm open to all ideas on this one, as I find using one CPU on this box for
 what I need to do a little too slow for most things.]
 
 Tia,
 
 	Steve
 
 
State-Changed-From-To: analyzed->closed 
State-Changed-By: dougb 
State-Changed-When: Mon Oct 30 23:18:47 PST 2000 
State-Changed-Why:  
Originator confirms that that this problem is solved in 
RELENG_4/5. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=20312 
>Unformatted:
