From vanmaren@vanmaren.aros.net  Tue Dec  1 20:01:42 1998
Received: from vanmaren.aros.net (vanmaren.aros.net [208.219.24.33])
          by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id UAA01180
          for <FreeBSD-gnats-submit@freebsd.org>; Tue, 1 Dec 1998 20:01:40 -0800 (PST)
          (envelope-from vanmaren@vanmaren.aros.net)
Received: (from vanmaren@localhost)
	by vanmaren.aros.net (8.9.1/8.9.1) id VAA01553;
	Tue, 1 Dec 1998 21:01:22 -0700 (MST)
	(envelope-from vanmaren)
Message-Id: <199812020401.VAA01553@vanmaren.aros.net>
Date: Tue, 1 Dec 1998 21:01:22 -0700 (MST)
From: vanmaren@cs.utah.edu
Reply-To: vanmaren@cs.utah.edu
To: FreeBSD-gnats-submit@freebsd.org
Subject: 450NX-based computers only probe the first PCI bus
X-Send-Pr-Version: 3.2

>Number:         8928
>Category:       kern
>Synopsis:       450NX-based computers only probe the first PCI bus
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Dec  1 20:10:01 PST 1998
>Closed-Date:    Fri Dec 18 19:01:16 PST 1998
>Last-Modified:  Fri Dec 18 19:01:36 PST 1998
>Originator:     Kevin Van Maren
>Release:        FreeBSD 3.0-CURRENT i386
>Organization:
University of Utah
>Environment:

This occurs in both 3.0-CURRENT and 2.2-STABLE with the 450NX chipset.

>Description:

MIOC: Memory and I/O controller.  It connects the CPU bus to the
memory subsystem and the PCI controller.  PXB: PCI Expander bridge,
connects the MIOC to either two 32-bit PCI busses or a 64-bit bus.
The MIOC can control one or two PXBs.

Any devices not plugged into the first PCI bus simply won't be found.
Most current 450NX-based computers are quad-capable with either
two 32-bit PCI busses (using one PXB), or with two 32-bit busses
and a 64-bit PCI bus (using two PXB chips).  In either case, only
devices on the first 32-bit PCI bus will be probed, as FreeBSD
doesn't know about the other PCI busses.

The problem stems from how the PCI busses are provided by the
chipset.  The processor bus interface attaches to the MIOC, which
in turn can connect to either one or two PXB PCI bus controller
chips (with either one or two PCI bus interfaces each).  Instead
of there being one PCI bus with the other busses "bridged" from
that one, they are connected in parallel to the MIOC.  Fancy
circuitry makes it all "work".  However, since the PXB chips are
probed as regular devices, and not as PCI-PCI bridges, the
PCI bus count isn't incremented.  Additionally, it is actually
possible for the BIOS to skip PCI busses when programming the 
bus IDs for the PXBs in the MIOC (to allow for hot-plug devices
with a PCI-PCI bridge).  Also, since the configuration for the
PXB is actually controlled by configuration registers in the
MIOC, the MIOC must be asked how many PCI busses there are.  It
would be much cleaner to set up the subordinatebus for each PXB
as it is probed, but the info is in the MIOC, not the PXB.

>How-To-Repeat:

Buy a quad Xeon.
Install FreeBSD.
Try to use a device not on the first bus.
[PCI busses bridged from the first bus will, of course, work,
but not the other ones.]

>Fix:

The hardware knows how to route requests to higher bus numbers properly;
the kernel just doesn't know to look for something there.

The fix is to read the BIOS-programmed configuration from the
MIOC configuration registers.  The highest bus number must belong
to the last bus on the last PXB, and there are 4 choices.  The
code just searches backwards until it finds one that has been configured.

Here are the patches.  Note that I haven't actually tested the ones
for 2.2-STABLE, but I think they will work...they do compile.
[Unfortunalty, I don't have good access to a 450NX box now.]

Patches for 2.2-STABLE: (in src/sys/pci)


Index: pcisupport.c
===================================================================
RCS file: /usr/lsrc/FreeBSD/CVS/src/sys/pci/pcisupport.c,v
retrieving revision 1.40.2.8
diff -u -r1.40.2.8 pcisupport.c
--- pcisupport.c	1998-06-17 09:29:09-06	1.40.2.8
+++ pcisupport.c	1998-12-01 20:58:41-07
@@ -172,6 +172,10 @@
 		return ("Intel 82450KX (Orion) PCI memory controller");
 	case 0x84c58086:
 		return ("Intel 82454GX (Orion) host to PCI bridge");
+	case 0x84ca8086:
+		return ("Intel 82451NX Memory and I/O Controller");
+	case 0x84cb8086:
+		return ("Intel 82454NX PCI Expander Bridge");
 	case 0x00051166:
 		return ("Ross (?) host to PCI bridge");
 	case 0x00221014:
@@ -726,6 +730,56 @@
 }
 
 static void
+config_450nx(pcici_t tag)
+{
+	unsigned long devmap;
+	int i;
+
+	/* Read the MIOC devmap to determine which PCI expanders are present */
+	devmap = (pci_conf_read(tag, 0xd4) >> 16) & 0x3c;
+
+	if (!devmap) {
+		printf("Error: 450NX MIOC: No PCI Expander Bridge present.\n");
+		return;
+	}
+
+#if 0
+	/*
+	 * This hack is in the spirit of the config_orion() routine.
+	 * It *would* work, except that some 450NX-based servers are
+	 * set up to SKIP PCI busses, presumably to support hot-plug
+	 * PCI cards that contain a single PCI-PCI bridge chip.
+	 */
+
+	/* Now pciroots just needs to get set to the number of bits set... */
+
+	pciroots = 0;
+	for (i = 2; i < 6; i++)
+		if (devmap & (1 << i))
+			pciroots++;
+#else
+	/*
+	 * Since the buses are configured in order, we just have to
+	 * find the highest bus, and use those numbers.
+	 * This is `wrong', but there is nothing else I can really do...
+	 */
+	if (devmap & 0x20) {			/* B1, 0xd5 */
+		pciroots = (pci_conf_read(tag, 0xd4) >> 8) & 0xff;
+	} else if (devmap & 0x10) {		/* A1, 0xd4 */
+		pciroots = pci_conf_read(tag, 0xd4) & 0xff;
+	} else if (devmap & 0x8) {		/* B0, 0xd2 */
+		pciroots = (pci_conf_read(tag, 0xd0) >> 16) & 0xff;
+	} else /* if (devmap & 0x4) */ {	/* A0, 0xd1 */
+		pciroots = (pci_conf_read(tag, 0xd0) >> 8) & 0xff;
+	}
+#endif
+
+	if (bootverbose)
+		printf("config_450nx: %d PCI busses found\n", pciroots);
+}
+
+
+static void
 chipset_attach (pcici_t config_id, int unit)
 {
 	switch (pci_conf_read (config_id, PCI_ID_REG)) {
@@ -738,6 +792,9 @@
 		break;
 	case 0x00051166: /* Ross ??? */
 		config_Ross (config_id);
+		break;
+	case 0x84ca8086: /* Intel 450NX */
+		config_450nx (config_id);
 		break;
 	}
 #ifndef PCI_QUIET




Patches for 3.0-CURRENT: in src/sys/pci

Index: pcisupport.c
===================================================================
RCS file: /usr/lsrc/FreeBSD/CVS/src/sys/pci/pcisupport.c,v
retrieving revision 1.74
diff -u -r1.74 pcisupport.c
--- pcisupport.c	1998-11-26 14:57:52-07	1.74
+++ pcisupport.c	1998-12-01 20:28:31-07
@@ -118,6 +122,7 @@
  * XXX Both fixbushigh_orion() and fixbushigh_i1225() are bogus in that way,
  * that they store the highest bus number to scan in this device's config 
  * data, though it is about PCI buses attached to the CPU independently!
+ * The same goes for fixbushigh_450nx.
  */
 
 static void
@@ -137,6 +142,57 @@
 		tag->secondarybus = tag->subordinatebus = sublementarybus +1;
 }
 
+
+/*
+ * This reads the PCI config space for the 82451NX MIOC in the 450NX
+ * chipset to determine the PCI bus configuration.
+ *
+ * Assuming the BIOS has set up the MIOC properly, this will correctly
+ * report the number of PCI busses in the system.
+ *
+ * A small problem is that the Host to PCI bridge control is in the MIOC,
+ * while the host-pci bridges are separate PCI devices.  So it really
+ * isn't easily possible to set up the subordinatebus mappings as the
+ * 82454NX PCI expander bridges are probed, although that makes the
+ * most sense.
+ */
+static void
+fixbushigh_450nx(pcici_t tag)
+{
+	int subordinatebus;
+	unsigned long devmap;
+
+	/*
+	 * Read the DEVMAP field, so we know which fields to check.
+	 * If the Host-PCI bridge isn't marked as present by the BIOS,
+	 * we have to assume it doesn't exist.
+	 * If this doesn't find all the PCI busses, complain to the
+	 * BIOS vendor.  There is nothing more we can do.
+	 */
+	devmap = pci_cfgread(tag, 0xd6, 2) & 0x3c;
+	if (!devmap)
+		panic("450NX MIOC: No host to PCI bridges marked present.\n");
+	/*
+	 * Since the buses are configured in order, we just have to
+	 * find the highest bus, and use those numbers.
+	 */
+	if (devmap & 0x20) {			/* B1 */
+		subordinatebus = pci_cfgread(tag, 0xd5, 1);
+	} else if (devmap & 0x10) {		/* A1 */
+		subordinatebus = pci_cfgread(tag, 0xd4, 1);
+	} else if (devmap & 0x8) {		/* B0 */
+		subordinatebus = pci_cfgread(tag, 0xd2, 1);
+	} else /* if (devmap & 0x4) */ {	/* A0 */
+		subordinatebus = pci_cfgread(tag, 0xd1, 1);
+	}
+	if (bootverbose)
+		printf("fixbushigh_450nx: subordinatebus is %d\n",
+			subordinatebus);
+
+	tag->secondarybus = tag->subordinatebus = subordinatebus;
+}
+
+
 static void
 fixwsc_natoma(pcici_t tag)
 {
@@ -231,6 +287,7 @@
 	case 0x84c58086:
 		return ("Intel 82453KX/GX (Orion) PCI memory controller");
 	case 0x84ca8086:
+		fixbushigh_450nx(tag);
 		return ("Intel 82451NX Memory and I/O Controller");
 	case 0x84cb8086:
 		return ("Intel 82454NX PCI Expander Bridge");


>Release-Note:
>Audit-Trail:

From: Kevin Van Maren <vanmaren@fast.cs.utah.edu>
To: FreeBSD-gnats-submit@FreeBSD.ORG, freebsd-bugs@FreeBSD.ORG
Cc:  Subject: Re: kern/8928: 450NX-based computers only probe the first PCI bus
Date: Fri, 4 Dec 1998 10:21:00 -0700 (MST)

 Just a couple of things:  The 2.2-STABLE patches have been tested
 (on an IBM 7000 M10), and they do work.
 
 The IBM allocates a PCI bus for every expansion slot in
 the system, in addition to the 3 busses (0-16 are "used").
 
 The Dell PowerEdge 6300 also sets up the MIOC registers "wrong".
 However, the Dell just programs the "highest used bus number" to
 255 on the last PXB.  So the PCI code probes all 256 possible
 busses...
 
 So if (subordinatebus == 255), it should probably be set to (NBUS-2)
 for SMP, and presumably something "reasonable" otherwise.  I can't
 tell how many "primary" PCI busses there are in the computer, but
 (because of the IBM 7000) that doesn't help much.  I can read the
 lowest PCI bus number per PXB, but there is no easy way to infer
 the highest buds number.
 
 Perhaps we should just read the device id for every device on every
 PCI bus and program the subordinatebus to the highest bus number
 on which we found something.  Yuck.
 
 Why is the BIOS on every 450NX-based Xeon screwed up?  The intel
 bios didn't even bother to config the bus if it thought it was empty.
 
 Kevin

From: David Malone <dwmalone@maths.tcd.ie>
To: freebsd-gnats-submit@freebsd.org
Cc:  Subject: Re: kern/8928 450NX-based computers only probe the first PCI bus
Date: Thu, 17 Dec 1998 17:20:14 +0000

 I've been using the following patch on our Dell 6300 machine and it
 seems to be working fine. If the number of busses is 255 I've just
 set the number of busses to be NBUS - 2 if NBUS is defined or 10
 otherwise. I had to include opt_smp.h to get NBUS...
 
 	David.
 
 
 *** pcisupport.c.orig	Wed Dec  2 22:29:31 1998
 --- pcisupport.c	Thu Dec  3 12:40:09 1998
 ***************
 *** 42,47 ****
 --- 42,48 ----
   */
   
   #include "opt_pci.h"
 + #include "opt_smp.h"
   
   #include <sys/param.h>
   #include <sys/systm.h>
 ***************
 *** 118,123 ****
 --- 119,125 ----
    * XXX Both fixbushigh_orion() and fixbushigh_i1225() are bogus in that way,
    * that they store the highest bus number to scan in this device's config 
    * data, though it is about PCI buses attached to the CPU independently!
 +  * The same goes for fixbushigh_450nx.
    */
   
   static void
 ***************
 *** 137,142 ****
 --- 139,204 ----
   		tag->secondarybus = tag->subordinatebus = sublementarybus +1;
   }
   
 + 
 + /*
 +  * This reads the PCI config space for the 82451NX MIOC in the 450NX
 +  * chipset to determine the PCI bus configuration.
 +  *
 +  * Assuming the BIOS has set up the MIOC properly, this will correctly
 +  * report the number of PCI busses in the system.
 +  *
 +  * A small problem is that the Host to PCI bridge control is in the MIOC,
 +  * while the host-pci bridges are separate PCI devices.  So it really
 +  * isn't easily possible to set up the subordinatebus mappings as the
 +  * 82454NX PCI expander bridges are probed, although that makes the
 +  * most sense.
 +  */
 + static void
 + fixbushigh_450nx(pcici_t tag)
 + {
 +        int subordinatebus;
 +        unsigned long devmap;
 + 
 +        /*
 +         * Read the DEVMAP field, so we know which fields to check.
 +         * If the Host-PCI bridge isn't marked as present by the BIOS,
 +         * we have to assume it doesn't exist.
 +         * If this doesn't find all the PCI busses, complain to the
 +         * BIOS vendor.  There is nothing more we can do.
 +         */
 +        devmap = pci_cfgread(tag, 0xd6, 2) & 0x3c;
 +        if (!devmap)
 +                panic("450NX MIOC: No host to PCI bridges marked present.\n");
 +        /*
 +         * Since the buses are configured in order, we just have to
 +         * find the highest bus, and use those numbers.
 +         */
 +        if (devmap & 0x20) {                        /* B1 */
 +                subordinatebus = pci_cfgread(tag, 0xd5, 1);
 +        } else if (devmap & 0x10) {         /* A1 */
 +                subordinatebus = pci_cfgread(tag, 0xd4, 1);
 +        } else if (devmap & 0x8) {          /* B0 */
 +                subordinatebus = pci_cfgread(tag, 0xd2, 1);
 +        } else /* if (devmap & 0x4) */ {    /* A0 */
 +                subordinatebus = pci_cfgread(tag, 0xd1, 1);
 +        }
 +        if (subordinatebus == 255) {
 +                printf("fixbushigh_450nx: bogus highest bus (%d) - reducing\n",
 + 			subordinatebus);
 + #ifdef NBUS
 +                subordinatebus = NBUS - 2;
 + #else
 +                subordinatebus = 10;
 + #endif
 +        }
 +        if (bootverbose)
 +                printf("fixbushigh_450nx: subordinatebus is %d\n",
 +                        subordinatebus);
 + 
 +        tag->secondarybus = tag->subordinatebus = subordinatebus;
 + }
 + 
 + 
   static void
   fixwsc_natoma(pcici_t tag)
   {
 ***************
 *** 231,236 ****
 --- 293,299 ----
   	case 0x84c58086:
   		return ("Intel 82453KX/GX (Orion) PCI memory controller");
   	case 0x84ca8086:
 +                 fixbushigh_450nx(tag);
   		return ("Intel 82451NX Memory and I/O Controller");
   	case 0x84cb8086:
   		return ("Intel 82454NX PCI Expander Bridge");
State-Changed-From-To: open->closed 
State-Changed-By: msmith 
State-Changed-When: Fri Dec 18 19:01:16 PST 1998 
State-Changed-Why:  
Changes proposed were tested and incorporated. 
>Unformatted:
