From tedm@portsoft.com  Wed May 13 17:04:38 1998
Received: from mail.portsoft.com (mail.portsoft.com [208.207.3.226])
          by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id RAA20644
          for <FreeBSD-gnats-submit@freebsd.org>; Wed, 13 May 1998 17:04:12 -0700 (PDT)
          (envelope-from tedm@portsoft.com)
Received: from test.portsoft.com ([192.168.1.33])
	by mail.portsoft.com (8.8.5/8.8.5) with ESMTP id RAA08832;
	Wed, 13 May 1998 17:03:41 -0700 (PDT)
Received: (from tedm@localhost)
	by test.portsoft.com (8.8.8/8.8.5) id RAA00450;
	Wed, 13 May 1998 17:03:07 -0700 (PDT)
Message-Id: <199805140003.RAA00450@test.portsoft.com>
Date: Wed, 13 May 1998 17:03:07 -0700 (PDT)
From: Ted Mittelstaedt the Almighty System Admin <tedm@portsoft.com>
Reply-To: tedm@portsoft.com
To: FreeBSD-gnats-submit@freebsd.org, tedm@portsoft.com
Subject: Tyan Tomcat III locks SMP-GENERIC kernel on boot
X-Send-Pr-Version: 3.2

>Number:         6628
>Category:       kern
>Synopsis:       Tyan Tomcat III locks SMP-GENERIC kernel, 3.0-980426-SNAP
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed May 13 17:10:03 PDT 1998
>Closed-Date:    Sat May 16 13:36:52 PDT 1998
>Last-Modified:  Sat May 16 13:37:42 PDT 1998
>Originator:     Ted Mittelstaedt
>Release:        FreeBSD 3.0-980426-SNAP i386
>Organization:
Portland Software Inc.
>Environment:

Installed FreeBSD 3.0 snapshot on a Tyan Tomcat III system, 64MB of ram,
1 Adaptec 2940 PCI Ultra SCSI card, 1 Adaptec 2940 PCI Ultra Wide SCSI
card with a 8GB Seagate Wide SCSI disk on it, motherboard BIOS version 4.01.
Output of dmesg and mptable follows:

Copyright (c) 1992-1998 FreeBSD Inc.
Copyright (c) 1982, 1986, 1989, 1991, 1993
	The Regents of the University of California. All rights reserved.
FreeBSD 3.0-980426-SNAP #0: Sun Apr 26 10:14:16 GMT 1998
    root@make.ican.net:/usr/src/sys/compile/GENERIC
Timecounter "i8254"  frequency 1193182 Hz  cost 2537 ns
CPU: Pentium (199.43-MHz 586-class CPU)
  Origin = "GenuineIntel"  Id = 0x52c  Stepping=12
  Features=0x3bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,APIC>
real memory  = 67108864 (65536K bytes)
avail memory = 62423040 (60960K bytes)
Probing for devices on PCI bus 0:
chip0: <Intel 82439> rev 0x03 on pci0.0.0
chip1: <Intel 82371SB PCI to ISA bridge> rev 0x01 on pci0.7.0
ide_pci0: <Intel PIIX3 Bus-master IDE controller> rev 0x00 on pci0.7.1
chip2: <Intel 82371SB USB host controller> rev 0x01 int d irq 15 on pci0.7.2
fxp0: <Intel EtherExpress Pro 10/100B Ethernet> rev 0x02 int a irq 15 on pci0.17.0
fxp0: Ethernet address 00:a0:c9:5a:7c:c1
vga0: <Cirrus Logic GD5446 SVGA controller> rev 0x00 int a irq 12 on pci0.18.0
ahc0: <Adaptec 2940 Ultra SCSI host adapter> rev 0x00 int a irq 9 on pci0.19.0
ahc0: aic7880 Wide Channel, SCSI Id=7, 16 SCBs
ahc0: waiting for scsi devices to settle
scbus0 at ahc0 bus 0
sd0 at scbus0 target 0 lun 0
sd0: <SEAGATE ST19171W 0023> type 0 fixed SCSI 2
sd0: Direct-Access 8683MB (17783112 512 byte sectors)
ahc1: <Adaptec 2940 Ultra SCSI host adapter> rev 0x00 int a irq 11 on pci0.20.0
ahc1: aic7880 Single Channel, SCSI Id=7, 16 SCBs
ahc1: waiting for scsi devices to settle
scbus1 at ahc1 bus 0
cd0 at scbus1 target 3 lun 0
cd0: <NEC CD-ROM DRIVE:462 1.16> type 5 removable SCSI 2
cd0: CD-ROM can't get the size
st0 at scbus1 target 6 lun 0
st0: <ARCHIVE Python 00095-001 5.45> type 1 removable SCSI 2
st0: Sequential-Access density code 0x13,  drive empty
Probing for devices on the ISA bus:
sc0 at 0x60-0x6f irq 1 on motherboard
sc0: VGA color <16 virtual consoles, flags=0x0>
sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa
sio0: type 16550A
sio1 at 0x2f8-0x2ff irq 3 on isa
sio1: type 16550A
lpt0 at 0x378-0x37f irq 7 on isa
lpt0: Interrupt-driven port
lp0: TCP/IP capable interface
lpt1 at 0x378-0x37f on isa
lpt1 not probed due to I/O address conflict with lpt0 at 0x378
mse0 not found at 0x23c
psm0 not found at 0x60
fdc0 at 0x3f0-0x3f7 irq 6 drq 2 on isa
fdc0: FIFO enabled, 8 bytes threshold
fd0: 1.44MB 3.5in
wdc0 not found at 0x1f0
wdc1 not found at 0x170
npx0 on motherboard
npx0: INT 16 interface
Intel Pentium F00F detected, installing workaround
changing root device to sd0s1a
WARNING: / was not properly dismounted.



===============================================================================

MPTable, version 2.0.12

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

  location:			BIOS
  physical address:		0x000f5a60
  signature:			'_MP_'
  length:			16 bytes
  version:			1.4
  checksum:			0xc3
  mode:				Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

  physical address:		0x000f5a74
  signature:			'PCMP'
  base table length:		396
  version:			1.1
  checksum:			0x3a
  OEM ID:			'OEM00000'
  Product ID:			'PROD00000000'
  OEM table pointer:		0x00000000
  OEM table size:		0
  entry count:			28
  local APIC address:		0xfee00000
  extended table length:	0
  extended table checksum:	0

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
Processors:	APIC ID	Version	State		Family	Model	Step	Flags
		 0	 0x11	 BSP, usable	 5	 2	 1	 0x07bf
		 1	 0x11	 AP, usable	 5	 2	 1	 0x07bf
--
Bus:		Bus ID	Type
		 0	 PCI   
		 1	 ISA   
--
I/O APICs:	APIC ID	Version	State		Address
		 2	 0x11	 usable		 0xfec00000
--
I/O Ints:	Type	Polarity    Trigger	Bus ID	 IRQ	APIC ID	INT#
		ExtINT	 conforms    conforms	     1	   0	      2	   0
		INT	 conforms    conforms	     1	   1	      2	   1
		INT	 conforms    conforms	     1	   0	      2	   2
		INT	 conforms    conforms	     1	   3	      2	   3
		INT	 conforms    conforms	     1	   4	      2	   4
		INT	 conforms    conforms	     1	   5	      2	   5
		INT	 conforms    conforms	     1	   6	      2	   6
		INT	 conforms    conforms	     1	   7	      2	   7
		INT	 conforms    conforms	     1	   8	      2	   8
		INT	 conforms    conforms	     1	   9	      2	   9
		INT	 conforms    conforms	     1	  10	      2	  10
		INT	 conforms    conforms	     1	  11	      2	  11
		INT	 conforms    conforms	     1	  12	      2	  12
		INT	 conforms    conforms	     1	  13	      2	  13
		INT	 conforms    conforms	     1	  14	      2	  14
		INT	 conforms    conforms	     1	  15	      2	  15
		INT	active-lo       level	     0	20:A	      2	  16
		INT	active-lo       level	     0	19:A	      2	  17
		INT	active-lo       level	     0	18:A	      2	  18
		INT	active-lo       level	     0	17:A	      2	  19
		SMI	 conforms    conforms	     1	   0	      2	  23
--
Local Ints:	Type	Polarity    Trigger	Bus ID	 IRQ	APIC ID	INT#
		ExtINT	 conforms    conforms	     0	 0:A	    255	   0
		NMI	 conforms    conforms	     0	 0:A	    255	   1

-------------------------------------------------------------------------------

# SMP kernel config file options:


# Required:
options		SMP			# Symmetric MultiProcessor Kernel
options		APIC_IO			# Symmetric (APIC) I/O

# Useful:
#options		SMP_AUTOSTART		# start the additional CPUs during boot

# Optional (built-in defaults will work in most cases):
#options		NCPU=2			# number of CPUs
#options		NBUS=2			# number of busses
#options		NAPIC=1			# number of IO APICs
#options		NINTR=24		# number of INTs

# Rogue hardware:
#
#  Tyan Tomcat II:
#options		SMP_TIMER_NC		# 
#
#  SuperMicro P6DNE:
#options		SMP_TIMER_NC		# 

===============================================================================


# SMP kernel config file options:


# Required:
options		SMP			# Symmetric MultiProcessor Kernel
options		APIC_IO			# Symmetric (APIC) I/O

# Useful:
#options		SMP_AUTOSTART		# start the additional CPUs during boot

# Optional (built-in defaults will work in most cases):
#options		NCPU=2			# number of CPUs
#options		NBUS=2			# number of busses
#options		NAPIC=1			# number of IO APICs
#options		NINTR=24		# number of INTs

# Rogue hardware:
#
#  Tyan Tomcat II:
#options		SMP_TIMER_NC		# 
#
#  SuperMicro P6DNE:
#options		SMP_TIMER_NC		# 

===============================================================================

>Description:


After compiling the SMP kernel and installing it, a reboot produces a list of
all the devices, then as soon as the second CPU is started the system locks up.
I cold-booted several times, once it actually started running /etc/rc but 
locked about a half-second into it.

Rebooting the system under a uniprocessor kernel that I compiled works fine
without any detected problems. 

>How-To-Repeat:


Compile the SMP kernel and restart, the kernel configuration file I used is
as follows:


#
# SMP-GENERIC -- Smp machine with WD/AHx/NCR/BTx family disks
#
# For more information read the handbook part System Administration -> 
# Configuring the FreeBSD Kernel -> The Configuration File. 
# The handbook is available in /usr/share/doc/handbook or online as
# latest version from the FreeBSD World Wide Web server 
# <URL:http://www.FreeBSD.ORG/>
#
# An exhaustive list of options and more detailed explanations of the 
# device lines is present in the ./LINT configuration file. If you are 
# in doubt as to the purpose or necessity of a line, check first in LINT.
#
#	$Id: SMP-GENERIC,v 1.11 1998/04/22 18:18:14 mjacob Exp $

machine		"i386"
# SMP does NOT support 386/486 CPUs.
#cpu		"I386_CPU"
#cpu		"I486_CPU"

cpu		"I586_CPU"
# cpu		"I686_CPU"
ident		TESTME
maxusers	30

# Create a SMP capable kernel (mandatory options):
options		SMP			# Symmetric MultiProcessor Kernel
options		APIC_IO			# Symmetric (APIC) I/O

# Optional, these are the defaults:
#options		NCPU=2			# number of CPUs
#options		NBUS=4			# number of busses
#options		NAPIC=1			# number of IO APICs
#options		NINTR=24		# number of INTs

# Lets always enable the kernel debugger for SMP.
options		DDB

# SMP shouldn't need x87 emulation, disable by default.
#options		MATH_EMULATE		#Support for x87 emulation

options		INET			#InterNETworking
options		FFS			#Berkeley Fast Filesystem
options		NFS			#Network Filesystem
options		MSDOSFS			#MSDOS Filesystem
options		"CD9660"		#ISO 9660 Filesystem
options		PROCFS			#Process filesystem
options		"COMPAT_43"		#Compatible with BSD 4.3 [KEEP THIS!]
options		SCSI_DELAY=15		#Be pessimistic about Joe SCSI device
options		BOUNCE_BUFFERS		#include support for DMA bounce buffers
options		UCONSOLE		#Allow users to grab the console
options		FAILSAFE		#Be conservative
options		USERCONFIG		#boot -c editor
options		VISUAL_USERCONFIG	#visual boot -c editor

config		kernel	root on sd0

controller	isa0
# controller	eisa0
controller	pci0

controller	fdc0	at isa? port "IO_FD1" bio irq 6 drq 2 vector fdintr
disk		fd0	at fdc0 drive 0
disk		fd1	at fdc0 drive 1
# tape		ft0	at fdc0 drive 2

# options		"CMD640"	# work around CMD640 chip deficiency
# controller	wdc0	at isa? port "IO_WD1" bio irq 14 vector wdintr
# disk		wd0	at wdc0 drive 0
# disk		wd1	at wdc0 drive 1

# controller	wdc1	at isa? port "IO_WD2" bio irq 15 vector wdintr
# disk		wd2	at wdc1 drive 0
# disk		wd3	at wdc1 drive 1

# options		ATAPI		#Enable ATAPI support for IDE bus
# options		ATAPI_STATIC	#Don't do it as an LKM
# device		wcd0	#IDE CD-ROM

# A single entry for any of these controllers (ncr, ahb, ahc, amd) is
# sufficient for any number of installed devices.
# controller	ncr0
# controller	amd0
# controller	ahb0
controller	ahc0
# controller	isp0

# controller	bt0	at isa? port "IO_BT0" bio irq ? vector bt_isa_intr
# controller	uha0	at isa? port "IO_UHA0" bio irq ? drq 5 vector uhaintr
# controller	aha0	at isa? port "IO_AHA0" bio irq ? drq 5 vector ahaintr
# controller	aic0	at isa? port 0x340 bio irq 11 vector aicintr
# controller	nca0	at isa? port 0x1f88 bio irq 10 vector ncaintr
# controller	nca1	at isa? port 0x350 bio irq 5 vector ncaintr
# controller	sea0	at isa? bio irq 5 iomem 0xc8000 iosiz 0x2000 vector seaintr

controller	scbus0

device		sd0

# device		od0	#See LINT for possible `od' options.

device		st0

device		cd0	#Only need one of these, the code dynamically grows

# device		wt0	at isa? port 0x300 bio irq 5 drq 1 vector wtintr
# device		mcd0	at isa? port 0x300 bio irq 10 vector mcdintr

# controller	matcd0	at isa? port 0x230 bio

# device		scd0	at isa? port 0x230 bio

# syscons is the default console driver, resembling an SCO console
device		sc0	at isa? port "IO_KBD" tty irq 1 vector scintr
# Enable this and PCVT_FREEBSD for pcvt vt220 compatible console driver
#device		vt0	at isa? port "IO_KBD" tty irq 1 vector pcrint
#options		PCVT_FREEBSD=210	# pcvt running on FreeBSD >= 2.0.5
options		XSERVER			# include code for XFree86
#options		FAT_CURSOR		# start with block cursor
# If you have a ThinkPAD, uncomment this along with the rest of the PCVT lines
#options		PCVT_SCANSET=2		# IBM keyboards are non-std

device		npx0	at isa? port "IO_NPX" irq 13 vector npxintr

#
# Laptop support (see LINT for more options)
#
# device		apm0    at isa?	disable	flags 0x31 # Advanced Power Management

# PCCARD (PCMCIA) support
#controller	card0
#device		pcic0	at card?
#device		pcic1	at card?

device		sio0	at isa? port "IO_COM1" tty irq 4 vector siointr
device		sio1	at isa? port "IO_COM2" tty irq 3 vector siointr
device		sio2	at isa? disable port "IO_COM3" tty irq 5 vector siointr
device		sio3	at isa? disable port "IO_COM4" tty irq 9 vector siointr

device		lpt0	at isa? port? tty irq 7 vector lptintr
# device		lpt1	at isa? port? tty
# device		mse0	at isa? port 0x23c tty irq 5 vector mseintr

# device		psm0	at isa? disable port "IO_KBD" conflicts tty irq 12 vector psmintr

# Order is important here due to intrusive probes, do *not* alphabetize
# this list of network interfaces until the probes have been fixed.
# Right now it appears that the ie0 must be probed before ep0. See
# revision 1.20 of this file.
# device de0
device fxp0
# device vx0

# device ed0 at isa? port 0x280 net irq  5 iomem 0xd8000 vector edintr
# device ed1 at isa? port 0x300 net irq  5 iomem 0xd8000 vector edintr
# device ie0 at isa? port 0x300 net irq 10 iomem 0xd0000 vector ieintr
# device ie1 at isa? port 0x360 net irq  7 iomem 0xd0000 vector ieintr
# device ep0 at isa? port 0x300 net irq 10 vector epintr
# device ex0 at isa? port? net irq? vector exintr
# device fe0 at isa? port 0x300 net irq ? vector feintr
# device le0 at isa? port 0x300 net irq 5 iomem 0xd0000 vector le_intr
# device lnc0 at isa? port 0x280 net irq 10 drq 0 vector lncintr
# device ze0 at isa? port 0x300 net irq 5 iomem 0xd8000 vector zeintr
# device zp0 at isa? port 0x300 net irq 10 iomem 0xd8000 vector zpintr

pseudo-device	loop
pseudo-device	ether
pseudo-device	sl	1
# iijppp uses tun instead of ppp device
#pseudo-device	ppp	1
pseudo-device	tun	1
pseudo-device	pty	16
pseudo-device	gzip		# Exec gzipped a.out's

# KTRACE enables the system-call tracing facility ktrace(2).
# This adds 4 KB bloat to your kernel, and slightly increases
# the costs of each syscall.
options		KTRACE		#kernel tracing


>Fix:

Maybe bring back the option "SMP_TIMER_NC" to fix it?  It looks like an
earlier SMP kernel did work on this hardware until the APIC_IO became
required.

Unfortunately, I only had about 4 hours to test on this machine - if 3.0
had run SMP we would have used this as a server, but since it won't the
system is consigned to use as an NT system and won't be generally available
in the future.  Oh well.
>Release-Note:
>Audit-Trail:

From: Garrett Wollman <wollman@khavrinen.lcs.mit.edu>
To: tedm@portsoft.com
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: kern/6628: Tyan Tomcat III locks SMP-GENERIC kernel on boot
Date: Wed, 13 May 1998 21:07:30 -0400 (EDT)

 <<On Wed, 13 May 1998 17:03:07 -0700 (PDT), Ted Mittelstaedt the Almighty System Admin <tedm@portsoft.com> said:
 
 > After compiling the SMP kernel and installing it, a reboot produces a list of
 > all the devices, then as soon as the second CPU is started the system locks up.
 
 I have seen these symptoms on my Intel Buckeyes (BB440FX).  I found
 that there was a setting in the BIOS for ``route interrupus via I/O
 APIC'' which needed to be set for MP to work.
 
 -GAWollman
 
 --
 Garrett A. Wollman   | O Siem / We are all family / O Siem / We're all the same
 wollman@lcs.mit.edu  | O Siem / The fires of freedom 
 Opinions not those of| Dance in the burning flame
 MIT, LCS, CRS, or NSA|                     - Susan Aglukark and Chad Irschick

From: Ted Mittelstaedt <tedm@portsoft.com>
To: "tedm@portsoft.com" <tedm@portsoft.com>,
        "'Garrett Wollman'"
	 <wollman@khavrinen.lcs.mit.edu>
Cc: "FreeBSD-gnats-submit@FreeBSD.ORG" <FreeBSD-gnats-submit@FreeBSD.ORG>,
        "'FreeBSD SMP mailing list'" <freebsd-smp@FreeBSD.ORG>,
        "'FreeBSD FAQ Maintainer'" <pds@FreeBSD.ORG>
Subject: RE: kern/6628: Tyan Tomcat III locks SMP-GENERIC kernel on boot
Date: Thu, 14 May 1998 09:46:44 -0700

 Oh Geeze!  I should have played around more with BIOS settings!
 
 There is NO setting for "route interrupus via I/O APIC" in the Tyan Tomcat's BIOS.  (Is that anything like Coitus Interruptus?) :)
 
 However, there IS a BIOS setting for the Tyan board under the "Chipset Features Setup" menu item labeled (innocuously enough):
 
 "Chipset Special Features"  Enabled/Disabled.
 
 Setting this to "Enabled" seems to fix the problem!  I hope someone puts this in the hardware FAQ for 3.0!
 
 This was on BIOS version 4.01 for the Tomcat IIID - I had to flash the motherboard with current BIOS code.  One hiccup on the flashing process - when I flashed it I got to 1F00 and the flash program locked up, but waiting 5 minutes and cold-booting the b oard seemed fine.  I even flashed it twice and it locked up both times in the same location.  Pretty hairy when the flash program is sitting there blinking "Do Not Cold Boot While This Program Is Running" messages.
 
 The one noticeable difference in the mptable output after flashing the BIOS:  version 4.01 of the BIOS changes the version number in the MP Floating Pointer Structure from version 1.1 to 1.4
 
 Ted
 
 ----------
 From:  Garrett Wollman[SMTP:wollman@khavrinen.lcs.mit.edu]
 Sent:  Wednesday, May 13, 1998 2:08 PM
 To:  tedm@portsoft.com
 Cc:  FreeBSD-gnats-submit@FreeBSD.ORG
 Subject:  kern/6628: Tyan Tomcat III locks SMP-GENERIC kernel on boot
 
 <<On Wed, 13 May 1998 17:03:07 -0700 (PDT), Ted Mittelstaedt the Almighty System Admin <tedm@portsoft.com> said:
 
 > After compiling the SMP kernel and installing it, a reboot produces a list of
 > all the devices, then as soon as the second CPU is started the system locks up.
 
 I have seen these symptoms on my Intel Buckeyes (BB440FX).  I found
 that there was a setting in the BIOS for ``route interrupus via I/O
 APIC'' which needed to be set for MP to work.
 
 -GAWollman
 
 --
 Garrett A. Wollman   | O Siem / We are all family / O Siem / We're all the same
 wollman@lcs.mit.edu  | O Siem / The fires of freedom 
 Opinions not those of| Dance in the burning flame
 MIT, LCS, CRS, or NSA|                     - Susan Aglukark and Chad Irschick
 
 
State-Changed-From-To: open->closed 
State-Changed-By: des 
State-Changed-When: Sat May 16 13:36:52 PDT 1998 
State-Changed-Why:  
Originator tracked down problem to a BIOS setting. 
>Unformatted:
