From ma@dt.e-technik.uni-dortmund.de  Sun Apr 14 07:12:23 2002
Return-Path: <ma@dt.e-technik.uni-dortmund.de>
Received: from krusty.e-technik.uni-dortmund.de (krusty.E-Technik.Uni-Dortmund.DE [129.217.163.1])
	by hub.freebsd.org (Postfix) with ESMTP
	id 43E0737B405; Sun, 14 Apr 2002 07:12:22 -0700 (PDT)
Received: from merlin.emma.line.org (localhost [127.0.0.1])
	by krusty.e-technik.uni-dortmund.de (Postfix) with ESMTP
	id 9C104A3831; Sun, 14 Apr 2002 16:12:20 +0200 (CEST)
Received: by merlin.emma.line.org (Postfix, from userid 500)
	id 2EAFE2C073; Sun, 14 Apr 2002 16:12:17 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
	by merlin.emma.line.org (Postfix) with ESMTP id CC5D3B763
	for <emma@localhost>; Sun, 14 Apr 2002 16:11:07 +0200 (CEST)
Received: from pop3.web.de
	by localhost with POP3 (fetchmail-5.9.11)
	for emma@localhost (single-drop); Sun, 14 Apr 2002 16:11:07 +0200 (CEST)
Received: from [217.81.248.165] (helo=merlin.emma.line.org)
	by mx05.web.de with esmtp (WEB.DE(Exim) 4.43 #48)
	id 16wkKk-0006k1-00
	for matthias.andree@web.de; Sun, 14 Apr 2002 15:46:22 +0200
Received: (from emma@localhost)
	by merlin.emma.line.org (8.11.6/8.11.6) id g3EDk2G00380;
	Sun, 14 Apr 2002 15:46:02 +0200 (CEST)
	(envelope-from emma)
Message-Id: <200204141346.g3EDk2G00380@merlin.emma.line.org>
Date: Sun, 14 Apr 2002 15:46:02 +0200 (CEST)
From: Matthias Andree <ma@dt.e-technik.uni-dortmund.de>
Reply-To: Matthias Andree <ma@dt.e-technik.uni-dortmund.de>
To: FreeBSD-gnats-submit@freebsd.org
Cc: sos@freebsd.org
Subject: kernel panic with hw.ata.tags=1 in ata-disk.c:710
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         37060
>Category:       kern
>Synopsis:       kernel panic with hw.ata.tags=1 in ata-disk.c:710
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    sos
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Apr 14 07:20:01 PDT 2002
>Closed-Date:    Mon Apr 28 11:30:13 PDT 2003
>Last-Modified:  Mon Apr 28 11:30:13 PDT 2003
>Originator:     Matthias Andree
>Release:        FreeBSD 4.5-STABLE i386
>Organization:
>Environment:
System: FreeBSD emma1.emma.line.org 4.5-STABLE FreeBSD 4.5-STABLE #1: Sun Apr 14 13:33:10 CEST 2002 root@emma1.emma.line.org:/usr/src/sys/compile/MA i386


	
>Description:
FreeBSD 4-STABLE, checked out around local midnight, 2002-04-14.

When I use 'set hw.ata.tags=1' in the FreeBSD loader, the kernel panics (or
jumps into the debugger).

Leaving hw.ata.tags alone (not setting it) is fine, the machine boots up and
works properly.

This may be related to the "tagged acting up" recently reported on
freebsd-stable@freebsd.org

The last boot -v message is: "Creating DISK ad2"

(copied with pen & paper):
Fault trap 12: page fault while in kernel mode.
Fault virtual address:0x1c
fault code: supervisor read, page not present
current process: 0 (swapper)
irq mask: bio

DDB:

Stopped at: ad_service+0x30    testb $0x8,0x1c(%eax)
(Looks as though %eax is NULL)

trace:
ad_service +0x30
ad_transfer +0x253
ata_start +0x98
adstrategy
ar_rw
ar_promise_read_conf
ata_raiddisk_attach
ad_attach
ata_boot_attach

The offending line is 710 (as per gdb "info *(ad_service+0x30)"):

   705	    /* do we have to check the other device on this channel ? */
   706	    if (adp->device->channel->flags & ATA_QUEUED && change) {
   707		int device = adp->device->unit;
   708	
   709		if (adp->device->unit == ATA_MASTER) {
   710		    if (adp->device->channel->devices & ATA_ATA_SLAVE &&
   711			((struct ad_softc *)
   712			 (adp->device->channel->
   713			  device[ATA_DEV(ATA_SLAVE)].driver))->flags&AD_F_TAG_ENABLED)
   714			device = ATA_SLAVE;
   715		}

ata-related dmesg:

atapci0: <VIA 82C686 ATA66 controller> port 0xffa0-0xffaf at device 7.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0

atacontrol list:
ATA channel 0:
    Master:  ad0 <Maxtor 4W060H4/AAH41310> ATA/ATAPI rev 6
    Slave:       no device present
ATA channel 1:
    Master:  ad2 <IBM-DTLA-307045/TX6OA6AA> ATA/ATAPI rev 5
    Slave:   ad3 <WDC AC420400D/J58OA30K> ATA/ATAPI rev 4
(all do UDMA66 -- yes I know DTLA are unreliable, two out of seven died here)

FWIW, probably unrelated: the machine is to boot from ad3, which was ad0
before I bought the Maxtor drive and added a Fujitsu UWSCSI drive.

The machine also has a Promise PDC-20265R chip (onboard), but it's switched
off (jumper or BIOS, I don't recall) and does not show up in dmesg.

I'm willing to help further, I can use gdb and have a machine that I can
attach via null modem cable, if need be. However, I'm not acquainted with
FreeBSD's kernel debugger (ddb), so any instructions as to the ddb use
should be verbose. Thanks in advance.

>How-To-Repeat:
on my system, pressing any key in the loader,
then entering:

set hw.ata.tags=1
boot

will cause the problem. Adding a line "hw.ata.tags=1" to /boot/loader.conf
also triggers the problem.

	
>Fix:
Workaround: comment out hw.ata.tags line in /boot/loader.conf. Comes at the
expense of tagged queueing, obviously.

Fix: will probably require changes to the ata driver that I cannot do.

	


>Release-Note:
>Audit-Trail:

From: matthias.andree@web.de
To: freebsd-gnats-submit@FreeBSD.org, ma@dt.e-technik.uni-dortmund.de
Cc: matthias.andree@uni-dortmund.de
Subject: Re: kern/37060: kernel panic with hw.ata.tags=1
Date: Mon, 20 May 2002 12:16:43 +0200 (CEST)

 Note that ATA_STATIC_ID is enabled in my kernel.
 Not sure if that matters, but it might.
 

From: Matthias Andree <ma@dt.e-technik.uni-dortmund.de>
To: freebsd-gnats-submit@FreeBSD.org, ma@dt.e-technik.uni-dortmund.de
Cc:  
Subject: Re: kern/37060: kernel panic with hw.ata.tags=1 in ata-disk.c:710
Date: Mon, 20 May 2002 17:42:15 +0200

 ATA_STATIC_ID does not have any impact. Trying to dig up precisely what
 pointer is NULL.

From: Matthias Andree <matthias.andree@stud.uni-dortmund.de>
To: Andrew Gallatin <gallatin@cs.duke.edu>
Cc: matthias.andree@web.de, freebsd-gnats-submit@freebsd.org,
	sos@freebsd.org
Subject: Re: kern/37060
Date: Mon, 20 May 2002 20:43:35 +0200

 On Mon, 20 May 2002, Andrew Gallatin wrote:
 
 > >It would be helpful to know which pointer was null.  There
 > >are many of them on line 710 of ata-disk.c
 
 Ok, it looks as though bad things happen when the non-existant primary
 slave is probed. I used boot -dg, set a breakpoint at ad_service and
 after successfully detecting the first drive, I got some info.
 
 The most important lines from below, consistent with the trap
 (ATA_DEV(ATA_SLAVE) == 1):
 
 (kgdb) print ((struct ad_softc *)(adp->device->channel->device[1].driver))
 $9 = (struct ad_softc *) 0x0
 
 So the problem happens probably at line #713 when dereferencing
 ->flags.
 
 
 Here's an excerpt from a typescript remote gdb session:
 
 Remote debugging using /dev/cuaa1
 ad_service (adp=0xc19ce400, change=1) at ../../dev/ata/ata-disk.c:706
 706	    if (adp->device->channel->flags & ATA_QUEUED && change) {
 (kgdb) print adp->device
 $1 = (struct ata_device *) 0xc190922c
 (kgdb) print *adp->device
 $2 = {channel = 0xc1909200, unit = 0, name = 0xc1949dc0 "ad1", 
   param = 0xc19d8c00, driver = 0xc19ce400, flags = 0, mode = 68, cmd = 0, 
   result = 0x0}
 (kgdb) print *adp->device->channel
 $3 = {dev = 0xc191cf00, unit = 1, r_io = 0xc191df80, r_altio = 0xc191df00, 
   r_bmio = 0xc191de80, r_irq = 0xc191dfc0, ih = 0xc101b000, 
   intr_func = 0xc0149a00 <ata_pci_intr>, chiptype = 91296006, alignment = 1, 
   flags = 0, device = {{channel = 0xc1909200, unit = 0, 
       name = 0xc1949dc0 "ad1", param = 0xc19d8c00, driver = 0xc19ce400, 
       flags = 0, mode = 68, cmd = 0, result = 0x0}, {channel = 0xc1909200, 
       unit = 16, name = 0x0, param = 0xc19d8e00, driver = 0x0, flags = 0, 
       mode = 0, cmd = 0, result = 0x0}}, devices = 3, status = 80 'P', 
   error = 0 '\000', active = 32, ata_queue = {tqh_first = 0x0, 
     tqh_last = 0xc1909280}, atapi_queue = {tqh_first = 0x0, 
     tqh_last = 0xc1909288}, running = 0xc19d7e00}
 (kgdb) print *adp->device->channel->devices
 $4 = 3
 (kgdb) print adp->device->channel->device
 $5 = {channel = 0xc1909200, unit = 0, name = 0xc1949dc0 "ad1", 
   param = 0xc19d8c00, driver = 0xc19ce400, flags = 0, mode = 68, cmd = 0, 
   result = 0x0}
 (kgdb) print adp->device->channel->device[1]
 $6 = {channel = 0xc1909200, unit = 16, name = 0x0, param = 0xc19d8e00, 
   driver = 0x0, flags = 0, mode = 0, cmd = 0, result = 0x0}
 (kgdb) print adp->device->channel->device[2]
 $7 = {channel = 0x3, unit = 80, name = 0x20 <Address 0x20 out of bounds>, 
   param = 0x0, driver = 0xc1909280, flags = 0, mode = -1047489912, 
   cmd = -1046643200, result = 0x0}
 (kgdb) print adp->device->channel->device[3]
 $8 = {channel = 0x0, unit = 0, name = 0x0, param = 0x0, driver = 0x0, 
   flags = 0, mode = 0, cmd = 0, result = 0x0}
 (kgdb) l
 701	
 702	int
 703	ad_service(struct ad_softc *adp, int change)
 704	{
 705	    /* do we have to check the other device on this channel ? */
 706	    if (adp->device->channel->flags & ATA_QUEUED && change) {
 707		int device = adp->device->unit;
 708	
 709		if (adp->device->unit == ATA_MASTER) {
 710		    if (adp->device->channel->devices & ATA_ATA_SLAVE &&
 (kgdb) l
 711			((struct ad_softc *)
 712			 (adp->device->channel->
 713			  device[ATA_DEV(ATA_SLAVE)].driver))->flags&AD_F_TAG_ENABLED)
 714			device = ATA_SLAVE;
 715		}
 716		else {
 717		    if (adp->device->channel->devices & ATA_ATA_MASTER &&
 718			((struct ad_softc *)
 719			 (adp->device->channel->
 720			  device[ATA_DEV(ATA_MASTER)].driver))->flags&AD_F_TAG_ENABLED)
 (kgdb) print ((struct ad_softc *)(adp->device->channel->device[1].driver))->flags
 Cannot access memory at address 0x1c.
 (kgdb) print ((struct ad_softc *)(adp->device->channel->device[1].driver))
 $9 = (struct ad_softc *) 0x0
 (kgdb) l
 721			device = ATA_MASTER;
 722		}
 723		if (device != adp->device->unit &&
 724		    ((struct ad_softc *)
 725		     (adp->device->channel->
 726		      device[ATA_DEV(device)].driver))->outstanding > 0) {
 727		    ATA_OUTB(adp->device->channel->r_io, ATA_DRIVE, ATA_D_IBM | device);
 728		    adp = adp->device->channel->device[ATA_DEV(device)].driver;
 729		    DELAY(1);
 730		}
 (kgdb) print adp->device->unit
 $10 = 0
 
 > Ack, this is a boot problem, so a crashdump is going to be hard.
 > Can you print out adp->device->channel->devices and
 > device[ATA_DEV(ATA_SLAVE)].drive  and
 > ((struct ad_softc *)(adp->device->channel->device[ATA_DEV(ATA_SLAVE)].driver))->flags
 > in ad_service, prior to the line which causes the panic?
 
 So the driver for that drive is NULL. Find the rest above.
 
 -- 
 Matthias Andree

From: Andrew Gallatin <gallatin@cs.duke.edu>
To: Matthias Andree <matthias.andree@stud.uni-dortmund.de>
Cc: matthias.andree@web.de, freebsd-gnats-submit@freebsd.org,
	sos@freebsd.org
Subject: Re: kern/37060
Date: Mon, 20 May 2002 15:53:15 -0400 (EDT)

 Matthias Andree writes:
  > On Mon, 20 May 2002, Andrew Gallatin wrote:
  > 
  > > >It would be helpful to know which pointer was null.  There
  > > >are many of them on line 710 of ata-disk.c
  > 
  > Ok, it looks as though bad things happen when the non-existant primary
  > slave is probed. I used boot -dg, set a breakpoint at ad_service and
  > after successfully detecting the first drive, I got some info.
  > 
  > The most important lines from below, consistent with the trap
  > (ATA_DEV(ATA_SLAVE) == 1):
  > 
  > (kgdb) print ((struct ad_softc *)(adp->device->channel->device[1].driver))
  > $9 = (struct ad_softc *) 0x0
  > 
  > So the problem happens probably at line #713 when dereferencing
  > ->flags.
 
 
 What if you protect those access with a check for a non-null driver?
 
 Also, what happens if you comment out the call to the
 ata_raiddisk_attach() in ad_attach() ?
 
 Drew
 
 

From: Matthias Andree <matthias.andree@web.de>
To: Andrew Gallatin <gallatin@cs.duke.edu>
Cc: Matthias Andree <matthias.andree@stud.uni-dortmund.de>,
	matthias.andree@web.de, freebsd-gnats-submit@freebsd.org,
	sos@freebsd.org
Subject: Re: kern/37060
Date: Tue, 21 May 2002 00:12:02 +0200

 On Mon, May 20, 2002 at 03:53:15PM -0400, Andrew Gallatin wrote:
 [ata-disk.c:710]
 > What if you protect those access with a check for a non-null driver?
 
 Didn't try yet.
 
 > Also, what happens if you comment out the call to the
 > ata_raiddisk_attach() in ad_attach() ?
 
 Good question: Skipping the if(ata_raiddisk_attach(adp)) call with "return" 
 in a remote gdb on all occasions lets the machine boot up properly.
 
 One observation of the failed runs:
 
 The first raiddisk_attach (Primary Master) has a NULL table, and 
 subsequently allocates one. The Primary Slave is skipped right away.
 
 When FreeBSD then tries to attach ad1 (ad2 with ATA_STATIC_ID,
 secondary master), it crashes with the known symptoms.

From: Matthias Andree <matthias.andree@web.de>
To: freebsd-gnats-submit@FreeBSD.org,
	ma@dt.e-technik.uni-dortmund.de, sos@freebsd.org,
	Andrew Gallatin <gallatin@cs.duke.edu>
Cc:  
Subject: Re: kern/37060: kernel panic with hw.ata.tags=1 in ata-disk.c:710
Date: Tue, 21 May 2002 01:17:27 +0200

 I believe I know now what's going on.
 
 ata_raiddisk_attach in ata-disk.c:218 (ad_attach()) tries to read a RAID
 signature from the newly detected drive. In my case, that's ad1, the
 secondary master. At that time, ad0 (primary master) and ad1 are known,
 ad2 is not.
 
    217      /* if this disk belongs to an ATA RAID dont print the probe */
    218      if (ata_raiddisk_attach(adp))
    219          adp->flags |=3D AD_F_RAID_SUBDISK;
    220      else
    221          ad_print(adp);
    222  }
 
 now, ata_raiddisk_attach uses ar_rw, which calls upon ata_start, ata-all.c:=
 661.
 
    675      s =3D splbio();
    676  #if NATADISK > 0
    677      /* find & call the responsible driver if anything on the ATA qu=
 eue *
 /
    678      if (TAILQ_EMPTY(&ch->ata_queue)) {
    679          if (ch->devices & (ATA_ATA_MASTER) && ch->device[MASTER].dr=
 iver)
    680              ad_start(&ch->device[MASTER]);
    681          if (ch->devices & (ATA_ATA_SLAVE) && ch->device[SLAVE].driv=
 er)
    682              ad_start(&ch->device[SLAVE]);
    683      }
    684      if ((ad_request =3D TAILQ_FIRST(&ch->ata_queue))) {
    685          TAILQ_REMOVE(&ch->ata_queue, ad_request, chain);
    686          ch->active =3D ATA_ACTIVE_ATA;
    687          ch->running =3D ad_request;
    688          if (ad_transfer(ad_request) =3D=3D ATA_OP_CONTINUES) {
    689              splx(s);
    690              return;
    691          }
    692      }
 
 ata_start knows it has no driver for the slave yet (ad2, secondary
 slave, has not yet been probed).
 
 However, in l. 688, ad_transfer is called, which itself (ata-disk.c:495)
 calls ad_service:
    469          if (adp->device->mode >=3D ATA_DMA &&
    470              !ata_dmasetup(adp->device->channel, adp->device->unit,
    471                            request->dmatab, request->data, request->=
 bytecount)) {
    472              request->flags |=3D ADR_F_DMA_USED;
    473              request->currentsize =3D request->bytecount;
    474 =20
    475              /* do we have tags enabled ? */
    476              if (adp->flags & AD_F_TAG_ENABLED) {
 =2E..
    491                  /* if ATA bus RELEASE check for SERVICE */
    492                  if (adp->flags & AD_F_TAG_ENABLED &&
    493                      ATA_INB(adp->device->channel->r_io, ATA_IREASON=
 ) &
    494                      ATA_I_RELEASE)
    495                      return ad_service(adp, 1);
 
 ad_service now tries to figure the slave flags at lines 710-713 and
 failes because adp->device->channel->device[0] is uninitialized yet is.
 
 I'm not sure as to what the proper fix is, as I don't understand the
 "change" flag and its implications.
 
 However, I set a breakpoint at line #706, and on the first five hits, I
 issued "set change=3D0" and "c", and the drives probed fine. I then
 removed the breakpoint and checked what the system would do: it booted
 up fine.
 
    702  int
    703  ad_service(struct ad_softc *adp, int change)
    704  {
    705      /* do we have to check the other device on this channel ? */
    706      if (adp->device->channel->flags & ATA_QUEUED && change) {
    707          int device =3D adp->device->unit;
    708 =20
    709          if (adp->device->unit =3D=3D ATA_MASTER) {
    710              if (adp->device->channel->devices & ATA_ATA_SLAVE &&
    711                  ((struct ad_softc *)
    712                   (adp->device->channel->
    713                    device[ATA_DEV(ATA_SLAVE)].driver))->flags&AD_F_T=
 AG_EN
 ABLED)
    714                  device =3D ATA_SLAVE;
 =2E..
    732      adp->device->channel->status =3D
    733          ATA_INB(adp->device->channel->r_altio, ATA_ALTSTAT);
    734  =20
    735      /* do we have a SERVICE request from the drive ? */
    736      if (adp->flags & AD_F_TAG_ENABLED &&
    737          adp->outstanding > 0 &&
 =2E..
 
 I'm not sure what the proper fix to this is, if it is to temporary
 disable Tagged Queueing until all drives are probed, fix ad_service to
 set change =3D 0 locally when the other drive is uninitialized, whatever.
 
 With the "set change =3D 0" gdb intervention, here's the ATA dmesg:
 
 atapci0: <VIA 82C686 ATA66 controller> port 0xffa0-0xffaf at device 7.1 on =
 pci0
 ata0: iobase=3D0x01f0 altiobase=3D0x03f6 bmaddr=3D0xffa0
 ata0: mask=3D03 ostat0=3D50 ostat2=3D00
 ata0-master: ATAPI 00 00
 ata0-slave: ATAPI 00 00
 ata0: mask=3D03 stat0=3D50 stat1=3D00
 ata0-master: ATA 01 a5
 ata0: devices=3D01
 ata0: at 0x1f0 irq 14 on atapci0
 ata1: iobase=3D0x0170 altiobase=3D0x0376 bmaddr=3D0xffa8
 ata1: mask=3D03 ostat0=3D50 ostat2=3D50
 ata1-master: ATAPI 00 00
 ata1-slave: ATAPI 00 00
 ata1: mask=3D03 stat0=3D50 stat1=3D50
 ata1-master: ATA 01 a5
 ata1-slave: ATA 01 a5
 ata1: devices=3D03
 ata1: at 0x170 irq 15 on atapci0
 =2E..
 ad0: success setting UDMA4 on VIA chip
 Creating DISK ad0
 ar: FreeBSD check1 failed
 ad0: <Maxtor 4W060H4/AAH41310> ATA-6 disk at ata0-master
 ad0: 58610MB (120033900 sectors), 119081 C, 16 H, 63 S, 512 B
 ad0: 16 secs/int, 1 depth queue, UDMA66
 ad0: piomode=3D4 dmamode=3D2 udmamode=3D5 cblid=3D1
 ad1: success setting UDMA4 on VIA chip
 Creating DISK ad1
 ar: FreeBSD check1 failed
 ad1: <IBM-DTLA-307045/TX6OA6AA> ATA-5 disk at ata1-master
 ad1: 43979MB (90069840 sectors), 89355 C, 16 H, 63 S, 512 B
 ad1: 16 secs/int, 32 depth queue, tagged UDMA66
 ad1: piomode=3D4 dmamode=3D2 udmamode=3D5 cblid=3D1
 ad2: success setting UDMA4 on VIA chip
 Creating DISK ad2
 ar: FreeBSD check1 failed
 ad2: <WDC AC420400D/J58OA30K> ATA-4 disk at ata1-slave
 ad2: 19470MB (39876480 sectors), 39560 C, 16 H, 63 S, 512 B
 ad2: 16 secs/int, 1 depth queue, UDMA66
 ad2: piomode=3D4 dmamode=3D2 udmamode=3D4 cblid=3D1
 =2E..
 [on with SCSI probes]
 
 Nothing is mounted from ad1, and I want an "OK" from S=F8ren before I try
 to use ad1 with the "set change=3D0" during the bootup to see if tagged
 queueing actually works.
 
 --=20
 Matthias Andree
Responsible-Changed-From-To: freebsd-bugs->sos 
Responsible-Changed-By: dwmalone 
Responsible-Changed-When: Tue Jun 11 14:38:17 PDT 2002 
Responsible-Changed-Why:  
Soren - Matthias has provided quite a lot of detail here - can 
you have a look at it? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=37060 
State-Changed-From-To: open->feedback 
State-Changed-By: sos 
State-Changed-When: Mon Aug 19 12:04:46 PDT 2002 
State-Changed-Why:  
That bug has been solved long ago, please update to RELENG_4 and 
get back to me with the result. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=37060 

From: Matthias Andree <matthias.andree@web.de>
To: freebsd-gnats-submit@FreeBSD.org, ma@dt.e-technik.uni-dortmund.de
Cc:  
Subject: Re: kern/37060: kernel panic with hw.ata.tags=1 in ata-disk.c:710
Date: Sat, 19 Oct 2002 12:35:47 +0200

 Soeren (sos) has fixed the problem in summer 2002 (it turned out the ata
 driver did not cope with one tags-capable and one non-capable drive,
 which has been fixed), so this PR can be closed.
State-Changed-From-To: feedback->closed 
State-Changed-By: sos 
State-Changed-When: Mon Apr 28 11:28:59 PDT 2003 
State-Changed-Why:  
Problem solved. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=37060 
>Unformatted:
