From sa@main.svzserv.kemerovo.su  Tue Dec 23 04:35:19 2003
Return-Path: <sa@main.svzserv.kemerovo.su>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id EACC016A50C
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 23 Dec 2003 04:35:19 -0800 (PST)
Received: from main.svzserv.kemerovo.su (hq.svzserv.kemerovo.su [213.184.65.65])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 0811E43D50
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 23 Dec 2003 04:35:14 -0800 (PST)
	(envelope-from sa@main.svzserv.kemerovo.su)
Received: from main.svzserv.kemerovo.su (sa@localhost [127.0.0.1])
	by main.svzserv.kemerovo.su (8.12.9/8.12.9) with ESMTP id hBNCZA16051656
	for <FreeBSD-gnats-submit@freebsd.org>; Tue, 23 Dec 2003 19:35:10 +0700 (NKZ)
	(envelope-from sa@main.svzserv.kemerovo.su)
Received: (from sa@localhost)
	by main.svzserv.kemerovo.su (8.12.9/8.12.9/Submit) id hBNCZ97r051640;
	Tue, 23 Dec 2003 19:35:09 +0700 (NKZ)
	(envelope-from sa)
Message-Id: <200312231235.hBNCZ97r051640@main.svzserv.kemerovo.su>
Date: Tue, 23 Dec 2003 19:35:09 +0700 (NKZ)
From: Eugene Grosbein <eugen@kuzbass.ru>
Reply-To: eugen@kuzbass.ru
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Post-PAE stable SMP machine freezes
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         60526
>Category:       kern
>Synopsis:       Post-PAE stable SMP machine freezes
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Dec 23 04:40:18 PST 2003
>Closed-Date:    Wed May 03 16:31:14 GMT 2006
>Last-Modified:  Wed May 03 16:31:14 GMT 2006
>Originator:     Eugene Grosbein
>Release:        FreeBSD 4.8-STABLE i386
>Organization:
Svyaz Service JSC
>Environment:
System: FreeBSD main.svzserv.kemerovo.su 4.8-STABLE FreeBSD 4.8-STABLE #4: Fri Dec 19 13:44:49 NKZ 2003 sa@main.svzserv.kemerovo.su:/usr/obj/usr/src/sys/MAIN i386
	CPUTYPE=p3 and no other optimizations
	SMP machine with old ServerWorks chipset

>Description:
	
	This machine worked rock-stable with 4-STABLE for long time more
than 2 years. And it is still rock-stable with 4.8-STABLE cvsup'd upto
date=2003.08.08.00.00.00 (plus security patches).
It freezes hard with later 4-STABLE versions.

It boots, works short time as expected and hangs: network services
do not respond, kernel does not respond to pings, syscons does not
respond to Alt-Fn (consoles do not switch), kernel does not break to DDB
using Ctrl-Alt-ESC. No error messages in system logs located on SCSI drive.
No error messages in system logs that are being sent over network to
another server. Power down/up cycle is needed to reboot the box.

	Its uptime depends on ata(4) activity. When ATA disk is mounted
read-write, it takes several minutes (sometimes less than a minute)
to hang after going to multiuser. When ATA disk is mounted read-only,
it hangs just several times per day. When ATA disk is not mounted,
it hangs too, but may work for long time before freeze occures.
Downgraded to 4.8-STABLE, it does not hang at all. The box is loaded
ftp server, proxy server, mail server. FTP data reside on ATA drive,
the rest is on SCSI.

	This host ran 4.8-STABLE of March 2003 for several month when I've
upgraded to 4-STABLE. It starts to hang, I downgraded to mentioned date
and it works nice again. I tried to upgrade to STABLE twice more since then
and both times saw this bug is not corrected. I saw ones a message on
the console about ATA timeout but it does not appear with recent 4-STABLE.

	Here comes dmesg.boot

Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 4.8-STABLE #4: Fri Dec 19 13:44:49 NKZ 2003
    sa@main.svzserv.kemerovo.su:/usr/obj/usr/src/sys/MAIN
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel Pentium III (866.43-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x686  Stepping = 6
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 536805376 (524224K bytes)
config> q
avail memory = 518467584 (506316K bytes)
Programming 16 pins in IOAPIC #0
Programming 16 pins in IOAPIC #1
FreeBSD/SMP: Multiprocessor motherboard: 2 CPUs
 cpu0 (BSP): apic id:  3, version: 0x00040011, at 0xfee00000
 cpu1 (AP):  apic id:  0, version: 0x00040011, at 0xfee00000
 io0 (APIC): apic id:  4, version: 0x000f0011, at 0xfec00000
 io1 (APIC): apic id:  5, version: 0x000f0011, at 0xfec01000
Preloaded elf kernel "kernel" at 0xc03c0000.
Preloaded userconfig_script "/boot/kernel.conf" at 0xc03c009c.
VESA: v2.0, 4096k memory, flags:0x0, mode table:0xc034ed22 (1000022)
VESA: ATI MACH64
Pentium Pro MTRR support enabled
md0: Malloc disk
Using $PIR table, 268435454 entries at 0xc00fdf10
npx0: <math processor> on motherboard
npx0: INT 16 interface
pcib0: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
IOAPIC #1 intpin 3 -> irq 2
IOAPIC #1 intpin 2 -> irq 5
IOAPIC #1 intpin 9 -> irq 9
pci0: <PCI bus> on pcib0
pci0: <ATI Mach64-GV graphics accelerator> at 2.0 irq 2
fxp0: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port 0x5400-0x543f mem 0xfb000000-0xfb0fffff,0xfb201000-0xfb201fff irq 5 at device 3.0 on pci0
fxp0: Ethernet address 00:d0:b7:b6:2b:f9
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp1: <Intel 82557/8/9 EtherExpress Pro/100(B) Ethernet> port 0x5440-0x547f mem 0xfb100000-0xfb1fffff,0xfb202000-0xfb202fff irq 9 at device 9.0 on pci0
fxp1: Ethernet address 00:02:b3:26:a9:52
inphy1: <i82555 10/100 media interface> on miibus1
inphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <ServerWorks IB6566 PCI to ISA bridge> at device 15.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <ServerWorks ROSB4 ATA33 controller> port 0x5480-0x548f,0x374-0x377,0x170-0x177 at device 15.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pcib1: <ServerWorks NB6635 3.0LE host to PCI bridge> on motherboard
IOAPIC #1 intpin 0 -> irq 10
IOAPIC #1 intpin 1 -> irq 11
pci1: <PCI bus> on pcib1
ahc0: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x5800-0x58ff mem 0xfd000000-0xfd000fff irq 10 at device 4.0 on pci1
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
ahc1: <Adaptec aic7899 Ultra160 SCSI adapter> port 0x6000-0x60ff mem 0xfd001000-0xfd001fff irq 11 at device 4.1 on pci1
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
orm0: <Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc97ff,0xc9800-0xcf7ff,0xcf800-0xd07ff on isa0
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 on isa0
sio0: type 16550A
sio1 at port 0x2f8-0x2ff irq 3 flags 0x10 on isa0
sio1: type 16550A
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppi0: <Parallel I/O> on ppbus0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
APIC_IO: routing 8254 via 8259 and IOAPIC #0 intpin 0
DUMMYNET initialized (011031)
IP packet filtering initialized, divert enabled, rule-based forwarding enabled, default to deny, unlimited logging
IPsec: Initialized Security Association Processing.
SMP: AP CPU #1 Launched!
ad0: 29410MB <QUANTUM FIREBALLP AS30.0> [59755/16/63] at ata0-master UDMA33
acd0: CDROM <ASUS CD-S400/A> at ata0-slave PIO4
Waiting 15 seconds for SCSI devices to settle
Mounting root from ufs:/dev/da0s1a
da0 at ahc0 bus 0 target 0 lun 0
da0: <QUANTUM ATLAS_V_18_WLS 0230> Fixed Direct Access SCSI-3 device 
da0: 160.000MB/s transfers (80.000MHz, offset 63, 16bit), Tagged Queueing Enabled
da0: 17510MB (35861388 512 byte sectors: 255H 63S/T 2232C)

>How-To-Repeat:

	It repeats by itself very soon after reboot for recent 4.9-STABLE.
	I had important data loss on my SCSI drive because of unclean
	reboot already.

>Fix:

	Unknown
>Release-Note:
>Audit-Trail:

From: Eugene Grosbein <eugen@kuzbass.ru>
To: bug-followup@freebsd.org
Cc: sos@freebsd.org, stable@freebsd.org
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Thu, 18 Mar 2004 11:00:09 +0700

 Hi!
 
 According to
 http://freebsd.rambler.ru/bsdmail/freebsd-hackers_2003/msg03936.html
 the problem may be related to the <ServerWorks ROSB4 ATA33 controller>
 and ATA write cache (hw.ata.wc=1).
 
 Note however, I had no kernel panic with post-PAE STABLE.
 I had kernel lockup. I can understand kernel panic due to the faulty
 devices but not endless loop that it seems to happen.
 
 It's known (and noted in the Errata for 4.9-RELEASE)
 that PAE integration broke hw.ata.tags. Perhaps, its time
 either to document that hw.ata.wc is broken for noted controller too,
 either to forbid write cache for it by force.
 
 I run my machine using pre-PAE 4.8-STABLE with security patches still.
 
 Eugene Grosbein

From: =?KOI8-R?Q?S=3Fren_Schmidt?= <sos@DeepCore.dk>
To: Eugene Grosbein <eugen@kuzbass.ru>
Cc: bug-followup@FreeBSD.ORG, sos@FreeBSD.ORG, stable@FreeBSD.ORG
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Thu, 18 Mar 2004 08:30:02 +0100

 Eugene Grosbein wrote:
 > Hi!
 > 
 > According to
 > http://freebsd.rambler.ru/bsdmail/freebsd-hackers_2003/msg03936.html
 > the problem may be related to the <ServerWorks ROSB4 ATA33 controller>
 > and ATA write cache (hw.ata.wc=1).
 > 
 > Note however, I had no kernel panic with post-PAE STABLE.
 > I had kernel lockup. I can understand kernel panic due to the faulty
 > devices but not endless loop that it seems to happen.
 > 
 > It's known (and noted in the Errata for 4.9-RELEASE)
 > that PAE integration broke hw.ata.tags. Perhaps, its time
 > either to document that hw.ata.wc is broken for noted controller too,
 > either to forbid write cache for it by force.
 
 I have a hard time beliving WC can be the problem. WC is a function of 
 the disk drive it has nothing todo with the controller or even the way 
 we talk to the disk...
 That said, the ROSB4 is known to be wierd, bad things can easily happend 
 with it in some setups and there is noting we can do about it in SW 
 (less using PIO mode). However some HW vendors seem to have found a HW 
 solution to the problem (ASUS is one of them, my old CUR-DLS doesn't 
 show the problems at all)...
 
 -- 
 -S?ren
 

From: Doug White <dwhite@gumbysoft.com>
To: Eugene Grosbein <eugen@kuzbass.ru>
Cc: bug-followup@freebsd.org, stable@freebsd.org, sos@freebsd.org
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Thu, 18 Mar 2004 10:23:43 -0800 (PST)

 On Thu, 18 Mar 2004, Eugene Grosbein wrote:
 
 > According to
 > http://freebsd.rambler.ru/bsdmail/freebsd-hackers_2003/msg03936.html
 > the problem may be related to the <ServerWorks ROSB4 ATA33 controller>
 > and ATA write cache (hw.ata.wc=1).
 
 The problem is with DMA mode on these controllers, not write caching.
 Write caching may help to get the data rate up fast enough to trigger the
 bug, and turning it off slows things down.
 
 I HIGHLY recommend using a DIFFERENT controller for disks. The ROSB4 is OK
 for CDROMs and the like, but use a different controller for the system
 drive.  Promise controllers work great. :)
 
 ATA tagging is known to work with only a few disk models.  You should
 enable ATA tags ONLY if you KNOW your drive supports it, and test
 extensively to ensure stability.
 
 -- 
 Doug White                    |  FreeBSD: The Power to Serve
 dwhite@gumbysoft.com          |  www.FreeBSD.org

From: =?ISO-8859-1?Q?S=F8ren_Schmidt?= <sos@DeepCore.dk>
To: Doug White <dwhite@gumbysoft.com>
Cc: Eugene Grosbein <eugen@kuzbass.ru>, bug-followup@FreeBSD.ORG,
	stable@FreeBSD.ORG, sos@FreeBSD.ORG
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Thu, 18 Mar 2004 20:54:17 +0100

 Doug White wrote:
   >>According to
 >>http://freebsd.rambler.ru/bsdmail/freebsd-hackers_2003/msg03936.html
 >>the problem may be related to the <ServerWorks ROSB4 ATA33 controller>
 >>and ATA write cache (hw.ata.wc=1).
 > The problem is with DMA mode on these controllers, not write caching.
 > Write caching may help to get the data rate up fast enough to trigger the
 > bug, and turning it off slows things down.
 > 
 > I HIGHLY recommend using a DIFFERENT controller for disks. The ROSB4 is OK
 > for CDROMs and the like, but use a different controller for the system
 > drive.  Promise controllers work great. :)
 
 Cant disagree with the last sentence, however my only i386 SMP machine 
 has exactly the ROSB4 chip in it, and I use that for disks etc, newer 
 had a problem with it, but again this is an ASUS board and they 
 supposedly fixed the problem in HW (the *only* way to fix it btw).
 
 > ATA tagging is known to work with only a few disk models.  You should
 > enable ATA tags ONLY if you KNOW your drive supports it, and test
 > extensively to ensure stability.
 
 To be more exact, only a very few disks supports tags, in fact I only 
 know of IBM and a select few WDC disks that actually anounces support 
 for tags. The ATA driver (in -stable) only tries to enable tags if the 
 HW says it supports it and its a known to work drive. However the PAE 
 import has broken tags support so it doesn't work on any disks :(
 
 
 -- 
 -Sren
 

From: Eugene Grosbein <eugen@kuzbass.ru>
To: Doug White <dwhite@gumbysoft.com>
Cc: bug-followup@freebsd.org, stable@freebsd.org, sos@freebsd.org
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Fri, 19 Mar 2004 09:06:41 +0700

 Doug White wrote:
 
 > I HIGHLY recommend using a DIFFERENT controller for disks. The ROSB4 is OK
 > for CDROMs and the like, but use a different controller for the system
 > drive.  Promise controllers work great. :)
 
 Hmm, I use SMP motherboard Intel STL2 with ServerWorks ServerSet III LE
 and integrated ROSB4:
 http://support.intel.com/support/motherboards/server/STL2/
 I can't found any notes here about the controller.
 
 Could you please point me to any kind of "official" announcement
 from Intel, chipset or controller producer?
 
 > ATA tagging is known to work with only a few disk models.  You should
 > enable ATA tags ONLY if you KNOW your drive supports it, and test
 > extensively to ensure stability.
 
 It DID work with my IBM drive (IC35L040AVER, noted in ata(4) man page)
 before PAE intergration without problems. Now I turned it off - 
 it panices 4.9-STABLE under heavy load.
 
 Eugene

From: Doug White <dwhite@gumbysoft.com>
To: Eugene Grosbein <eugen@kuzbass.ru>
Cc: bug-followup@freebsd.org, stable@freebsd.org, sos@freebsd.org
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Fri, 19 Mar 2004 16:10:36 -0800 (PST)

 On Fri, 19 Mar 2004, Eugene Grosbein wrote:
 
 > Doug White wrote:
 >
 > > I HIGHLY recommend using a DIFFERENT controller for disks. The ROSB4 is OK
 > > for CDROMs and the like, but use a different controller for the system
 > > drive.  Promise controllers work great. :)
 >
 > Hmm, I use SMP motherboard Intel STL2 with ServerWorks ServerSet III LE
 > and integrated ROSB4:
 > http://support.intel.com/support/motherboards/server/STL2/
 > I can't found any notes here about the controller.
 >
 > Could you please point me to any kind of "official" announcement
 > from Intel, chipset or controller producer?
 
 Someone referred me to a EETimes article, and I have hundreds of Tyan
 S2510/S2518 boards I can reproduce the problem on under multiple operating
 systems.
 
 Considering the ROSB4 is only ATA33, you probably want something faster.
 :)
 
 -- 
 Doug White                    |  FreeBSD: The Power to Serve
 dwhite@gumbysoft.com          |  www.FreeBSD.org

From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: Doug White <dwhite@gumbysoft.com>
Cc: stable@freebsd.org, bug-followup@freebsd.org, sos@freebsd.org
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Sat, 20 Mar 2004 14:01:20 +0700

 On Fri, Mar 19, 2004 at 04:10:36PM -0800, Doug White wrote:
 
 > > Hmm, I use SMP motherboard Intel STL2 with ServerWorks ServerSet III LE
 > > and integrated ROSB4:
 > > http://support.intel.com/support/motherboards/server/STL2/
 > > I can't found any notes here about the controller.
 > >
 > > Could you please point me to any kind of "official" announcement
 > > from Intel, chipset or controller producer?
 > 
 > Someone referred me to a EETimes article, and I have hundreds of Tyan
 > S2510/S2518 boards I can reproduce the problem on under multiple operating
 > systems.
 
 I'm sure you are right, I'm just looking for a better description
 of problem, not just "it is buggy and leads to problems".
 
 Eugene Grosbein

From: Eugene Grosbein <eugen@grosbein.pp.ru>
To: Doug White <dwhite@gumbysoft.com>
Cc: Eugene Grosbein <eugen@kuzbass.ru>, stable@freebsd.org,
	bug-followup@freebsd.org, sos@freebsd.org
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Sun, 21 Mar 2004 14:50:20 +0700

 Doug White wrote:
 
 > Someone referred me to a EETimes article, and I have hundreds of Tyan
 > S2510/S2518 boards I can reproduce the problem on under multiple operating
 > systems.
 
 I googled a little for ROSB4.
 
 Here http://www.ussg.iu.edu/hypermail/linux/kernel/0206.1/1212.html
 I've found the next statement:
 
 > The system will lock solid when both IDE channels are accessed,
 > and either one is using DMA. Since I want DMA, I simply abandon the
 > secondary channel.
 
 Is it possible do such a thing using FreeBSD?
 
 Next, Linux seem to have a workarond(?) since 2002:
 
 http://www.ussg.iu.edu/hypermail/linux/kernel/0206.1/0996.html
 
 Soren, could you please look at this?
 
 Eugene

From: =?KOI8-R?Q?S=3Fren_Schmidt?= <sos@DeepCore.dk>
To: Eugene Grosbein <eugen@grosbein.pp.ru>
Cc: Doug White <dwhite@gumbysoft.com>,
	Eugene Grosbein <eugen@kuzbass.ru>, stable@FreeBSD.ORG,
	bug-followup@FreeBSD.ORG, sos@FreeBSD.ORG
Subject: Re: kern/60526: Post-PAE stable SMP machine freezes
Date: Sun, 21 Mar 2004 10:51:10 +0100

 Eugene Grosbein wrote:
 > Doug White wrote:
 > 
 > 
 >>Someone referred me to a EETimes article, and I have hundreds of Tyan
 >>S2510/S2518 boards I can reproduce the problem on under multiple operating
 >>systems.
 > 
 > 
 > I googled a little for ROSB4.
 > 
 > Here http://www.ussg.iu.edu/hypermail/linux/kernel/0206.1/1212.html
 > I've found the next statement:
 > 
 > 
 >>The system will lock solid when both IDE channels are accessed,
 >>and either one is using DMA. Since I want DMA, I simply abandon the
 >>secondary channel.
 > 
 > 
 > Is it possible do such a thing using FreeBSD?
 > 
 > Next, Linux seem to have a workarond(?) since 2002:
 > 
 > http://www.ussg.iu.edu/hypermail/linux/kernel/0206.1/0996.html
 > 
 > Soren, could you please look at this?
 
 That fix newer worked AFAIK, today linux just dont allow UDMA on the 
 ROSB4 at all.
 
 There is no good SW fix, and its also not established that a HW fix 
 exists as well, however I cannot reproduce the problem on my ASUS 
 CUR-DLS engineering sample, but ASUS wont tell anything about it...
 
 -- 
 -S?ren
 

From: Eugene Grosbein <eugen@kuzbass.ru>
To: bug-followup@freebsd.org
Cc: sos@freebsd.org
Subject: kern/60526 : Post-PAE stable SMP machine freezes
Date: Wed, 03 May 2006 21:36:59 +0800

 Hi!
 
 I've moved all my data from ROSB4 to SCSI drives
 and this machine runs fine under 4.11-STABLE now.
 This PR should be closed.
 
 Eugene Grosbein
State-Changed-From-To: open->closed 
State-Changed-By: linimon 
State-Changed-When: Wed May 3 16:29:30 UTC 2006 
State-Changed-Why:  
Submitter requested that this be closed.  Apparently the ROSB4 just 
simply can't be made to work with UDMA. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=60526 
>Unformatted:
