From nobody@FreeBSD.org  Thu Aug 16 12:20:05 2007
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 9950D16A419
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Aug 2007 12:20:05 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 8A1FC13C481
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Aug 2007 12:20:05 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.1/8.14.1) with ESMTP id l7GCK5Bl072525
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 16 Aug 2007 12:20:05 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.1/8.14.1/Submit) id l7GCK5xZ072524;
	Thu, 16 Aug 2007 12:20:05 GMT
	(envelope-from nobody)
Message-Id: <200708161220.l7GCK5xZ072524@www.freebsd.org>
Date: Thu, 16 Aug 2007 12:20:05 GMT
From: graham bradley <gbradley@rocketmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: gbde partitions fail at 28bit/48bit LBA addressing boundary
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         115572
>Category:       kern
>Synopsis:       [gbde] [patch] gbde partitions fail at 28bit/48bit LBA addressing boundary
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-geom
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Aug 16 12:30:01 GMT 2007
>Closed-Date:    Mon Jun 23 19:15:38 UTC 2008
>Last-Modified:  Mon Jun 23 19:15:38 UTC 2008
>Originator:     graham bradley
>Release:        FreeBSD 6.2-RELEASE
>Organization:
>Environment:
FreeBSD rabbit.net 6.2-RELEASE FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007     root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP  amd64
>Description:
#
# to illustrate faulty read/write of a gbde disk drive around the 137gb boundary

# prove the raw disk can be written everywhere (a few hours on a 500gb drive)

dd if=/dev/zero of=/dev/ad4 bs=64k

# or for smaller blocks....

dd if=/dev/zero of=/dev/ad4 bs=4k

# set the entire /dev/ad4 drive up as a gbde partition

gbde init ad4 -L /etc/gbde/ad4.lock
#...... it wants passwords here ......

gbde attach ad4 -l /etc/gbde/ad4.lock
#..... and password again .....

dd if=/dev/zero of=/dev/ad4 bs=64k oseek=1972156 count=1000
dd if=/dev/zero of=/dev/ad4 bs=4k oseek=31806314 count=1000

# which all works fine, but now if we write to the whole disk

dd if=/dev/zero of=/dev/ad4 bs=64k

# we see.....

rabbit kernel: ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434

# we have an error at around the (physical drive) block 268435434,
#  at the 28/48 bit LBA boundary.   n.b. the gbde partition address is about 130gb,
# as the block addresses are mapped in gbde to fit in the key blocks.

# oddly, trying smaller block sizes down to 2k still fail, whilst 1k and 512b succeed.

# This suggests that the earlier long dd write (somewhere) has trampled on (something)......

# to be sure the disk is still physically ok, we can do....

gbde detach ad4

dd if=/dev/zero of=/dev/ad4 bs=64k

# and the disk drive appears ok, including across the 137gb boundary.

#-------------------------------------------------------------------------------------------

# in service, with the gbde partition mounted, this write error is repeating each time
# the soft-update is triggered (every 30 seconds), as the disc is filled with data.....

rabbit kernel: g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
rabbit kernel: ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434

# exact details of the error vary according to file sizes, and start to appear after the disk
# is filled beyond 60-100gb of files.

# not using soft-update leaves us with the same problem, but it is less frequently flagged.

# It is impossible after the errors start to achieve a snapshot, although fsck claims
# the partition is fine!
# Although fsck sanctions the disk as clean, the errors left behind slowly propagate
# into the stored data in apparently random fashion, fsck then finds soft-update
# inconsistencies, and sometimes files are silently left broken.


>How-To-Repeat:
re-initialise a large new drive with new keys and attach gbde
no need for a newfs on the partition

large dd write near/across the 137gb boundary seems ok

attempt write /dev/zero from start to finish of the gbde partition,
this fails as dd hits the (physical) 137gb boundary,
at about 130gb in the gbde partition.

detach gbde, and show that write is ok to the raw drive.

check smartctl to show drive itself is error-free.
>Fix:
# n.b. geli does not map the partition on the physical drive, and apparently there's
#      no such problem.

# so write across the entire disk.....

geli init /dev/ad4
# .... password ....
geli attach /dev/ad4
# .... password ....

dd if=/dev/zero of=/dev/ad4.eli bs=64k

# completes ok, so I am switching over to geli for big drives.



Patch attached with submission follows:

#
# to illustrate faulty read/write of a gbde disk drive around the 137gb boundary

# prove the raw disk can be written everywhere (a few hours on a 500gb drive)

dd if=/dev/zero of=/dev/ad4 bs=64k

# or for smaller blocks....

dd if=/dev/zero of=/dev/ad4 bs=4k

# set the entire /dev/ad4 drive up as a gbde partition

gbde init ad4 -L /etc/gbde/ad4.lock
#...... it wants passwords here ......

gbde attach ad4 -l /etc/gbde/ad4.lock
#..... and password again .....

dd if=/dev/zero of=/dev/ad4 bs=64k oseek=1972156 count=1000
dd if=/dev/zero of=/dev/ad4 bs=4k oseek=31806314 count=1000

# which all works fine, but now if we write to the whole disk

dd if=/dev/zero of=/dev/ad4 bs=64k

# we see.....

rabbit kernel: ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434

# we have an error at around the (physical drive) block 268435434,
#  at the 28/48 bit LBA boundary.   n.b. the gbde partition address is about 130gb,
# as the block addresses are mapped in gbde to fit in the key blocks.

# oddly, trying smaller block sizes down to 2k still fail, whilst 1k and 512b succeed.

# This suggests that the earlier long dd write (somewhere) has trampled on (something)......

# to be sure the disk is still physically ok, we can do....

gbde detach ad4

dd if=/dev/zero of=/dev/ad4 bs=64k

# and the disk drive appears ok, including across the 137gb boundary.

#------------------------------------------------------------------------------------------- 

# in service, with the gbde partition mounted, this write error is repeating each time
# the soft-update is triggered (every 30 seconds), as the disc is filled with data.....

rabbit kernel: g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
rabbit kernel: ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434

# exact details of the error vary according to file sizes, and start to appear after the disk
# is filled beyond 60-100gb of files.

# not using soft-update leaves us with the same problem, but it is less often flagged.

# It is impossible after the errors start to achieve a snapshot, although fsck claims
# the partition is fine!
# Although fsck sanctions the disk as clean, the errors left behind slowly propagate
# into the stored data in apparently random fashion, fsck then finds soft-update
# inconsistencies, and sometimes files are silently left broken.

#------------------------------------------------------------------------------------------- 

# n.b. geli does not map the partition on the physical drive, and apparently there's
#      no such problem.

# so write across the entire disk.....

dd if=/dev/zero of=/dev/ad4.eli bs=64k

# completes ok, so I am switching over to that method for big drives.

#-------------------------------------------------------------------------------------------

# DIAGNOSTICS

# This is a new 500gb drive, and smartctl -a /dev/ad4 shows no drive errors afterwards.....

rabbit# smartctl -a /dev/ad4
smartctl version 5.37 [amd64-portbld-freebsd6.2] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD5000AAKS-22TMA0
Serial Number:    WD-WCAPW3258761
Firmware Version: 12.01C01
User Capacity:    500,107,862,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Thu Aug 16 11:11:53 2007 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                 (12000) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 150) minutes.
Conveyance self-test routine
recommended polling time:        (   6) minutes.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0003   187   187   021    Pre-fail  Always       -       5625
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       25
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000e   200   200   051    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       857
 10 Spin_Retry_Count        0x0012   100   253   051    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0012   100   253   051    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       23
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       28
194 Temperature_Celsius     0x0022   120   105   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0012   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   051    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


#-------------------------------------------------------------------------------------------

# the output of dmesg when I do these tests ......
 
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE #0: Fri Jan 12 08:43:30 UTC 2007
    root@portnoy.cse.buffalo.edu:/usr/obj/usr/src/sys/SMP
ACPI APIC Table: <Nvidia AWRDACPI>
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) 64 Processor 3500+ (2210.77-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20ff2  Stepping = 2
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
  AMD Features2=0x1<LAHF>
real memory  = 1073676288 (1023 MB)
avail memory = 1023930368 (976 MB)
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff,0xcf0-0xcf3 on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <NVIDIA nForce3-250 AGP Controller> mem 0xf4000000-0xf5ffffff at device 0.0 on pci0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfa003000-0xfa003fff irq 20 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 4 ports with 4 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xfa004000-0xfa004fff irq 21 at device 2.1 on pci0
ohci1: [GIANT-LOCKED]
usb1: OHCI version 1.0, legacy support
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 4 ports with 4 removable, self powered
ehci0: <NVIDIA nForce3 250 USB 2.0 controller> mem 0xfa005000-0xfa0050ff irq 22 at device 2.2 on pci0
ehci0: [GIANT-LOCKED]
usb2: EHCI version 1.0
usb2: companion controllers, 4 ports each: usb0 usb1
usb2: <NVIDIA nForce3 250 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub2: 8 ports with 8 removable, self powered
pci0: <multimedia, audio> at device 6.0 (no driver attached)
atapci0: <nVidia nForce3 Pro UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 8.0 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
atapci1: <nVidia nForce3 Pro SATA150 controller> port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xdc00-0xdc0f,0xe000-0xe07f irq 21 at device 10.0 on pci0
ata2: <ATA channel 0> on atapci1
ata3: <ATA channel 1> on atapci1
pcib1: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <display, VGA> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> at device 14.0 on pci0
pci2: <ACPI PCI bus> on pcib2
skc0: <Marvell Gigabit Ethernet> port 0xa000-0xa0ff mem 0xf9000000-0xf9003fff irq 19 at device 11.0 on pci2
skc0: Marvell Yukon Lite Gigabit Ethernet rev. (0x9)
sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
sk0: Ethernet address: 00:0f:ea:ff:33:86
miibus0: <MII bus> on sk0
e1000phy0: <Marvell 88E1000 Gigabit PHY> on miibus0
e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0: <Standard parallel printer port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse, device ID 3
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2210769739 Hz quality 800
Timecounters tick every 1.000 msec
ad0: 117800MB <IC35L120AVV207 0 V24OA66A> at ata0-master UDMA100
ad1: 117800MB <IC35L120AVV207 0 V24OA66A> at ata0-slave UDMA100
ad2: 238475MB <WDC WD2500SB-01KBC0 08.02D08> at ata1-master UDMA100
ad3: 58644MB <IC35L060AVV207 0 V22OA63A> at ata1-slave UDMA100
ad4: 476940MB <WDC WD5000AAKS-22TMA0 12.01C01> at ata2-master SATA150
ad6: 476940MB <WDC WD5000AAKS-75TMA0 12.01C01> at ata3-master SATA150
Trying to mount root from ufs:/dev/ad0s1a
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding disabled, default to deny, logging disabled
sk0: link state changed to UP
sk0: link state changed to DOWN
sk0: link state changed to UP
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
g_vfs_done():ad4.bde[WRITE(offset=130278653952, length=16384)]error = 5
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435434
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435450
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435450
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435450
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435450
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435450
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435454
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435454
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435454
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435433
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435449
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435449
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435453
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435453
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435453
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435453
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435453
ad4: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435449
ad4: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435449
ad4: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=10<NID_NOT_FOUND> LBA=268435433
GEOM_ELI: Device ad4.eli created.
GEOM_ELI: Encryption: AES-CBC 128
GEOM_ELI:     Crypto: software


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: linimon 
State-Changed-When: Thu Aug 16 16:45:59 UTC 2007 
State-Changed-Why:  
I don't see the claimed patch? 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115572 
State-Changed-From-To: feedback->open 
State-Changed-By: linimon 
State-Changed-When: Fri Aug 17 22:07:57 UTC 2007 
State-Changed-Why:  
Feedback received. 


Responsible-Changed-From-To: freebsd-bugs->freebsd-geom 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Fri Aug 17 22:07:57 UTC 2007 
Responsible-Changed-Why:  

http://www.freebsd.org/cgi/query-pr.cgi?pr=115572 

From: linimon@lonesome.com (Mark Linimon)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/115572: [gbde] gbde partitions fail at 28bit/48bit LBA addressing boundary
Date: Fri, 17 Aug 2007 17:07:23 -0500

 ----- Forwarded message from Graham <gbradley@rocketmail.com> -----
 
 Sorry, but my only solution at the moment is to use geli !
 
 We should not guess at the cause, I know, but I am writing chunks in
 succession to a newly gbde initialised drive, checking access after
 each shot, with the intent of narrowing down to the specific write
 which causes the damage --- i.e. the next read and write at the 28/48
 boundary fails.
 
 This may take a little time, but I'd like to find a small specific
 write block address which does the damage.  It will then be easier
 (with a specific case) to work through the block address mapping which
 ghappens in gbde.  Then maybe a fix....hope this strategy is ok.
 
 On an optimistic note --- if we can pin this down, it may help explain
 some of the other gbde corruption cases in the bug list.
 
 I will keep you posted as soon as I can find a small, specific case
 which causes the problem.
 
 regards
 Graham
 
 ----- End forwarded message -----

From: linimon@lonesome.com (Mark Linimon)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/115572: [gbde] gbde partitions fail at 28bit/48bit LBA addressing boundary
Date: Fri, 17 Aug 2007 17:07:53 -0500

 ----- Forwarded message from Graham <gbradley@rocketmail.com> -----
 
 further to yesterday, I run this sequence...
 1. init and attach a gbde volume,
 2. write zeros from start of the volume, in 1GiB bursts, until dd hits
 an i/o error,
 3. detach gbde volume
 
 this error came after 121GiB written.
 
 Then I have run a repeated sequence of...
 1. init and attach a gbde volume,
 2. write using dd 1GiB of zeros starting 121 GiB into the device
 3. detach gbde volume
 
 like this.....
 
 #! /usr/local/bin/bash
 #
 trap 'gbde detach ad4;exit 9' 1 2 3 13 15
 #
 while [ 0 -eq 0 ]
 do
 gbde init ad4 -P password
 gbde attach ad4 -p password
 START=1982464
 COUNT=16384
 dd if=/dev/zero of=/dev/ad4.bde bs=64k oseek=$START count=$COUNT
 gbde detach ad4
 done
 exit 3
 #
 
 (and then I checked smartctl for no errors on the drive.)
 
 which gives an i/o error at different block counts, always with an LBA
 at or more often just a few sectors below the LBA=268435455
 
 The gbde-volume block offsets at error are much more varied, and a
 significant proportion of the tests do not show errors in this test.
 
 I have read the paper by Poul-Henning Kamp "GBDE-GEOM Disk Based
 Encryption" and it is clear that the gbde volume's sector addresses are
 mapped on the physical drive in a non-repeatable way each time "init"
 takes place.  This explains the movable nature of the error, and the
 symptoms perceived by the user when a filesystem is written in geom can
 vary according to whether the tripwire LBA address gets mapped to data,
 inodes, or superblock.
 
 I am afraid this sounds much like a mapping consistency issue in GBDE,
 the mapping is quite complex, and I expect there are others better
 placed to know where to look for it.
 
 I am happy to help if you wish, there is a long-standing issue here
 needs fixing, but I would need a prompt towards finding the code which
 implements the mapping.
 
 regards
 Graham
 
 ----- End forwarded message -----

From: Graham <gbradley@rocketmail.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/115572: [gbde] gbde partitions fail at 28bit/48bit LBA addressing boundary
Date: Sat, 18 Aug 2007 13:49:13 -0700 (PDT)

 I have stripped this problem down to its simplest form, which is that a
 block read or write where the block straddles the 137gb boundary fails.
 
 Simplest reproducible form:
 1. create a single partition on a large drive, so then the first
 partition starts at a sector offset of 1.  In my case this is
 /dev/ad4s1.
 
 2. attempt (say)....
 rabbit# dd if=/dev/zero of=/dev/ad4s1 oseek=2097151 count=1 bs=64k
 and the result is....
 dd: /dev/ad4s1: Input/output error
 1+0 records in
 0+0 records out
 0 bytes transferred in 0.000325 secs (0 bytes/sec)
 
 (If dd is performed on the raw drive, /dev/ad4 then block boundary is
 always a power of 2, and blocksize a smaller power of 2.  That's always
 ok.  But we can't assume we use drives that way.)
 
 So a transfer which starts in the 28-bit zone, but extends over into
 the 48-bit region, fails.  Such transfers happen in the superblock of
 certain size drives, and that plays havoc.  The sector mapping of gbde
 can do this, but soft-update gets screwed by this happening.  It's not
 actually to do with the crypto as I first suspected.
 
 
        
 ____________________________________________________________________________________Ready for the edge of your seat? 
 Check out tonight's top picks on Yahoo! TV. 
 http://tv.yahoo.com/

From: Graham <gbradley@rocketmail.com>
To: "Sren" Schmidt <sos@deepcore.dk>
Cc: bug-followup@FreeBSD.org, Poul-Henning Kamp <phk@phk.freebsd.dk>
Subject: Re: kern/115572: ata disk bug
Date: Sun, 19 Aug 2007 13:03:21 -0700 (PDT)

 thanks Sren....
 
 --- Sren Schmidt <sos@deepcore.dk> wrote:
 
 > Poul-Henning Kamp wrote:
 > > In message <200708182120.l7ILKBvF046099@freefall.freebsd.org>,
 > Graham writes:
 > >
 > >   
 > >> 2. attempt (say)....
 > >> rabbit# dd if=/dev/zero of=/dev/ad4s1 oseek=2097151 count=1 bs=64k
 > >> and the result is....
 > >> dd: /dev/ad4s1: Input/output error
 > >> 1+0 records in
 > >> 0+0 records out
 > >> 0 bytes transferred in 0.000325 secs (0 bytes/sec)
 > >>
 > >> (If dd is performed on the raw drive, /dev/ad4 then block boundary
 > is
 > >> always a power of 2, and blocksize a smaller power of 2.  That's
 > always
 > >> ok.  But we can't assume we use drives that way.)
 > >>
 > >> So a transfer which starts in the 28-bit zone, but extends over
 > into
 > >> the 48-bit region, fails.  Such transfers happen in the superblock
 > of
 > >> certain size drives, and that plays havoc.  The sector mapping of
 > gbde
 > >> can do this, but soft-update gets screwed by this happening.  It's
 > not
 > >> actually to do with the crypto as I first suspected.
 > >>     
 > >
 > > This is a problem in the ata disk driver.
 > >   
 > Yeah, the crossover point from using 28bit to 48bit addressing is 
 > flawed, below patch should fix that:
 > 
 > --- ata-all.c   23 Feb 2007 16:25:08 -0000      1.279
 > +++ ata-all.c   19 Aug 2007 09:25:58 -0000
 > @@ -738,7 +738,7 @@
 >  
 >      atadev->flags &= ~ATA_D_48BIT_ACTIVE;
 >  
 > -    if ((request->u.ata.lba >= ATA_MAX_28BIT_LBA ||
 > +    if (((request->u.ata.lba + request->u.ata.count) >= 
 > ATA_MAX_28BIT_LBA ||
 >          request->u.ata.count > 256) &&
 >         atadev->param.support.command2 & ATA_SUPPORT_ADDRESS48) {
 >  
 > 
 > 
 > -Sren
 > >
 > >
 > >   
 > 
 > 
 > 
 I have compiled with this patch and done an (admittedly superficial)
 test and the bug is not now visible.  I am quite happy to see this one
 marked as solved.  My apologies to P-H.K. for starting off on the wrong
 tack.
 
 Do I need to flag anyone else for this to get taken up in the stock
 kernel, or will this happen automatic now?
 
 regards
 Graham
 
 
 
        
 ____________________________________________________________________________________
 Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online.
 http://smallbusiness.yahoo.com/webhosting 

From: =?ISO-8859-1?Q?S=F8ren_Schmidt?= <sos@deepcore.dk>
To: Pete <xd888cc7541309c3f@f4n.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/115572: [gbde] gbde partitions fail at 28bit/48bit LBA addressing
 boundary
Date: Mon, 17 Sep 2007 18:32:45 +0200

 Pete wrote:
 > Does anyone know if S=F8ren Schmidt's patch (repeated below) is safe to=
 
 > use, and if it is, when it will be commited?
 >
 > I'm also experiencing this problem, but I haven't dared to try the
 > patch yet.
 >
 >  =20
 It is safe to use, it fixes a real bug :)
 
 Its just me being way too busy that has held it back from asking commit=20
 approval..
 
 -S=F8ren
 > The patch:
 >
 > --- ata-all.c 23 Feb 2007 16:25:08 -0000 1.279
 > +++ ata-all.c 19 Aug 2007 09:25:58 -0000
 > @@ -738,7 +738,7 @@
 >
 >  =20
 >> atadev-flags &=3D ~ATA_D_48BIT_ACTIVE;
 >>    =20
 >
 > - if ((request->u.ata.lba >=3D ATA_MAX_28BIT_LBA ||
 > + if (((request->u.ata.lba + request->u.ata.count) >=3D ATA_MAX_28BIT_L=
 BA ||
 >  =20
 >> request-> u.ata.count256) &&
 >> atadev-param.support.command2 & ATA_SUPPORT_ADDRESS48) {
 >>    =20
 >
 > .
 >
 >  =20
 
 

From: Pete <xd888cc7541309c3f@f4n.org>
To: bug-followup@FreeBSD.org
Cc: sos@deepcore.dk
Subject: Re: kern/115572: [gbde] gbde partitions fail at 28bit/48bit LBA addressing boundary
Date: Mon, 17 Sep 2007 18:30:11 +0200

 Does anyone know if Sren Schmidt's patch (repeated below) is safe to
 use, and if it is, when it will be commited?
 
 I'm also experiencing this problem, but I haven't dared to try the
 patch yet.
 
 The patch:
 
 --- ata-all.c 23 Feb 2007 16:25:08 -0000 1.279
 +++ ata-all.c 19 Aug 2007 09:25:58 -0000
 @@ -738,7 +738,7 @@
 
 > atadev-flags &= ~ATA_D_48BIT_ACTIVE;
 
 - if ((request->u.ata.lba >= ATA_MAX_28BIT_LBA ||
 + if (((request->u.ata.lba + request->u.ata.count) >= ATA_MAX_28BIT_LBA ||
 > request-> u.ata.count256) &&
 > atadev-param.support.command2 & ATA_SUPPORT_ADDRESS48) {

From: Antony Mawer <gnats@mawer.org>
To: bug-followup@FreeBSD.org, gbradley@rocketmail.com
Cc: sos@freebsd.org
Subject: Re: kern/115572: [gbde] [patch] gbde partitions fail at 28bit/48bit
 LBA addressing boundary
Date: Tue, 17 Jun 2008 23:29:53 +1000

 I believe this patch is now present in both 6.3-RELEASE and 7.0-RELEASE, 
 so I believe this PR can probably be closed now. Relevant commits:
 
 FreeBSD 7.0
 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/ata/ata-all.c#rev1.280
 
 FreeBSD 6.3
 http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/ata/ata-all.c#rev1.252.2.12
 
 --Antony
State-Changed-From-To: open->closed 
State-Changed-By: gavin 
State-Changed-When: Mon Jun 23 19:14:13 UTC 2008 
State-Changed-Why:  
Close, this is fixed in 6.3 and 7.0 with the commits mentioned 
in the audit log. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=115572 
>Unformatted:
