From nobody@FreeBSD.org  Thu Feb 12 08:28:58 2009
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B3BAB106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 12 Feb 2009 08:28:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id A2A678FC19
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 12 Feb 2009 08:28:58 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id n1C8SvZZ039308
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 12 Feb 2009 08:28:57 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id n1C8SvxW039307;
	Thu, 12 Feb 2009 08:28:57 GMT
	(envelope-from nobody)
Message-Id: <200902120828.n1C8SvxW039307@www.freebsd.org>
Date: Thu, 12 Feb 2009 08:28:57 GMT
From: Peter Trifonov <petert@dcn.infos.ru>
To: freebsd-gnats-submit@FreeBSD.org
Subject: sendfile() sends corrupted data
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         131602
>Category:       kern
>Synopsis:       [libc] sendfile(2) sends corrupted data
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    gavin
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Feb 12 08:30:01 UTC 2009
>Closed-Date:    Sat Sep 24 03:59:34 UTC 2011
>Last-Modified:  Sat Sep 24 03:59:34 UTC 2011
>Originator:     Peter Trifonov
>Release:        7.1
>Organization:
Saint-Petersburg State Polytechnic University
>Environment:
FreeBSD dcn.research.dcn 7.1-RELEASE FreeBSD 7.1-RELEASE #1: Mon Jan  5 15:19:14 MSK 2009     petert-admin@dcn.research.dcn:/usr/obj/usr/src/sys/SERVER  i386
>Description:
The server hosts a large collection of files, which are stored on a local IDE hard disk. The files are available both via Apache 2.2.11 web server and Samba 3.0.32. Sometimes both Apache and Samba return to clients corrupted files. If the corrupted file is returned by Apache, all subsequent requests to this file from Samba also return corrupted file, and vice versa. The problem arises sporadically (but quite often) for different files larger than 1M. 
The problem is not persistent, i.e. some files are later delivered correctly. 
Disabling sendfile BOTH in Apache and Samba solves the problem.
>How-To-Repeat:
Put a file larger than 1M to a directory available both via apache and samba. Try to download it via both of them. Compare md5 checksums of the original and downloaded files. 
>Fix:


>Release-Note:
>Audit-Trail:
State-Changed-From-To: open->feedback 
State-Changed-By: gavin 
State-Changed-When: Thu Feb 12 18:47:16 UTC 2009 
State-Changed-Why:  
To submitter: given the seriousness of this issue, I suspect I'd 
remember if somebody else had seen this, but I don't believe any 
one else has reported this.  Therefore, to begin with at least I 
think we need to suspect it's something particular to your system. 
Could you please provide the output of "dmesg" and "pciconf -l", and 
"ifconfig -a" (with hidden IP addresses if you so wish). 
Also, are you able to provide an example of a corrupted and a 
non-corrupted file somewhere for comparison?  (If you are not 
willing to publically provide these but are happy to share them 
directly with me, feel free). 


Responsible-Changed-From-To: freebsd-bugs->gavin 
Responsible-Changed-By: gavin 
Responsible-Changed-When: Thu Feb 12 18:47:16 UTC 2009 
Responsible-Changed-Why:  
Track 

http://www.freebsd.org/cgi/query-pr.cgi?pr=131602 

From: Yoshihiro Ota <ota@j.email.ne.jp>
To: bug-followup@FreeBSD.org
Cc: petert@dcn.infos.ru, gavin@FreeBSD.org
Subject: Re: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Thu, 12 Feb 2009 22:45:22 -0500

 Gavin,
 
 There is another PR related to sendfile.2.
 http://www.freebsd.org/cgi/query-pr.cgi?pr=131602
 
 On tmpfs, data is always corrupted.  This may not
 be the case as Peter says it is stored on local IDE disk
 and says it happened randomly.
 
 Just FYI.
 
 Hiro

From: Gavin Atkinson <gavin@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Sat, 14 Feb 2009 18:22:53 +0000

 -------- Forwarded Message --------
 From: Peter Trifonov <petert@dcn.infos.ru>
 Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
 Date: Fri, 13 Feb 2009 10:35:11 +0300
 
 Dear Gavin,
 
 Unfortunately, I cannot reboot the server now, and dmesg is currently
 flooded 
 with messages absolutely irrelevant to the problem.
 
 
 dcn# pciconf -l
 hostb0@pci0:0:0:0:      class=0x060000 card=0x25808086 chip=0x25808086 rev=0x0e hdr=0x00
 vgapci0@pci0:0:2:0:     class=0x030000 card=0x25821043 chip=0x25828086 rev=0x0e hdr=0x00
 vgapci1@pci0:0:2:1:     class=0x038000 card=0x25821043 chip=0x27828086 rev=0x0e hdr=0x00
 none0@pci0:0:27:0:      class=0x040300 card=0x814e1043 chip=0x26688086 rev=0x03 hdr=0x00
 pcib1@pci0:0:28:0:      class=0x060400 card=0x00000000 chip=0x26608086 rev=0x03 hdr=0x01
 pcib2@pci0:0:28:1:      class=0x060400 card=0x00000000 chip=0x26628086 rev=0x03 hdr=0x01
 pcib3@pci0:0:28:2:      class=0x060400 card=0x00000000 chip=0x26648086 rev=0x03 hdr=0x01
 pcib4@pci0:0:28:3:      class=0x060400 card=0x00000000 chip=0x26668086 rev=0x03 hdr=0x01
 uhci0@pci0:0:29:0:      class=0x0c0300 card=0x80a61043 chip=0x26588086 rev=0x03 hdr=0x00
 uhci1@pci0:0:29:1:      class=0x0c0300 card=0x80a61043 chip=0x26598086 rev=0x03 hdr=0x00
 uhci2@pci0:0:29:2:      class=0x0c0300 card=0x80a61043 chip=0x265a8086 rev=0x03 hdr=0x00
 uhci3@pci0:0:29:3:      class=0x0c0300 card=0x80a61043 chip=0x265b8086 rev=0x03 hdr=0x00
 ehci0@pci0:0:29:7:      class=0x0c0320 card=0x80a61043 chip=0x265c8086 rev=0x03 hdr=0x00
 pcib5@pci0:0:30:0:      class=0x060401 card=0x00000000 chip=0x244e8086 rev=0xd3 hdr=0x01
 isab0@pci0:0:31:0:      class=0x060100 card=0x00000000 chip=0x26408086 rev=0x03 hdr=0x00
 atapci0@pci0:0:31:1:    class=0x01018a card=0x80a61043 chip=0x266f8086 rev=0x03 hdr=0x00
 atapci1@pci0:0:31:2:    class=0x01018f card=0x26011043 chip=0x26518086 rev=0x03 hdr=0x00
 none1@pci0:0:31:3:      class=0x0c0500 card=0x80a61043 chip=0x266a8086 rev=0x03 hdr=0x00
 fxp0@pci0:1:8:0:        class=0x020000 card=0x80f81043 chip=0x10648086 rev=0x03 hdr=0x00
 rl0@pci0:1:10:0:        class=0x020000 card=0x13031186 chip=0x13001186 rev=0x10 hdr=0x00
 re0@pci0:1:11:0:        class=0x020000 card=0x311a1385 chip=0x816910ec rev=0x10 hdr=0x00
 
 dcn# ifconfig -a
 fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
         options=8<VLAN_MTU>
         ether 00:13:d4:4a:a9:5f
         inet 10.0.204.1 netmask 0xffffff00 broadcast 10.0.204.255
         media: Ethernet autoselect (100baseTX <full-duplex>)
         status: active
 rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
         options=8<VLAN_MTU>
         ether 00:11:95:5c:b9:ff
         inet 10.0.205.1 netmask 0xffffff00 broadcast 10.0.205.255
         media: Ethernet autoselect (10baseT/UTP)
         status: active
 re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
         ether 00:14:6c:8c:2d:90
         inet 10.0.103.1 netmask 0xffffff00 broadcast 10.0.103.255
         inet 195.209.228.109 netmask 0xfffffff8 broadcast 195.209.228.111
         inet 195.19.213.82 netmask 0xffffffe0 broadcast 195.19.213.95
         inet 10.0.104.1 netmask 0xffffff00 broadcast 10.0.104.255
         inet 10.0.200.1 netmask 0xffffff00 broadcast 10.0.200.255
         inet 195.209.228.108 netmask 0xffffffff broadcast 195.209.228.108
         inet 195.19.213.81 netmask 0xffffffff broadcast 195.19.213.81
         media: Ethernet autoselect (1000baseTX <full-duplex>)
         status: active
 plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 mtu 1500
 lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
         inet 127.0.0.1 netmask 0xff000000 
 tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
         inet 10.0.201.1 --> 10.0.201.2 netmask 0xffffffff 
         Opened by PID 1067
 
 The problem was observed while accessing the server from a PC in
 10.0.103.0/24 network, which was also equipped with
 Gigabit Ethernet card. 
 
 Please see the attached files:
 
 1273.pdf  - the original (good) file. MD5= 592f254bad89156262d83ba0c95e07fc
 1273_apache.pdf - the file downloaded via Apache with EnableSendfile=on.
 MD5= 68b5abf85d6455736d2bf94a7761f925
 
 [ files are at http://people.freebsd.org/~gavin/PRs/131602/ ]
 
 MD5 computed at the server after accessing the file via Apache is also
 68b5abf85d6455736d2bf94a7761f925, i.e. the 
 memory-cached file is also corrupt. 
 
 I did not enable sendfile in Samba, since the problem seems not to be
 related to its interaction with Apache. 
 
 

From: Gavin Atkinson <gavin@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Sat, 14 Feb 2009 18:25:32 +0000

 -------- Forwarded Message --------
 From: Peter Trifonov <petert@dcn.infos.ru>
 To: 'Gavin Atkinson' <gavin@FreeBSD.org>
 Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
 Date: Fri, 13 Feb 2009 15:56:52 +0300
 
 
 Copyright (c) 1992-2009 The FreeBSD Project.
 Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
 FreeBSD is a registered trademark of The FreeBSD Foundation.
 FreeBSD 7.1-RELEASE #1: Mon Jan  5 15:19:14 MSK 2009
     petert-admin@dcn.research.dcn:/usr/obj/usr/src/sys/SERVER
 Timecounter "i8254" frequency 1193182 Hz quality 0
 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.51-MHz 686-class CPU)
   Origin =3D "GenuineIntel"  Id =3D 0xf43  Stepping =3D 3
 =20
 Features=3D0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,=
 MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=3D0x649d<SSE3,DTES64,MON,DS_CPL,EST,CNXT-ID,CX16,xTPR>
   AMD Features=3D0x20000000<LM>
   Logical CPUs per core: 2
 real memory  =3D 1065091072 (1015 MB)
 avail memory =3D 1033064448 (985 MB)
 ACPI APIC Table: <A M I  OEMAPIC >
 ioapic0 <Version 2.0> irqs 0-23 on motherboard
 kbd1 at kbdmux0
 cryptosoft0: <software crypto> on motherboard
 acpi0: <A M I OEMXSDT> on motherboard
 acpi0: [ITHREAD]
 acpi0: Power Button (fixed)
 acpi0: reservation of 0, a0000 (3) failed
 acpi0: reservation of 100000, 3f700000 (3) failed
 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 pci0: <ACPI PCI bus> on pcib0
 vgapci0: <VGA-compatible display> port 0x8800-0x8807 mem
 0xcfd80000-0xcfdfffff,0xd0000000-0xdfffffff,0xcfe80000-0xcfebffff irq 16 at device 2.0 on pci0
 agp0: <Intel 82915G (915G GMCH) SVGA controller> on vgapci0
 agp0: detected 7932k stolen memory
 agp0: aperture size is 256M
 vgapci1: <VGA-compatible display> mem 0xcfe00000-0xcfe7ffff at device 2.1 on pci0
 pci0: <multimedia> at device 27.0 (no driver attached)
 pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
 pci5: <ACPI PCI bus> on pcib1
 pcib2: <ACPI PCI-PCI bridge> at device 28.1 on pci0
 pci4: <ACPI PCI bus> on pcib2
 pcib3: <ACPI PCI-PCI bridge> at device 28.2 on pci0
 pci3: <ACPI PCI bus> on pcib3
 pcib4: <ACPI PCI-PCI bridge> at device 28.3 on pci0
 pci2: <ACPI PCI bus> on pcib4
 uhci0: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-A> port 0x8880-0x889f irq 23 at device 29.0 on pci0
 uhci0: [GIANT-LOCKED]
 uhci0: [ITHREAD]
 usb0: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-A> on uhci0
 usb0: USB revision 1.0
 uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
 uhub0: 2 ports with 2 removable, self powered
 uhci1: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-B> port 0x8c00-0x8c1f irq 19 at device 29.1 on pci0
 uhci1: [GIANT-LOCKED]
 uhci1: [ITHREAD]
 usb1: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-B> on uhci1
 usb1: USB revision 1.0
 uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
 uhub1: 2 ports with 2 removable, self powered
 uhci2: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-C> port 0x9000-0x901f irq 18 at device 29.2 on pci0
 uhci2: [GIANT-LOCKED]
 uhci2: [ITHREAD]
 usb2: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-C> on uhci2
 usb2: USB revision 1.0
 uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
 uhub2: 2 ports with 2 removable, self powered
 uhci3: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-D> port 0x9080-0x909f irq 16 at device 29.3 on pci0
 uhci3: [GIANT-LOCKED]
 uhci3: [ITHREAD]
 usb3: <Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-D> on uhci3
 usb3: USB revision 1.0
 uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
 uhub3: 2 ports with 2 removable, self powered
 ehci0: <Intel 82801FB (ICH6) USB 2.0 controller> mem 0xcfeffc00-0xcfefffff irq 23 at device 29.7 on pci0
 ehci0: [GIANT-LOCKED]
 ehci0: [ITHREAD]
 usb4: EHCI version 1.0
 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
 usb4: <Intel 82801FB (ICH6) USB 2.0 controller> on ehci0
 usb4: USB revision 2.0
 uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
 uhub4: 8 ports with 8 removable, self powered
 pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
 pci1: <ACPI PCI bus> on pcib5
 fxp0: <Intel 82562EZ (ICH6)> port 0xac00-0xac3f mem 0xcfffe000-0xcfffefff irq 20 at device 8.0 on pci1
 miibus0: <MII bus> on fxp0
 inphy0: <i82562ET 10/100 media interface> PHY 1 on miibus0
 inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 fxp0: Ethernet address: 00:13:d4:4a:a9:5f
 fxp0: [ITHREAD]
 rl0: <D-Link DFE-530TX+ 10/100BaseTX> port 0xa400-0xa4ff mem 0xcffff800-0xcffff8ff irq 22 at device 10.0 on pci1
 miibus1: <MII bus> on rl0
 rlphy0: <RealTek internal media interface> PHY 0 on miibus1
 rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
 rl0: Ethernet address: 00:11:95:5c:b9:ff
 rl0: [ITHREAD]
 re0: <RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet> port 0xa800-0xa8ff mem 0xcffffc00-0xcffffcff irq 21 at device 11.0 on pci1
 re0: Chip rev. 0x04000000
 re0: MAC rev. 0x00000000
 miibus2: <MII bus> on re0
 rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus2
 rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
 re0: Ethernet address: 00:14:6c:8c:2d:90
 re0: [FILTER]
 isab0: <PCI-ISA bridge> at device 31.0 on pci0
 isa0: <ISA bus> on isab0
 atapci0: <Intel ICH6 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
 ata0: <ATA channel 0> on atapci0
 ata0: [ITHREAD]
 ata1: <ATA channel 1> on atapci0
 ata1: [ITHREAD]
 atapci1: <Intel ICH6 SATA150 controller> port 0x9c00-0x9c07,0x9880-0x9883,0x9800-0x9807,0x9480-0x9483,0x9400-0x940f irq 19 at device 31.2 on pci0
 atapci1: [ITHREAD]
 ata2: <ATA channel 0> on atapci1
 ata2: [ITHREAD]
 ata3: <ATA channel 1> on atapci1
 ata3: [ITHREAD]
 pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
 acpi_button0: <Power Button> on acpi0
 fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
 fdc0: [FILTER]
 fd0: <1440-KB 3.5" drive> on fdc0 drive 0
 atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
 atkbd0: <AT Keyboard> irq 1 on atkbdc0
 kbd0 at atkbd0
 atkbd0: [GIANT-LOCKED]
 atkbd0: [ITHREAD]
 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
 sio0: type 16550A
 sio0: [FILTER]
 cpu0: <ACPI CPU> on acpi0
 ACPI Warning (tbutils-0243): Incorrect checksum in table [OEMB] -  52, should be 45 [20070320]
 est0: <Enhanced SpeedStep Frequency Control> on cpu0
 p4tcc0: <CPU Frequency Thermal Control> on cpu0
 pmtimer0 on isa0
 sc0: <System console> at flags 0x100 on isa0
 sc0: VGA <16 virtual consoles, flags=3D0x300>
 vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
 ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
 ppbus0: <Parallel port bus> on ppc0
 ppbus0: [ITHREAD]
 plip0: <PLIP network interface> on ppbus0
 plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
 lpt0: <Printer> on ppbus0
 lpt0: Interrupt-driven port
 ppi0: <Parallel I/O> on ppbus0
 ppc0: [GIANT-LOCKED]
 ppc0: [ITHREAD]
 sio1: configured irq 3 not in bitmap of probed irqs 0
 sio1: port may not be enabled
 Timecounter "TSC" frequency 2992511700 Hz quality 800
 Timecounters tick every 1.000 msec
 IPsec: Initialized Security Association Processing.
 ipfw2 initialized, divert enabled, nat loadable, rule-based forwarding
 enabled, default to deny, logging disabled
 acd0: CDROM <HL-DT-ST CD-ROM GCR-8522B/1.03> at ata0-master PIO4
 ad4: 152627MB <SAMSUNG HD160JJ WU100-31> at ata2-master SATA150
 ad6: 152627MB <SAMSUNG HD160JJ WU100-31> at ata3-master SATA150
 Trying to mount root from ufs:/dev/ad4s1a
 re0: link state changed to UP
 
 
 > Which of these interfaces will the file have been served from?
 > (Although to be honest I no longer suspect the network interface is to
 > blame, it may be a useful datapoint).
 
 The file is served via re0. I was not able to reproduce the problem via rl0.

 > Thanks for them - the file corruption is very specific, and consists of
 > MIME headers overwriting data.  Do you mind if I share those files with
 > somebody else who may be able to investigate further?
 
 Surely. The original file is publicly available=20
 (http://www.math.uni-hamburg.de/home/diestel/books/graph.theory/) , so you
 can do whatever you want with it.
 I can try to collect additional samples, if necessary.=20
 
State-Changed-From-To: feedback->feedback  
State-Changed-By: gavin 
State-Changed-When: Tue Feb 17 06:10:03 UTC 2009 
State-Changed-Why:  
To submitter: a couple more questions: 
- Has this server run any previous versions of FreeBSD without 
showing this issue?  i.e. do you know if this problem has been 
introduced since 7.0 or similar? 
- Is the file corruption always identical?  e.g. If you serve 
the file, reboot then serve the file again, will they have 
the same MD5 sum? 
- You said that the same corruption is seen while serving with 
Samba too?  Is it possible to supply a copy of the same file 
corrupted with samba (with no apache involvement)?  The 
corruption seen in the supplied files does seem to be http- 
related. 
- What user/group does httpd run as? 

Thanks 

http://www.freebsd.org/cgi/query-pr.cgi?pr=131602 

From: Gavin Atkinson <gavin@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Tue, 17 Feb 2009 12:03:20 +0000

 -------- Forwarded Message --------
 From: Peter Trifonov <petert@dcn.infos.ru>
 Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
 Date: Tue, 17 Feb 2009 12:17:11 +0300
 
 Dear Gavin,
 
 1. The problem was discovered after the server was upgraded from version
 6.3.
 2. I have tried several times, but was not able to reproduce the problem
 neither with Apache, nor with Samba.
    However, I do remember that the same file downloaded some time ago via
 the Squid proxy, which is running on 
    the same server, had different MD5 sum. Unfortunately, I did not keep it.
 I will keep the server running with sendfile
    enabled both in Apache and in Samba, and let you know immediately when
 the problem re-appears. 
 3. httpd runs with uid=gid=80 (www).
 
 With best regards,
 P. Trifonov

From: Gavin Atkinson <gavin@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc:  
Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Tue, 17 Feb 2009 17:13:21 +0000

 -------- Forwarded Message --------
 From: Peter Trifonov <petert@dcn.infos.ru>
 Subject: RE: kern/131602: [libc] sendfile(2) sends corrupted data
 Date: Tue, 17 Feb 2009 19:51:14 +0300
 
 I have obtained another version of the corrupted file via Apache. It again
 contains some http-related garbage. 
 Samba returns the same corrupted file if it is accessed immediately after it
 was downloaded via Apache. However, 
 Samba returns good file if the server file cache is purged by downloading
 some huge file. Samba serves many other files, 
 which are not accessible via Apache, and no corruption was observed for
 them. So the problem seems to be related to Apache. 
 
 [ Second corrupt example file at http://people.freebsd.org/~gavin/PRs/131602/ ]

From: Mari Kotlov <mkotlov@gmail.com>
To: bug-followup@freebsd.org, petert@dcn.infos.ru
Cc:  
Subject: Re: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Thu, 2 Apr 2009 18:04:44 -0400

 --00221532c5b4a561080466999e83
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: 7bit
 
 Hi Gavin,I believe I also reproduced the same issue with sendfile and large
 files (some get corrupted, others do not). The files I have problems with
 are all at least 1MB in size. My system was originally at 6.3-RC2 and I did
 not experience the same problem with sendfile.
 However, at 7.1, I pretty consistently get a problem.
 I am transferring zip files from FreeBSD system to WinXP on a non-blocking
 socket via HTTP. I could also capture wireshark trace if it is helpful. In
 all cases, sendfile is called mutliple times and indicates a successful end
 of transmission (e.g. return value 0). I have no problems transferring
 smaller zip files, e.g. 130KB.
 Please let me know if I can be of any assistance - e.g. provide corrupted
 files, etc.
 Thanks,
 Mari.
 
 --00221532c5b4a561080466999e83
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Hi Gavin,<div>I believe I also reproduced the same issue with sendfile and =
 large files (some get corrupted, others do not). The files I have problems =
 with are all at least 1MB in size. My system was originally at 6.3-RC2 and =
 I did not experience the same problem with sendfile.</div>
 
 <div>However, at 7.1, I pretty consistently get a problem.</div><div>I am t=
 ransferring zip files from FreeBSD system to WinXP on a non-blocking socket=
  via HTTP. I could also capture wireshark trace if it is helpful. In all ca=
 ses, sendfile is called mutliple times and indicates a successful end of tr=
 ansmission (e.g. return value 0). I have no problems transferring smaller z=
 ip files, e.g. 130KB.</div>
 <div>Please let me know if I can be of any assistance - e.g. provide corrup=
 ted files, etc.</div><div>Thanks,<br>Mari.</div>
 
 --00221532c5b4a561080466999e83--

From: Gavin Atkinson <gavin@FreeBSD.org>
To: Mari Kotlov <mkotlov@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Wed, 08 Apr 2009 12:20:48 +0100

 On Thu, 2009-04-02 at 22:30 +0000, Mari Kotlov wrote:
 >  Hi Gavin,I believe I also reproduced the same issue with sendfile and large
 >  files (some get corrupted, others do not). The files I have problems with
 >  are all at least 1MB in size. My system was originally at 6.3-RC2 and I did
 >  not experience the same problem with sendfile.
 >  However, at 7.1, I pretty consistently get a problem.
 >  I am transferring zip files from FreeBSD system to WinXP on a non-blocking
 >  socket via HTTP. I could also capture wireshark trace if it is helpful. In
 >  all cases, sendfile is called mutliple times and indicates a successful end
 >  of transmission (e.g. return value 0). I have no problems transferring
 >  smaller zip files, e.g. 130KB.
 >  Please let me know if I can be of any assistance - e.g. provide corrupted
 >  files, etc.
 
 Thanks for that, it's a useful datapoint.  Are you running on i386 or
 amd64?  How much memory does the system have?  Also, can you please
 compare the corrupted files you are seeing with the uncorrupted versions
 and confirm that you, too, are seeing HTTP headers injected into the
 data?
 
 Thanks,
 
 Gavin

From: Mari Kotlov <mkotlov@gmail.com>
To: Gavin Atkinson <gavin@freebsd.org>
Cc: bug-followup@freebsd.org
Subject: Re: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Wed, 8 Apr 2009 11:36:46 -0400

 --0003255756163b15cd04670ce690
 Content-Type: text/plain; charset=ISO-8859-1
 Content-Transfer-Encoding: 7bit
 
 Hi Gavin,I am running on i386. Yep, I did a binary compare and I definitely
 see HTTP headers injected into my zip file. I am using DHCP, local home
 network (behind a firewall). Also, if it helps, using 0 as a sendfile option
 for number of bytes to transmit does not work if HTTP headers are included,
 although this should probably be a separate bug... Or perhaps zero is not
 intended to work if I am using headers or trailers? I have wireshark trace,
 diff output of two binary files (as a hex dump) and both the corrupted and
 intact versions of the file I am transmitting. Please let me know if you
 need any of these.
 
 Here is the memory information:
 $> grep memory /var/run/dmesg.boot
 real memory  = 1063804928 (1014 MB)
 avail memory = 1018863616 (971 MB)
 agp0: detected 7932k stolen memory
 
 $> sysctl -a|grep mem
 vm.kmem_size_scale: 3
 vm.kmem_size_max: 335544320
 vm.kmem_size_min: 0
 vm.kmem_size: 335544320
 vfs.ufs.dirhash_mem: 2094138
 vfs.ufs.dirhash_maxmem: 2097152
 debug.fwmem_debug: 0
 hw.physmem: 1042280448
 hw.usermem: 846311424
 hw.realmem: 1063804928
 hw.firewire.fwmem.speed: 2
 hw.firewire.fwmem.eui64_lo: 0
 hw.firewire.fwmem.eui64_hi: 0
 hw.cbb.start_memory: 2281701376
 hw.pci.host_mem_start: 2147483648
 p1003_1b.memlock: 0
 p1003_1b.memlock_range: 0
 p1003_1b.memory_protection: 0
 p1003_1b.shared_memory_objects: 1
 
 --0003255756163b15cd04670ce690
 Content-Type: text/html; charset=ISO-8859-1
 Content-Transfer-Encoding: quoted-printable
 
 Hi Gavin,<div>I am running on i386. Yep, I did a binary compare and I defin=
 itely see HTTP headers injected into my zip file. I am using DHCP, local ho=
 me network (behind a firewall). Also, if it helps, using 0 as a sendfile op=
 tion for number of bytes to transmit does not work if HTTP headers are incl=
 uded, although this should probably be a separate bug... Or perhaps zero is=
  not intended to work if I am using headers or trailers? I have wireshark t=
 race, diff output of two binary files (as a hex dump) and both the corrupte=
 d and intact versions of the file I am transmitting. Please let me know if =
 you need any of these.<br>
 <br>Here is the memory information:<br>$&gt; grep memory /var/run/dmesg.boo=
 t<br>real memory=A0 =3D 1063804928 (1014 MB)<br>avail memory =3D 1018863616=
  (971 MB)<br>agp0: detected 7932k stolen memory<br><br>$&gt; sysctl -a|grep=
  mem<br>
 vm.kmem_size_scale: 3<br>vm.kmem_size_max: 335544320<br>vm.kmem_size_min: 0=
 <br>vm.kmem_size: 335544320<br>vfs.ufs.dirhash_mem: 2094138<br>vfs.ufs.dirh=
 ash_maxmem: 2097152<br>debug.fwmem_debug: 0<br>hw.physmem: 1042280448<br>
 hw.usermem: 846311424<br>hw.realmem: 1063804928<br>hw.firewire.fwmem.speed:=
  2<br>hw.firewire.fwmem.eui64_lo: 0<br>hw.firewire.fwmem.eui64_hi: 0<br>hw.=
 cbb.start_memory: 2281701376<br>hw.pci.host_mem_start: 2147483648<br>p1003_=
 1b.memlock: 0<br>
 p1003_1b.memlock_range: 0<br>p1003_1b.memory_protection: 0<br>p1003_1b.shar=
 ed_memory_objects: 1<br><br></div>
 
 --0003255756163b15cd04670ce690--

From: Gavin Atkinson <gavin@FreeBSD.org>
To: Mari Kotlov <mkotlov@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Thu, 09 Apr 2009 12:17:13 +0100

 On Wed, 2009-04-08 at 12:39 -0400, Mari Kotlov wrote:
 > Hi Gavin,
 > Below is what ifconfig shows me. I am not really sure how I can find
 > out which driver I am using. There is no loader.conf and nothing
 > custom in rc.conf. I don't have a sample program I can readily send
 > you, the code that I modified is part of a much larger existing server
 > application. It would be a pain to extract the code from it, but I'll
 > be happy to send you snippets of my code having to do with sendfile. I
 > have not tried Apache (although I could probably install it on my
 > FreeBSD machine if needed). Attached is the diff between two hex dump
 > files (I was transmitting curl-7.19.3.zip.
 > 
 > bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
 > 1500
 > options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
 > media: Ethernet autoselect (100baseTX <full-duplex>)
 
 That's sufficient, thanks.  All other reports so far were using Realtek
 network cards, so it's good to be able to rule out the type of network
 interface as the cause.
 
 I'm going to carry on trying to reproduce this issue.
 
 Thanks,
 
 Gavin
 

From: "Ming Fu" <Ming.Fu@watchguard.com>
To: <bug-followup@FreeBSD.org>
Cc:  
Subject: Re: kern/131602: [libc] sendfile(2) sends corrupted data
Date: Wed, 30 Jun 2010 12:21:28 -0700

 This is a multi-part message in MIME format.
 
 ------_=_NextPart_001_01CB1889.710D73E5
 Content-Type: text/plain;
 	charset="us-ascii"
 Content-Transfer-Encoding: quoted-printable
 
 Hi,=20
 
 =20
 
 I hit this problem on my software as well. I was writing a program to
 encode a large file in http chunk encoding format and send it to a tcp
 socket.
 
 My system is 6.3 release.
 
 =20
 
 So basically the program is doing something like, the file is a 8M text
 file containing only alpha characters, the file is opened read only.
 
                =20
 
                 while (displacement < file_size) {
 
                                 write (sock, "800\r\n", 5); /* 2k in hex
 */
 
                                 sendfile(file, sock, displacement, 2048,
 NULL, &sbytes, 0);
 
                                 displacement +=3D sbytes;
 
                 }
 
 =20
 
 I found the "800\r\n" was wrote into the file in various locations. The
 file size did not change. I can't tell if the file was corrupted in any
 other way. But the original file does not contain any numeric
 characters.
 
 =20
 
 I tried to use the sf_hdtr.headers to write the chunk length encoding
 instead of the separate write. The result is the same, some 800\r\n got
 into the file.=20
 
 =20
 
 I started to make a copy of the file every time sendfile() returns, so
 after the program finishes, I got a large collection of the snapshots of
 the file I was sending. I can tell the file started to have 800\r\n in
 them after about 20K was sent. There could be up to 10 "800\r\n" got
 added to the file when the program is done.
 
 =20
 
 Hope this can help reproduce the problem.
 
 =20
 
 Ming
 
 =20
 
 =20
 
 =20
 
 
 ------_=_NextPart_001_01CB1889.710D73E5
 Content-Type: text/html;
 	charset="us-ascii"
 Content-Transfer-Encoding: quoted-printable
 
 <html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
 xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
 xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
 xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
 xmlns=3D"http://www.w3.org/TR/REC-html40">
 
 <head>
 <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
 charset=3Dus-ascii">
 <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
 <style>
 <!--
  /* Font Definitions */
  @font-face
 	{font-family:"Cambria Math";
 	panose-1:2 4 5 3 5 4 6 3 2 4;}
 @font-face
 	{font-family:Calibri;
 	panose-1:2 15 5 2 2 2 4 3 2 4;}
  /* Style Definitions */
  p.MsoNormal, li.MsoNormal, div.MsoNormal
 	{margin:0cm;
 	margin-bottom:.0001pt;
 	font-size:11.0pt;
 	font-family:"Calibri","sans-serif";}
 a:link, span.MsoHyperlink
 	{mso-style-priority:99;
 	color:blue;
 	text-decoration:underline;}
 a:visited, span.MsoHyperlinkFollowed
 	{mso-style-priority:99;
 	color:purple;
 	text-decoration:underline;}
 span.EmailStyle17
 	{mso-style-type:personal-compose;
 	font-family:"Calibri","sans-serif";
 	color:windowtext;}
 .MsoChpDefault
 	{mso-style-type:export-only;}
 @page Section1
 	{size:612.0pt 792.0pt;
 	margin:72.0pt 72.0pt 72.0pt 72.0pt;}
 div.Section1
 	{page:Section1;}
 -->
 </style>
 <!--[if gte mso 9]><xml>
  <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
 </xml><![endif]--><!--[if gte mso 9]><xml>
  <o:shapelayout v:ext=3D"edit">
   <o:idmap v:ext=3D"edit" data=3D"1" />
  </o:shapelayout></xml><![endif]-->
 </head>
 
 <body lang=3DEN-CA link=3Dblue vlink=3Dpurple>
 
 <div class=3DSection1>
 
 <p class=3DMsoNormal>Hi, <o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>I hit this problem on my software as well. I was =
 writing a
 program to encode a large file in http chunk encoding format and send it =
 to a
 tcp socket.<o:p></o:p></p>
 
 <p class=3DMsoNormal>My system is 6.3 release.<o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>So basically the program is doing something like, =
 the file
 is a 8M text file containing only alpha characters, the file is opened =
 read
 only.<o:p></o:p></p>
 
 <p =
 class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <o:p></o:p></p>
 
 <p =
 class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; while (displacement &lt; file_size) =
 {<o:p></o:p></p>
 
 <p =
 class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp; write (sock,
 &quot;800\r\n&quot;, 5); /* 2k in hex */<o:p></o:p></p>
 
 <p =
 class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp; sendfile(file, sock,
 displacement, 2048, NULL, &amp;sbytes, 0);<o:p></o:p></p>
 
 <p =
 class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
 bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; displacement =
 +=3D sbytes;<o:p></o:p></p>
 
 <p =
 class=3DMsoNormal>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
 nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>I found the &quot;800\r\n&quot; was wrote into the =
 file in
 various locations. The file size did not change. I can't tell if the =
 file was corrupted
 in any other way. But the original file does not contain any numeric =
 characters.<o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>I tried to use the sf_hdtr.headers to write the =
 chunk length
 encoding instead of the separate write. The result is the same, some =
 800\r\n
 got into the file. <o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>I started to make a copy of the file every time =
 sendfile()
 returns, so after the program finishes, I got a large collection of the =
 snapshots
 of the file I was sending. I can tell the file started to have 800\r\n =
 in them
 after about 20K was sent. There could be up to 10 &quot;800\r\n&quot; =
 got added
 to the file when the program is done.<o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>Hope this can help reproduce the =
 problem.<o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal>Ming<o:p></o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
 
 </div>
 
 </body>
 
 </html>
 
 ------_=_NextPart_001_01CB1889.710D73E5--
State-Changed-From-To: feedback->closed 
State-Changed-By: eadler 
State-Changed-When: Sat Sep 24 03:58:49 UTC 2011 
State-Changed-Why:  
security-officer thinks this was fixed via FreeBSD-SA-10:07.mbuf;
please let us know if this can be replicated after applying that patch

htti://www.freebsd.org/cgi/query-pr.cgi?pr=131602 
>Unformatted:
