From hno@squid-cache.org  Fri Apr 25 21:35:23 2008
Return-Path: <hno@squid-cache.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D46D2106566B
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Apr 2008 21:35:23 +0000 (UTC)
	(envelope-from hno@squid-cache.org)
Received: from squid-cache.org (squid-cache.org [12.160.37.9])
	by mx1.freebsd.org (Postfix) with ESMTP id D467C8FC0A
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Apr 2008 21:35:18 +0000 (UTC)
	(envelope-from hno@squid-cache.org)
Received: from squid-cache.org (localhost [127.0.0.1])
	by squid-cache.org (8.14.2/8.14.0) with ESMTP id m3PLHrdl086353
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Apr 2008 15:17:53 -0600 (MDT)
	(envelope-from hno@squid-cache.org)
Received: (from hno@localhost)
	by squid-cache.org (8.14.2/8.14.2/Submit) id m3PLHr7J086352;
	Fri, 25 Apr 2008 15:17:53 -0600 (MDT)
	(envelope-from hno)
Message-Id: <200804252117.m3PLHr7J086352@squid-cache.org>
Date: Fri, 25 Apr 2008 15:17:53 -0600 (MDT)
From: Henrik Nordstrom <hno@squid-cache.org>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: Suspected sendfile data corruption
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         123095
>Category:       kern
>Synopsis:       [libc] sendfile(2): Suspected sendfile data corruption
>Confidential:   no
>Severity:       critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Apr 25 21:40:01 UTC 2008
>Closed-Date:    Thu Aug 05 08:23:09 UTC 2010
>Last-Modified:  Thu Aug 05 08:23:09 UTC 2010
>Originator:     Henrik Nordstrom
>Release:        FreeBSD 6.3-STABLE i386
>Organization:
Squid HTTP Proxy project
>Environment:
System: FreeBSD squid-cache.org 6.3-STABLE FreeBSD 6.3-STABLE #0: Tue Apr 8 10:08:24 MDT 2008 root@squid-cache.org:/usr/obj/usr/src/sys/SQUID-CACHE i386


>Description:

We suspect sendfile is corrupting files on the server

We have a bzr repository served read-only using Apache, and on certain
bzr accesses the repository files gets currupted with parts of the HTTP
headers send by Apache. Apache do NOT have write permission to these files, and
neither size or modification date gets changed when this corruption occurs.

bzr uses a lot of ranged GET requests, which by Apache gets translated
to sendfile() with a header and a range of the requested file.

Do not yet know if the corruption permanent on disk, or in memory only.

>How-To-Repeat:

Not yet sure how to repeat it outside our server. But the repository data
(both clean and corrputed) and Apache configuration details can be provided
on request.

It shows up on the server when trying to get a checkout of the Squid source
tree from the server using bzr

  bzr co --lightweght http://www.squid-cache.org/bzr/squid3/trunk

runs for a while, then bzr complains about an invalid record '\r' and the
files have been corrupted on the server.
>Fix:
>Release-Note:
>Audit-Trail:

From: Nate Eldredge <neldredge@math.ucsd.edu>
To: bug-followup@FreeBSD.org, hno@squid-cache.org
Cc:  
Subject: Re: kern/123095: [libc] sendfile(2): Suspected sendfile data corruption
Date: Mon, 6 Oct 2008 19:08:30 -0700 (PDT)

 Hi,
 
 I've been looking at another sendfile related bug (127213) and wondering 
 if this one is related.  Can you tell me the type of filesystem where the 
 corrupted files are located?
 
 Also, it would be interesting to know the following, if you can test 
 them:
 
 1. Does it still happen on newer versions?  7.0-RELEASE for example.
 
 2. If you unmount and remount the filesystem, do the files stay corrupted?
 
 3. What happens if the filesystem is mounted read-only?
 
 4. What happens if you mount with -o sync ?
 
 -- 
 
 Nate Eldredge
 neldredge@math.ucsd.edu

From: Henrik Nordstrom <hno@squid-cache.org>
To: Nate Eldredge <neldredge@math.ucsd.edu>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/123095: [libc] sendfile(2): Suspected sendfile data
	corruption
Date: Tue, 07 Oct 2008 12:27:50 +0200

 --=-WCRlk/FBae9i/bTPCY/u
 Content-Type: text/plain; charset=utf-8
 Content-Transfer-Encoding: quoted-printable
 
 On m=C3=A5n, 2008-10-06 at 19:08 -0700, Nate Eldredge wrote:
 > Hi,
 >=20
 > I've been looking at another sendfile related bug (127213) and wondering=20
 > if this one is related.  Can you tell me the type of filesystem where the=
 =20
 > corrupted files are located?
 
 Normal ufs with softupdates enabled.
 
 > Also, it would be interesting to know the following, if you can test=20
 > them:
 >=20
 > 1. Does it still happen on newer versions?  7.0-RELEASE for example.
 
 Can't test easily.
 
 Currently at 6.3-STABLE. /boot/kernel dated 8 April.
 
 > 2. If you unmount and remount the filesystem, do the files stay corrupted=
 ?
 
 Tested using a mdconfig virtual device with a file as backing store and
 then it stayed corrupted, even after removing the md device and creating
 it again to the backing store file. But I am not sure this says much.=20
 
 Hmm.. tried again to umount/mount the filesystem after disabling
 softupdates, and now that does seem to clear the error. Confused. Must
 have done something wrong the first time.. Now it's very consistent
 about clearing the error on umount/mount, no matter if soft-updates is
 enabled or not.
 
 > 3. What happens if the filesystem is mounted read-only?
 
 Still corrupted.
 
 > 4. What happens if you mount with -o sync ?
 
 The corruption still happens. In fact it even happens earlier most of
 the time (or at least bzr notices it earlier).
 
 Also tried without soft-updates and the error is the same.
 
 All tests done with a mdconfig device with a file as backing store.
 Can't create a new real slice to test on, but suspect the results would
 be the same.
 
 The bug is a little elusive. Running the same actions do not always
 produce the same result, and all attempts in creating a isolated
 testcase has failed so far. Well, the bug did manifest itself once in a
 created test case (replay of a specific request to Apache), but never
 again in 20 or so attempts..
 
 The full bzr checkout run always makes the problem show up however, but
 not always in the same files or locations. The sequence of requests sent
 to Apache by bzr is the same in each run.
 
 Regards
 Henrik
 
 --=-WCRlk/FBae9i/bTPCY/u
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Description: This is a digitally signed message part
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.7 (GNU/Linux)
 
 iQCVAwUASOs5okNPQ5Kbx8daAQJ75wP/WsR8NhTtQvSVyu/gDMAK1QCgvOfyRGiU
 AVmuCqVUj2boHEYfCHfgHFGUx5pKLq6IF7ulxP6+f9JT303hgwI7KQYhs2ps21WB
 5TXYK34LkXCDzNxsseGVfJv95AZVtk7GQSZGncKHqCtHn2fPZGRY5xPV9BqtSPfu
 6IYbcodRVYE=
 =tvim
 -----END PGP SIGNATURE-----
 
 --=-WCRlk/FBae9i/bTPCY/u--
 

From: Henrik Nordstrom <hno@squid-cache.org>
To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-bugs@FreeBSD.org
Cc:  
Subject: Re: kern/123095: Suspected sendfile data corruption
Date: Wed, 08 Oct 2008 13:30:25 +0200

 --=-/7S+v8SQ9/uYGR7R21E8
 Content-Type: text/plain
 Content-Transfer-Encoding: quoted-printable
 
 Disabling sendfile support in Apache do make the problem go away
 
 EnableSendfile off
 
 Regards
 Henrik
 
 --=-/7S+v8SQ9/uYGR7R21E8
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Description: This is a digitally signed message part
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.7 (GNU/Linux)
 
 iQCVAwUASOyZzkNPQ5Kbx8daAQJZxAP8D9U5OI10XYjySG5JhEujCT1aTPF/gPTh
 FSQmuglmwBrZMqnZ7oxeNfsvZ89T4hoKdj4KIYZT23eiZJ9a98DHoOWJeDSb9xXH
 6/C697LjrkWJ73xq3lavUPdpjjp/1fJ2Aqkd/0tpCK6zD8lTf30TSRXRvJOrFkw1
 fJyRtI6rlzw=
 =VjTt
 -----END PGP SIGNATURE-----
 
 --=-/7S+v8SQ9/uYGR7R21E8--
 
State-Changed-From-To: open->closed 
State-Changed-By: kib 
State-Changed-When: Thu Aug 5 08:21:18 UTC 2010 
State-Changed-Why:  
It is believed that r209964 fixes the issue. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=123095 
>Unformatted:
