From nobody@www.freebsd.org  Fri May 24 05:27:35 2002
Return-Path: <nobody@www.freebsd.org>
Received: from nwww.freebsd.org (www.FreeBSD.org [216.136.204.117])
	by hub.freebsd.org (Postfix) with ESMTP id 912D337B404
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 24 May 2002 05:27:34 -0700 (PDT)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by nwww.freebsd.org (8.12.2/8.12.2) with ESMTP id g4OCRYhG000189
	for <freebsd-gnats-submit@FreeBSD.org>; Fri, 24 May 2002 05:27:34 -0700 (PDT)
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.12.2/8.12.2/Submit) id g4OCRYwD000188;
	Fri, 24 May 2002 05:27:34 -0700 (PDT)
Message-Id: <200205241227.g4OCRYwD000188@www.freebsd.org>
Date: Fri, 24 May 2002 05:27:34 -0700 (PDT)
From: Greg Troxel <gdt@ir.bbn.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: soreceive fails to maintain invariant on UDP sockets if uiomove fails
X-Send-Pr-Version: www-1.0

>Number:         38495
>Category:       kern
>Synopsis:       soreceive fails to maintain invariant on UDP sockets if uiomove fails
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    bms
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri May 24 05:30:03 PDT 2002
>Closed-Date:    Sat Feb 03 04:01:44 GMT 2007
>Last-Modified:  Sat Feb  3 04:10:17 GMT 2007
>Originator:     Greg Troxel
>Release:        RELENG_4 as of 20020523 or so
>Organization:
BBN Technologies
>Environment:
>Description:
Problem observed on NetBSD, and on inspection, FreeBSD has the same
code and the same problem.  Kernel will likely hit KASSERT at
sys/kern/uipc_socket.c line 774 if uiomove fails on recvfrom on a UDP
socket.
See NetBSD PR bin/16990, at
http://www.NetBSD.org/cgi-bin/query-pr-single.pl?number=16990
>How-To-Repeat:
See NetBSD PR.
Basically, call recvfrom with a NULL pointer to data, and then call it
again.
>Fix:
Merge patch from NetBSD PR.
>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->bms 
Responsible-Changed-By: bms 
Responsible-Changed-When: Fri Jun 18 03:42:31 GMT 2004 
Responsible-Changed-Why:  
I'll look at this 

http://www.freebsd.org/cgi/query-pr.cgi?pr=38495 

From: Bruce M Simpson <bms@spc.org>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/38495: soreceive fails to maintain invariant on UDP sockets if uiomove fails
Date: Wed, 23 Jun 2004 03:53:44 +0100

 Code inspection suggests that the issue is present in -CURRENT,
 but have not produced a test case.
 
 ZERO_COPY_SOCKETS appeared at first to complicate the situation, but
 this is in fact not the case.

From: James Juran <James.Juran@baesystems.com>
To: gdt@ir.bbn.com, bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/38495: soreceive fails to maintain invariant on UDP
	sockets if uiomove fails
Date: Mon, 12 Sep 2005 15:34:50 -0400

 --=-v+hKfXmdKd8qhWBMoQx4
 Content-Type: text/plain
 Content-Transfer-Encoding: quoted-printable
 
 This test case shows the problem on 5.4-RELEASE when the kernel is
 compiled with options INVARIANTS and options INVARIANT_SUPPORT.  On an
 unfixed system it will trigger a panic().  With the fix I'll send in the
 next message, the second recvfrom will block.  Unfortunately I don't
 have a -STABLE or -CURRENT system to test this on, but it does appear
 that the problem is still present in HEAD.
 
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
 #include <sys/socket.h>
 #include <sys/un.h>
 #include <unistd.h>
 
 #define TEST_SOCKET "/tmp/test_socket"
 
 int main(void)
 {
     int s;
     struct sockaddr_un un;
     pid_t pid;
 
     s =3D socket(PF_LOCAL, SOCK_DGRAM, 0);
     if (s =3D=3D -1)
     {
 	perror("socket");
 	exit(1);
     }
 
     memset(&un, 0, sizeof(un));
     un.sun_family =3D AF_LOCAL;
     unlink(TEST_SOCKET);
     strcpy(un.sun_path, TEST_SOCKET);
     if (bind(s, (struct sockaddr *)&un, sizeof(un)) =3D=3D -1)
     {
 	perror("bind");
 	exit(1);
     }
 
     pid =3D fork();
     if (pid =3D=3D -1)
     {
 	perror("fork");
 	exit(1);
     }
     if (pid =3D=3D 0)
     {
 	int conn;
 	char buf[] =3D "AAAAAAAAA";
 
 	close(s);
 	conn =3D socket(AF_LOCAL, SOCK_DGRAM, 0);
 	if (conn =3D=3D -1)
 	{
 	    perror("child socket");
 	    exit(1);
 	}
 	if (sendto(conn, buf, sizeof(buf), 0, (struct sockaddr *)&un,
 	    sizeof(un)) !=3D sizeof(buf))
 	{
 	    perror("sendto");
 	    exit(1);
 	}
 	close(conn);
 	exit(0);
     }
 
     /* Make sure the data is there when we try to receive it. */
 
     sleep(10);
 
     if (recvfrom(s, (void *)-1, 1, 0, NULL, NULL) !=3D -1)
     {
 	fprintf(stderr, "First recvfrom didn't fail as expected\n");
 	exit(1);
     }
 
     /* This recvfrom will trigger the panic() on an unfixed system,
        or will block on a fixed system. */
 
     recvfrom(s, (void *)-1, 1, 0, NULL, NULL);
     fprintf(stderr, "Machine should be dead by now!\n");
     return 0;
 }
 
 
 --=-v+hKfXmdKd8qhWBMoQx4
 Content-Type: application/pgp-signature; name=signature.asc
 Content-Description: This is a digitally signed message part
 
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.1 (GNU/Linux)
 
 iD8DBQBDJdha5VGI7rYqFQsRAgttAKCQ6XLp9UK4CaHagxR+p4BZUeNtzACgnuBp
 TC2IbpPaEj48M4Yxswd6iN4=
 =HWHl
 -----END PGP SIGNATURE-----
 
 --=-v+hKfXmdKd8qhWBMoQx4--
 

From: James Juran <James.Juran@baesystems.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/38495: soreceive fails to maintain invariant on UDP sockets
 if uiomove fails
Date: Mon, 12 Sep 2005 17:03:29 -0400

 Sorry about the MIME in the previous message.  Hopefully this one comes
 out better.
 
 This patch fixes the problem in 5.4-RELEASE.  I don't have a -CURRENT
 or -STABLE system on which to test, but the patch does apply cleanly to
 both of those branches.
 
 James Juran
 
 --- uipc_socket.c~    Thu Mar 31 17:35:23 2005
 +++ uipc_socket.c    Mon Sep 12 14:52:40 2005
 @@ -1002,6 +1002,7 @@
     struct mbuf *nextrecord;
     int moff, type = 0;
     int orig_resid = uio->uio_resid;
 +    int mbuf_removed = 0;
 
     mp = mp0;
     if (psa != NULL)
 @@ -1121,6 +1122,7 @@
             m = m->m_next;
         } else {
             sbfree(&so->so_rcv, m);
 +            mbuf_removed = 1;
             so->so_rcv.sb_mb = m_free(m);
             m = so->so_rcv.sb_mb;
             sockbuf_pushsync(&so->so_rcv, nextrecord);
 @@ -1146,6 +1148,7 @@
                 m = m->m_next;
             } else {
                 sbfree(&so->so_rcv, m);
 +                mbuf_removed = 1;
                 so->so_rcv.sb_mb = m->m_next;
                 m->m_next = NULL;
                 *cme = m;
 @@ -1272,8 +1275,24 @@
 #endif /* ZERO_COPY_SOCKETS */
             error = uiomove(mtod(m, char *) + moff, (int)len, uio);
             SOCKBUF_LOCK(&so->so_rcv);
 -            if (error)
 +            if (error) {
 +                /*
 +                 * If any part of the record has been removed
 +                 * (such as the MT_SONAME mbuf, which will
 +                 * happen when PR_ADDR, and thus also
 +                 * PR_ATOMIC, is set), then drop the entire
 +                 * record to maintain the atomicity of the
 +                 * receive operation.
 +                 *
 +                 * This avoids a later panic when compiled
 +                 * with INVARIANT.
 +                 */
 +                if (m && mbuf_removed
 +                    && (pr->pr_flags & PR_ATOMIC))
 +                    (void) sbdroprecord_locked(&so->so_rcv);
 +
                 goto release;
 +            }
         } else
             uio->uio_resid -= len;
         SOCKBUF_LOCK_ASSERT(&so->so_rcv);
State-Changed-From-To: open->analyzed 
State-Changed-By: bms 
State-Changed-When: Wed Aug 2 14:09:50 UTC 2006 
State-Changed-Why:  
It looks like this issue is still present in FreeBSD. 

This was confirmed by code inspection in HEAD and by testing 
(using supplied test case) against 6.1-RELEASE. 

I am in the process of setting up my infrastructure to futz 
with kernels once again. I'll try to look at this soon. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=38495 
State-Changed-From-To: analyzed->patched 
State-Changed-By: bms 
State-Changed-When: Fri Sep 22 15:34:32 UTC 2006 
State-Changed-Why:  
Fix and test case are good, committed to HEAD pending MFC. 
Thank you! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=38495 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/38495: commit references a PR
Date: Fri, 22 Sep 2006 15:34:29 +0000 (UTC)

 bms         2006-09-22 15:34:17 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/kern             uipc_socket.c 
   Log:
   Fix a case where socket I/O atomicity is violated due to not dropping
   the entire record when a non-data mbuf is removed in the soreceive() path.
   This only triggers a panic directly when compiled with INVARIANTS.
   
   PR:             38495
   Submitted by:   James Juran
   MFC after:      1 week
   
   Revision  Changes    Path
   1.283     +16 -1     src/sys/kern/uipc_socket.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 

From: Bruce M Simpson <bms@incunabulum.net>
To: freebsd-gnats-submit@FreeBSD.org
Cc:  
Subject: Re: kern/38495: soreceive fails to maintain invariant on UDP sockets
 if uiomove fails
Date: Sat, 07 Oct 2006 10:35:15 +0100

 This is a multi-part message in MIME format.
 --------------000000040707040906080407
 Content-Type: text/plain; charset=ISO-8859-1; format=flowed
 Content-Transfer-Encoding: 7bit
 
 
 
 --------------000000040707040906080407
 Content-Type: message/rfc822;
  name="Re: [PATCH] MFC request kern/38495"
 Content-Transfer-Encoding: 7bit
 Content-Disposition: inline;
  filename="Re: [PATCH] MFC request kern/38495"
 
 Received: from mx2.internal (mx2.internal [10.202.2.201])
 	 by store23m.internal (Cyrus v2.3.7-fmsvn9412) with LMTPA;
 	 Sat, 07 Oct 2006 05:01:34 -0400
 X-Sieve: CMU Sieve 2.3
 X-Spam-score: 0.8
 X-Delivered-to: bms@incunabulum.net
 Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42])
 	by mx2.messagingengine.com (Postfix) with ESMTP id A9859161665
 	for <bms@incunabulum.net>; Sat,  7 Oct 2006 05:01:32 -0400 (EDT)
 Received: from fledge.watson.org (fledge.watson.org [209.31.154.41])
 	by cyrus.watson.org (Postfix) with ESMTP id 0334946CC4;
 	Sat,  7 Oct 2006 05:01:25 -0400 (EDT)
 Date: Sat, 7 Oct 2006 10:01:24 +0100 (BST)
 From: Robert Watson <rwatson@FreeBSD.org>
 X-X-Sender: robert@fledge.watson.org
 To: Bruce M Simpson <bms@incunabulum.net>
 cc: re@FreeBSD.org
 Subject: Re: [PATCH] MFC request kern/38495
 In-Reply-To: <45156088.8060003@incunabulum.net>
 Message-ID: <20061007095949.B80980@fledge.watson.org>
 References: <45156088.8060003@incunabulum.net>
 MIME-Version: 1.0
 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
 
 
 On Sat, 23 Sep 2006, Bruce M Simpson wrote:
 
 > I committed a fix for an invariant not being maintained on datagram sockets 
 > the other day, This bug has been around for a long time; see 
 > sys/kern/uipc_socket.c rev 1.283 for fix.
 >
 > Diff to RELENG_6 attached. Waiting for rwatson eyeballs.
 
 Bruce,
 
 Per our conversation on IRC, I don't think this fix is complete. 
 Specifically, it handles the case where there's a failure after handling the 
 address or control data, but not after a partial read of regular data.  There 
 appears to be at least one missing assignment of mbuf_removed in the main body 
 of the data handling section, from a casual reading.  I think we should defer 
 MFCing this change until we have a higher confidence it is right.  You might 
 want to prepare a revised version and correspond with the author of the patch, 
 as they've clearly spent time working on this, and may be able to offer 
 insight into what's going on -- for example, maybe there's a good reason not 
 to do what I suggest. :-)
 
 Thanks,
 
 Robert N M Watson
 Computer Laboratory
 University of Cambridge
 
 --------------000000040707040906080407--
State-Changed-From-To: patched->analyzed 
State-Changed-By: bms 
State-Changed-When: Sat Oct 7 11:10:20 UTC 2006 
State-Changed-Why:  
a more thorough fix is needed 

http://www.freebsd.org/cgi/query-pr.cgi?pr=38495 

From: James Juran <James.Juran@baesystems.com>
To: Bruce M Simpson <bms@incunabulum.net>
Cc: bug-followup@freebsd.org
Subject: Re: kern/38495
Date: Tue, 10 Oct 2006 09:17:56 -0400

 On Sat, 2006-10-07 at 10:37 +0100, Bruce M Simpson wrote:
 > FYI. I committed this to -CURRENT but some rethinking of the patch may 
 > be needed. It doesn't cover all of the cases, i.e. the invariant can 
 > still be incorrectly maintained in the case where multiple UDP fragments 
 > exist as an mbuf chain.
 > 
 > I'm trying to focus on stuff further on the stack, so this isn't a 
 > priority for me at the moment; if you could look again I'd really 
 > appreciate it.
 > 
 > Thanks!
 > BMS
 
 I'll try to look at this soon, but it may take a little while as I also
 have other priorities right now.
 
 Thanks.
 
 -- 
 James Juran
 Senior Secure Systems Analyst
 XTS-400 Operating Systems
 BAE Systems Information Technology
 James.Juran@baesystems.com
 (703) 563-8081
 

From: James Juran <james.juran@baesystems.com>
To: Bruce M Simpson <bms@incunabulum.net>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>,
        Greg Troxel <gdt@ir.bbn.com>
Subject: Re: kern/38495: soreceive fails to maintain invariant on UDP
	sockets if uiomove fails
Date: Wed, 20 Dec 2006 18:19:03 -0500

 On Sat, 2006-10-07 at 04:37 -0500, Bruce M Simpson wrote:
 > FYI. I committed this to -CURRENT but some rethinking of the patch
 > may  
 > be needed. It doesn't cover all of the cases, i.e. the invariant can  
 > still be incorrectly maintained in the case where multiple UDP
 > fragments  
 > exist as an mbuf chain.
 > 
 > I'm trying to focus on stuff further on the stack, so this isn't a  
 > priority for me at the moment; if you could look again I'd really  
 > appreciate it.
 > 
 > Thanks! 
 > BMS
 
 Yes, you and rwatson are right that the original patch was not fully
 correct.  In fact, in looking at this again I don't see why I didn't
 just use the same approach that NetBSD used to fix this issue, as
 mentioned previously in this bug.
 
 This patch will revert my previous patch and just do what NetBSD did.
 I've tested that the testcase I posted earlier for this bug still is
 handled properly with the 200611 snapshot of -CURRENT + this patch.
 
 Index: uipc_socket.c
 ===================================================================
 RCS file: /home/ncvs/src/sys/kern/uipc_socket.c,v
 retrieving revision 1.286
 diff -u -r1.286 uipc_socket.c
 --- uipc_socket.c	22 Nov 2006 23:54:29 -0000	1.286
 +++ uipc_socket.c	20 Dec 2006 22:19:15 -0000
 @@ -1439,7 +1439,6 @@
  	struct protosw *pr = so->so_proto;
  	struct mbuf *nextrecord;
  	int moff, type = 0;
 -	int mbuf_removed = 0;
  	int orig_resid = uio->uio_resid;
  
  	mp = mp0;
 @@ -1560,7 +1559,6 @@
  			m = m->m_next;
  		} else {
  			sbfree(&so->so_rcv, m);
 -			mbuf_removed = 1;
  			so->so_rcv.sb_mb = m_free(m);
  			m = so->so_rcv.sb_mb;
  			sockbuf_pushsync(&so->so_rcv, nextrecord);
 @@ -1586,7 +1584,6 @@
  				m = m->m_next;
  			} else {
  				sbfree(&so->so_rcv, m);
 -				mbuf_removed = 1;
  				so->so_rcv.sb_mb = m->m_next;
  				m->m_next = NULL;
  				*cme = m;
 @@ -1710,16 +1707,17 @@
  			SOCKBUF_LOCK(&so->so_rcv);
  			if (error) {
  				/*
 -				 * If any part of the record has been removed
 -				 * (such as the MT_SONAME mbuf, which will
 -				 * happen when PR_ADDR, and thus also
 -				 * PR_ATOMIC, is set), then drop the entire
 -				 * record to maintain the atomicity of the
 -				 * receive operation.
 +				 * The MT_SONAME mbuf has already been removed
 +				 * from the record, so it is necessary to
 +				 * remove the data mbufs, if any, to preserve
 +				 * the invariant in the case of PR_ADDR that
 +				 * requires MT_SONAME mbufs at the head of
 +				 * each record.
  				 */
 -				if (m && mbuf_removed &&
 -				    (pr->pr_flags & PR_ATOMIC))
 +				if (m && pr->pr_flags & PR_ATOMIC
 +				    && ((flags & MSG_PEEK) == 0)) {
  					(void)sbdroprecord_locked(&so->so_rcv);
 +				}
  				goto release;
  			}
  		} else
 
 
 
 -- 
 James Juran
 Senior Secure Systems Analyst
 BAE Systems Information Technology
 Information Assurance Group
 XTS Operating Systems
 james.juran@baesystems.com
 703-563-8081
State-Changed-From-To: analyzed->patched 
State-Changed-By: bms 
State-Changed-When: Sat Dec 23 21:07:22 UTC 2006 
State-Changed-Why:  
Committed, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=38495 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/38495: commit references a PR
Date: Sat, 23 Dec 2006 21:07:24 +0000 (UTC)

 bms         2006-12-23 21:07:07 UTC
 
   FreeBSD src repository
 
   Modified files:
     sys/kern             uipc_socket.c 
   Added files:
     tools/regression/sockets/pr_atomic Makefile pr_atomic.c 
   Log:
   Drop all received data mbufs from a socket's queue if the MT_SONAME
   mbuf is dropped, to preserve the invariant in the PR_ADDR case.
   
   Add a regression test to detect this condition, but do not hook it
   up to the build for now.
   
   PR:             kern/38495
   Submitted by:   James Juran
   Reviewed by:    sam, rwatson
   Obtained from:  NetBSD
   MFC after:      2 weeks
   
   Revision  Changes    Path
   1.287     +9 -11     src/sys/kern/uipc_socket.c
   1.1       +7 -0      src/tools/regression/sockets/pr_atomic/Makefile (new)
   1.1       +109 -0    src/tools/regression/sockets/pr_atomic/pr_atomic.c (new)
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 

From: Bruce M Simpson <bms@incunabulum.net>
To: James Juran <james.juran@baesystems.com>
Cc: bug-followup@FreeBSD.org, Robert Watson <rwatson@FreeBSD.org>, 
 Greg Troxel <gdt@ir.bbn.com>
Subject: Re: kern/38495: soreceive fails to maintain invariant on UDP	sockets
 if uiomove fails
Date: Sat, 23 Dec 2006 21:13:04 +0000

 Thank you very much for getting back to us and dealing with this.
 I've committed this now as well as the test case.
 
 I've put my copyright on the test case file as I worked on it to get it 
 into sync with the rest of the tcp/ip regression tests, if you'd rather 
 I put your name on too please do let me know, you are of course credited 
 as submitter.
 
 Best wishes for the break,
 BMS
State-Changed-From-To: patched->closed 
State-Changed-By: bms 
State-Changed-When: Sat Feb 3 04:01:28 UTC 2007 
State-Changed-Why:  
MFC committed, thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=38495 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/38495: commit references a PR
Date: Sat,  3 Feb 2007 04:01:32 +0000 (UTC)

 bms         2007-02-03 04:01:22 UTC
 
   FreeBSD src repository
 
   Modified files:        (Branch: RELENG_6)
     sys/kern             uipc_socket.c 
   Log:
   MFC:
    Drop all received data mbufs from a socket's queue if the MT_SONAME
    mbuf is dropped, to preserve the invariant in the PR_ADDR case.
   
   PR:             kern/38495
   Submitted by:   James Juran
   Reviewed by:    sam, rwatson
   Obtained from:  NetBSD
   
   Revision   Changes    Path
   1.242.2.8  +13 -1     src/sys/kern/uipc_socket.c
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
>Unformatted:
