From freebsd@grem.de  Mon Jun 24 00:08:19 2013
Return-Path: <freebsd@grem.de>
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
	by hub.freebsd.org (Postfix) with ESMTP id 4D6797C7
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 24 Jun 2013 00:08:19 +0000 (UTC)
	(envelope-from freebsd@grem.de)
Received: from mail.grem.de (outcast.grem.de [213.239.217.27])
	by mx1.freebsd.org (Postfix) with SMTP id 745451302
	for <FreeBSD-gnats-submit@freebsd.org>; Mon, 24 Jun 2013 00:08:17 +0000 (UTC)
Received: (qmail 83299 invoked by uid 0); 24 Jun 2013 00:02:38 -0000
Message-Id: <20130624000238.83298.qmail@mail.grem.de>
Date: 24 Jun 2013 00:02:38 -0000
From: Michael Gmelin <freebsd@grem.de>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: [patch] Multicast SO_REUSEADDR handled incorrectly
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         179901
>Category:       kern
>Synopsis:       [netinet] [patch] Multicast SO_REUSEADDR handled incorrectly
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    trociny
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Jun 24 00:10:00 UTC 2013
>Closed-Date:    Sun Aug 25 19:03:20 UTC 2013
>Last-Modified:  Sun Aug 25 19:03:20 UTC 2013
>Originator:     Michael Gmelin
>Release:        FreeBSD 9.1-RELEASE-p2 amd64
>Organization:
Grem Equity GmbH
>Environment:
System: FreeBSD box.grem.de 9.1-RELEASE-p2 FreeBSD 9.1-RELEASE-p2 #5 r249052M: Fri May 31 17:50:16 UTC
>Description:

Traditionally SO_REUSEADDR is used to indicate binding to the same
address and port is permitted in the case of multicast addresses. BSD
introduced the special sockopt SO_REUSEPORT and some point, but
SO_REUSEADDR still implied SO_REUSEPORT automatically, which is
important for portability reasons - client software expects
SO_REUSEADDR to be sufficient and the kernel indicates that this is the
way it should be for multicast addresses.

While debugging an issue causing a unit test of the port devel/ice to
fail, I realized that this is not handled correctly anymore [1].

As part of r227207, which was MFC'd in r227428 [2], handling of
sockopts was slightly changed and setting of the implicit SO_REUSEPORT
flag was moved to ip_output.c, which sets INP_REUSEPORT on
inp->inp_flags2, so that these flags can be checked when binding
additional sockets in in_pcb.c. Unfortunately this approach doesn't
work, since these socket options are set before bind is called, and
therefore checking IN_MULTICAST returns false at this point and the
required INP_REUSEPORT option is never set implicitely.

As a result, attempts to bind additional sockets to the same multicast
address and port fail with EADDRINUSE ("Address already in use"), which
breaks existing software.

As a workaround, applications can be fixed by either using
SO_REUSEPORT, or by calling setsockopt SO_REUSEADDR again after
calling bind (because at that point the address is known and can be
detected as multicast by ip_output.c).

Changing application code is not desireable for obvious reasons.

[1] http://lists.freebsd.org/pipermail/freebsd-ports/2013-June/084480.html
[2] http://svnweb.freebsd.org/base?view=revision&revision=227428

>How-To-Repeat:

Build the following program, which is also available at
http://blog.grem.de/multicast.c

# cc -o multicast -c multicast.c

SNIP
#include <sys/types.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <memory.h> 
#include <errno.h>
 
struct testdata {
  int port;
  int flag;
  int settwice;
  int expectOk;
};

#define TDSIZE 20
#define BASEPORT 5555

const struct testdata td[TDSIZE] = {
 {0, SO_REUSEADDR, 0, 1}, {0, SO_REUSEADDR, 0, 1}, {0, SO_REUSEPORT, 0, 1},
 {1, SO_REUSEPORT, 0, 1}, {1, SO_REUSEPORT, 0, 1}, {1, SO_REUSEADDR, 0, 1},
 {1, SO_REUSEPORT, 0, 1},
 {2, SO_REUSEADDR, 1, 1}, {2, SO_REUSEADDR, 1, 1}, {2, SO_REUSEPORT, 0, 1},
 {2, SO_REUSEADDR, 0, 1}, {2, SO_REUSEPORT, 0, 1},
 {3, 0, 0, 1}, {3, SO_REUSEADDR, 0, 0}, {3, SO_REUSEPORT, 0, 0},
 {4, SO_REUSEADDR, 0, 1}, {4, 0, 0, 0},
 {5, SO_REUSEPORT, 0, 1}, {5, SO_REUSEPORT, 0, 1}, {5, 0, 0, 0}, 
};

int main(int argc, char *argv[])
{
  int reuse = 1;
  struct sockaddr_in localSock;
  int sd;
  int i;
  int lastport = 0;
  int port;

  memset((char *) &localSock, 0, sizeof(localSock));
  localSock.sin_family = AF_INET;
  localSock.sin_addr.s_addr  = inet_addr("239.1.1.1");

  for (i = 0; i < TDSIZE; ++i) {
    port = BASEPORT + td[i].port;
    localSock.sin_port = htons(port);
    sd = socket(AF_INET, SOCK_DGRAM, 0);
    if (lastport != port) {
      printf("Port %i:\n", port);
      lastport = port;
    }
    if (td[i].flag) {
      printf("  Bind using SO_REUSE%s%s...",
            td[i].flag == SO_REUSEADDR ? "ADDR" : "PORT",
            td[i].settwice?" x 2":"....");
      setsockopt(sd, SOL_SOCKET, td[i].flag, (char *)&reuse, sizeof(reuse));
    }
    else {
      printf("  Bind without socketopts.......");
    }
    if (bind(sd, (struct sockaddr*)&localSock, sizeof(localSock))) {
      printf("FAIL (%sexpected): %s\n", td[i].expectOk?"NOT ":"", strerror(errno));
      close(sd);
    }
    else {
      printf("OK   (%sexpected)\n", td[i].expectOk?"":"NOT ");
    }
    if (td[i].settwice && td[i].flag)
     setsockopt(sd, SOL_SOCKET, td[i].flag, (char *)&reuse, sizeof(reuse));
  }
}
SNAP

Running this on 9.1-RELEASE gives the following result:

Port 5555:
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEADDR.......FAIL (NOT expected): Address already in use
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in use
Port 5556:
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in use
Port 5557:
  Bind using SO_REUSEADDR x 2...OK   (expected)
  Bind using SO_REUSEADDR x 2...OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in use
Port 5558:
  Bind without socketopts.......OK   (expected)
  Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
  Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
Port 5559:
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind without socketopts.......FAIL (expected): Address already in use
Port 5560:
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind without socketopts.......FAIL (expected): Address already in use

Out of curiosity, I ran the same code on an outdated version of FreeBSD
(7.2-RELEASE):

Port 5555:
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
  use
Port 5556:
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
  use
Port 5557:
  Bind using SO_REUSEADDR x 2...OK   (expected)
  Bind using SO_REUSEADDR x 2...OK   (expected)
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
  use
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
  use
Port 5558:
  Bind without socketopts.......OK   (expected)
  Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
  Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
Port 5559:
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind without socketopts.......FAIL (expected): Address already in use
Port 5560:
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind without socketopts.......FAIL (expected): Address already in use

This shows that there were issues with this code before r227207 as
well - but only when mixing SO_REUSEADDR and SO_REUSEPORT, which hardly
ever happens in practice. Nevertheless, this code didn't behave like it
should either - at least in my understanding.

>Fix:

Disclaimer: The patch needs to be reviewed by somebody who has a better
understanding of the code in question.

The idea of the attached patch is to introduce a new flag called
INP_REUSEADDR in in_pcb.h, which is set on inp->inp_flags2 in
ip_output.c whenever SO_REUSEADDR is set on a socket.  This way there
is an explicit record of the setsockopt call which can be utilized,
even though we don't know that the socket will be bound to a multicast
address yet.

The last step is to modify in_pcb.c (line 597), so that the condition
for emitting EADDRINUSE only matches if it's not a multicast address or
if it is a multicast address and neither INP_REUSEADDR nor
INP_REUSEPORT are set.

After applying the patch and rebuilding the kernel, multicast.c runs as
expected:

Port 5555:
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
Port 5556:
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
Port 5557:
  Bind using SO_REUSEADDR x 2...OK   (expected)
  Bind using SO_REUSEADDR x 2...OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
Port 5558:
  Bind without socketopts.......OK   (expected)
  Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
  Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
Port 5559:
  Bind using SO_REUSEADDR.......OK   (expected)
  Bind without socketopts.......FAIL (expected): Address already in use
Port 5560:
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind using SO_REUSEPORT.......OK   (expected)
  Bind without socketopts.......FAIL (expected): Address already in use

The patch is applied using:

# cd /usr/src 
# patch < /path/to/freebsd-multicast.patch


--- freebsd-multicast.patch begins here ---
--- sys/netinet/in_pcb.c.orig	2013-06-23 20:52:10.000000000 +0000
+++ sys/netinet/in_pcb.c	2013-06-23 21:14:45.000000000 +0000
@@ -594,7 +594,11 @@
 				    (reuseport & tw->tw_so_options) == 0)
 					return (EADDRINUSE);
 			} else if (t && (reuseport == 0 ||
-			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
+			    (!IN_MULTICAST(ntohl(sin->sin_addr.s_addr)) &&
+			     (t->inp_flags2 & INP_REUSEPORT) == 0) ||
+			    (IN_MULTICAST(ntohl(sin->sin_addr.s_addr)) &&
+			     (t->inp_flags2 &
+			      (INP_REUSEADDR | INP_REUSEPORT)) == 0))) {
 #ifdef INET6
 				if (ntohl(sin->sin_addr.s_addr) !=
 				    INADDR_ANY ||
--- sys/netinet/in_pcb.h.orig	2013-06-23 21:02:42.000000000 +0000
+++ sys/netinet/in_pcb.h	2013-06-23 21:03:40.000000000 +0000
@@ -542,6 +542,7 @@
 #define	INP_RT_VALID		0x00000002 /* cached rtentry is valid */
 #define	INP_PCBGROUPWILD	0x00000004 /* in pcbgroup wildcard list */
 #define	INP_REUSEPORT		0x00000008 /* SO_REUSEPORT option is set */
+#define	INP_REUSEADDR		0x00000010 /* SO_REUSEADDR option is set */
 
 /*
  * Flags passed to in_pcblookup*() functions.
--- sys/netinet/ip_output.c.orig	2013-06-23 21:03:51.000000000 +0000
+++ sys/netinet/ip_output.c	2013-06-23 21:04:58.000000000 +0000
@@ -915,6 +915,10 @@
 					else
 						inp->inp_flags2 &= ~INP_REUSEPORT;
 				}
+				if ((so->so_options & SO_REUSEADDR) != 0)
+					inp->inp_flags2 |= INP_REUSEADDR;
+				else
+					inp->inp_flags2 &= ~INP_REUSEADDR;
 				INP_WUNLOCK(inp);
 				error = 0;
 				break;
--- freebsd-multicast.patch ends here ---

>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-bugs->freebsd-net 
Responsible-Changed-By: linimon 
Responsible-Changed-When: Mon Jun 24 03:59:05 UTC 2013 
Responsible-Changed-Why:  
Over to maintainer(s). 

http://www.freebsd.org/cgi/query-pr.cgi?pr=179901 

From: Mikolaj Golub <trociny@FreeBSD.org>
To: bug-followup@FreeBSD.org, freebsd@grem.de
Cc:  
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Mon, 24 Jun 2013 23:29:42 +0300

 --9amGYk9869ThD9tj
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 Michael,
 
 Thank you for your analysis and the patch.
 
 I have the following notes to your patch though:
 
 1) INET6 needs fixing too.
 
 2) It looks like after introducing INP_REUSEADDR there is no need in
 handling the IN_MULTICAST case in ip_ctloutput().
 
 3) Actually you don't have to use IN_MULTICAST() in in_pcbbind_setup():
 the information is already encoded in reuseport variable.
 
 4) The patch not only fixes the regression introduced by r227207, but
 also changes the historical behavior before r227207. Although the
 change might be correct it is better to separate these issues. Feeling
 guilty for the regression introduced in r227207 I am eager to fix it
 ASAP, before 9.2 release. But I don't have strong opinion about
 changing the historical behavior.
 
 So, could you please look at the attached patch, which is based on
 your idea of INP_REUSEADDR flag? Now the code more resembles the code
 before r227207 in looks and I am a little more confident that there is
 no regression.
 
 I would appreciate any testing. Note, it is against CURRENT; STABLE
 will require patching in_pcb.h manually due to newly introduced
 INP_FREED flag.
 
 -- 
 Mikolaj Golub
 
 --9amGYk9869ThD9tj
 Content-Type: text/x-diff; charset=us-ascii
 Content-Disposition: attachment; filename="pr179901.1.patch"
 
 Index: sys/netinet/in_pcb.c
 ===================================================================
 --- sys/netinet/in_pcb.c	(revision 252162)
 +++ sys/netinet/in_pcb.c	(working copy)
 @@ -467,6 +467,23 @@ in_pcb_lport(struct inpcb *inp, struct in_addr *la
  
  	return (0);
  }
 +
 +/*
 + * Return cached socket options.
 + */
 +int
 +inp_so_options(const struct inpcb *inp)
 +{
 +   int so_options;
 +
 +   so_options = 0;
 +
 +   if ((inp->inp_flags2 & INP_REUSEPORT) != 0)
 +	   so_options |= SO_REUSEPORT;
 +   if ((inp->inp_flags2 & INP_REUSEADDR) != 0)
 +	   so_options |= SO_REUSEADDR;
 +   return (so_options);
 +}
  #endif /* INET || INET6 */
  
  #ifdef INET
 @@ -595,8 +612,8 @@ in_pcbbind_setup(struct inpcb *inp, struct sockadd
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t &&
 +			    (reuseport & inp_so_options(t)) == 0) {
  #ifdef INET6
  				if (ntohl(sin->sin_addr.s_addr) !=
  				    INADDR_ANY ||
 Index: sys/netinet/in_pcb.h
 ===================================================================
 --- sys/netinet/in_pcb.h	(revision 252162)
 +++ sys/netinet/in_pcb.h	(working copy)
 @@ -442,6 +442,7 @@ struct tcpcb *
  	inp_inpcbtotcpcb(struct inpcb *inp);
  void 	inp_4tuple_get(struct inpcb *inp, uint32_t *laddr, uint16_t *lp,
  		uint32_t *faddr, uint16_t *fp);
 +int	inp_so_options(const struct inpcb *inp);
  
  #endif /* _KERNEL */
  
 @@ -543,6 +544,7 @@ void 	inp_4tuple_get(struct inpcb *inp, uint32_t *
  #define	INP_PCBGROUPWILD	0x00000004 /* in pcbgroup wildcard list */
  #define	INP_REUSEPORT		0x00000008 /* SO_REUSEPORT option is set */
  #define	INP_FREED		0x00000010 /* inp itself is not valid */
 +#define	INP_REUSEADDR		0x00000020 /* SO_REUSEADDR option is set */
  
  /*
   * Flags passed to in_pcblookup*() functions.
 Index: sys/netinet/ip_output.c
 ===================================================================
 --- sys/netinet/ip_output.c	(revision 252162)
 +++ sys/netinet/ip_output.c	(working copy)
 @@ -900,13 +900,10 @@ ip_ctloutput(struct socket *so, struct sockopt *so
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(inp);
 -				if (IN_MULTICAST(ntohl(inp->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						inp->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						inp->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					inp->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					inp->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(inp);
  				error = 0;
  				break;
 Index: sys/netinet6/in6_pcb.c
 ===================================================================
 --- sys/netinet6/in6_pcb.c	(revision 252162)
 +++ sys/netinet6/in6_pcb.c	(working copy)
 @@ -265,8 +265,8 @@ in6_pcbbind(register struct inpcb *inp, struct soc
  					     INP_IPV6PROTO) ==
  					     (t->inp_vflag & INP_IPV6PROTO))))
  						return (EADDRINUSE);
 -				} else if (t && (reuseport == 0 ||
 -				    (t->inp_flags2 & INP_REUSEPORT) == 0) &&
 +				} else if (t &&
 +				    (reuseport & inp_so_options(t)) == 0 &&
  				    (ntohl(t->inp_laddr.s_addr) != INADDR_ANY ||
  				    (t->inp_vflag & INP_IPV6PROTO) != 0))
  					return (EADDRINUSE);
 Index: sys/netinet6/ip6_output.c
 ===================================================================
 --- sys/netinet6/ip6_output.c	(revision 252162)
 +++ sys/netinet6/ip6_output.c	(working copy)
 @@ -1477,13 +1477,10 @@ ip6_ctloutput(struct socket *so, struct sockopt *s
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(in6p);
 -				if (IN_MULTICAST(ntohl(in6p->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						in6p->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						in6p->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					in6p->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					in6p->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(in6p);
  				error = 0;
  				break;
 
 --9amGYk9869ThD9tj--

From: Michael Gmelin <freebsd@grem.de>
To: Mikolaj Golub <trociny@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Tue, 25 Jun 2013 13:39:38 +0200

 On Mon, 24 Jun 2013 23:29:42 +0300
 Mikolaj Golub <trociny@FreeBSD.org> wrote:
 
 > Michael,
 
 Hi,
 > 
 > Thank you for your analysis and the patch.
 > 
 > I have the following notes to your patch though:
 > 
 > 1) INET6 needs fixing too.
 Yes, but it seems like your patch is fixing the not all places in
 in6_pcb.c, I think you should modify the code at line 246 as well:
 
                         } else if (t && (reuseport == 0 ||
                             (t->inp_flags2 & INP_REUSEPORT) == 0)) {
                                 return (EADDRINUSE);
                         }
 
 so it says
                         } else if (t &&
                             (reuseport & inp_so_options(t)) == 0) {
 		
 Instead you're modifying the code at line 265 (which also seems to be
 affected, so it makes sense).
 
 > 
 > 2) It looks like after introducing INP_REUSEADDR there is no need in
 > handling the IN_MULTICAST case in ip_ctloutput().
 I think so too, in the end REUSEADDR and REUSEPORT are set
 independently now and REUSEADDR only gets its special meaning when being
 checked during binding. Therefore the code in ip_output should always
 do the right thing.
 
 > 
 > 3) Actually you don't have to use IN_MULTICAST() in
 > in_pcbbind_setup(): the information is already encoded in reuseport
 > variable.
 That seems pretty implicit to me, but it also keeps the original logic
 more or less intact. Whatever works best for you I guess.
 
 > 
 > 4) The patch not only fixes the regression introduced by r227207, but
 > also changes the historical behavior before r227207. Although the
 > change might be correct it is better to separate these issues. Feeling
 > guilty for the regression introduced in r227207 I am eager to fix it
 > ASAP, before 9.2 release. But I don't have strong opinion about
 > changing the historical behavior.
 
 I have a hard time believing anybody relied on the previous behaviour,
 so I wouldn't try to resemble it.
 
 > 
 > So, could you please look at the attached patch, which is based on
 > your idea of INP_REUSEADDR flag? Now the code more resembles the code
 > before r227207 in looks and I am a little more confident that there is
 > no regression.
 > 
 > I would appreciate any testing. Note, it is against CURRENT; STABLE
 > will require patching in_pcb.h manually due to newly introduced
 > INP_FREED flag.
 > 
 
 Once 1) has been resolved I can test on a machine running 9.1-RELEASE
 later (the patch is small enough to apply it manually). I will run the
 "unit test" code from multicast.c I sent earlier and add IPv6 test
 cases to it as well.
 
 Cheers,
 Michael
 
 
 -- 
 Michael Gmelin

From: Mikolaj Golub <trociny@FreeBSD.org>
To: Michael Gmelin <freebsd@grem.de>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Tue, 25 Jun 2013 18:24:55 +0300

 --tThc/1wpZn/ma/RB
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On Tue, Jun 25, 2013 at 01:39:38PM +0200, Michael Gmelin wrote:
 
 > Yes, but it seems like your patch is fixing the not all places in
 > in6_pcb.c, I think you should modify the code at line 246 as well:
 > 
 >                         } else if (t && (reuseport == 0 ||
 >                             (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 >                                 return (EADDRINUSE);
 >                         }
 > 
 > so it says
 >                         } else if (t &&
 >                             (reuseport & inp_so_options(t)) == 0) {
 > 		
 
 Good catch! I missed this because I was preparing the patch using
 r227207 as a reference, but this had been missed there too (fixed
 later in r233272 by glebius).
 
 > Once 1) has been resolved I can test on a machine running 9.1-RELEASE
 > later (the patch is small enough to apply it manually). I will run the
 > "unit test" code from multicast.c I sent earlier and add IPv6 test
 > cases to it as well.
 
 The updated patch is attached. Thanks.
 
 -- 
 Mikolaj Golub
 
 --tThc/1wpZn/ma/RB
 Content-Type: text/x-diff; charset=us-ascii
 Content-Disposition: attachment; filename="pr179901.2.patch"
 
 Index: sys/netinet/in_pcb.c
 ===================================================================
 --- sys/netinet/in_pcb.c	(revision 251760)
 +++ sys/netinet/in_pcb.c	(working copy)
 @@ -467,6 +467,23 @@ in_pcb_lport(struct inpcb *inp, struct in_addr *la
  
  	return (0);
  }
 +
 +/*
 + * Return cached socket options.
 + */
 +int
 +inp_so_options(const struct inpcb *inp)
 +{
 +   int so_options;
 +
 +   so_options = 0;
 +
 +   if ((inp->inp_flags2 & INP_REUSEPORT) != 0)
 +	   so_options |= SO_REUSEPORT;
 +   if ((inp->inp_flags2 & INP_REUSEADDR) != 0)
 +	   so_options |= SO_REUSEADDR;
 +   return (so_options);
 +}
  #endif /* INET || INET6 */
  
  #ifdef INET
 @@ -595,8 +612,7 @@ in_pcbbind_setup(struct inpcb *inp, struct sockadd
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t && (reuseport & inp_so_options(t)) == 0) {
  #ifdef INET6
  				if (ntohl(sin->sin_addr.s_addr) !=
  				    INADDR_ANY ||
 Index: sys/netinet/in_pcb.h
 ===================================================================
 --- sys/netinet/in_pcb.h	(revision 251760)
 +++ sys/netinet/in_pcb.h	(working copy)
 @@ -442,6 +442,7 @@ struct tcpcb *
  	inp_inpcbtotcpcb(struct inpcb *inp);
  void 	inp_4tuple_get(struct inpcb *inp, uint32_t *laddr, uint16_t *lp,
  		uint32_t *faddr, uint16_t *fp);
 +int	inp_so_options(const struct inpcb *inp);
  
  #endif /* _KERNEL */
  
 @@ -543,6 +544,7 @@ void 	inp_4tuple_get(struct inpcb *inp, uint32_t *
  #define	INP_PCBGROUPWILD	0x00000004 /* in pcbgroup wildcard list */
  #define	INP_REUSEPORT		0x00000008 /* SO_REUSEPORT option is set */
  #define	INP_FREED		0x00000010 /* inp itself is not valid */
 +#define	INP_REUSEADDR		0x00000020 /* SO_REUSEADDR option is set */
  
  /*
   * Flags passed to in_pcblookup*() functions.
 Index: sys/netinet/ip_output.c
 ===================================================================
 --- sys/netinet/ip_output.c	(revision 251760)
 +++ sys/netinet/ip_output.c	(working copy)
 @@ -900,13 +900,10 @@ ip_ctloutput(struct socket *so, struct sockopt *so
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(inp);
 -				if (IN_MULTICAST(ntohl(inp->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						inp->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						inp->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					inp->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					inp->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(inp);
  				error = 0;
  				break;
 Index: sys/netinet6/in6_pcb.c
 ===================================================================
 --- sys/netinet6/in6_pcb.c	(revision 251760)
 +++ sys/netinet6/in6_pcb.c	(working copy)
 @@ -243,8 +243,7 @@ in6_pcbbind(register struct inpcb *inp, struct soc
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t && (reuseport & inp_so_options(t)) == 0) {
  				return (EADDRINUSE);
  			}
  #ifdef INET
 @@ -265,8 +264,8 @@ in6_pcbbind(register struct inpcb *inp, struct soc
  					     INP_IPV6PROTO) ==
  					     (t->inp_vflag & INP_IPV6PROTO))))
  						return (EADDRINUSE);
 -				} else if (t && (reuseport == 0 ||
 -				    (t->inp_flags2 & INP_REUSEPORT) == 0) &&
 +				} else if (t &&
 +				    (reuseport & inp_so_options(t)) == 0 &&
  				    (ntohl(t->inp_laddr.s_addr) != INADDR_ANY ||
  				    (t->inp_vflag & INP_IPV6PROTO) != 0))
  					return (EADDRINUSE);
 Index: sys/netinet6/ip6_output.c
 ===================================================================
 --- sys/netinet6/ip6_output.c	(revision 251760)
 +++ sys/netinet6/ip6_output.c	(working copy)
 @@ -1477,13 +1477,10 @@ ip6_ctloutput(struct socket *so, struct sockopt *s
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(in6p);
 -				if (IN_MULTICAST(ntohl(in6p->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						in6p->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						in6p->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					in6p->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					in6p->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(in6p);
  				error = 0;
  				break;
 
 --tThc/1wpZn/ma/RB--

From: Michael Gmelin <freebsd@grem.de>
To: Mikolaj Golub <trociny@FreeBSD.org>
Cc: bug-followup@FreeBSD.org
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Wed, 26 Jun 2013 15:03:40 +0200

 On Tue, 25 Jun 2013 18:24:55 +0300
 Mikolaj Golub <trociny@FreeBSD.org> wrote:
 
 > On Tue, Jun 25, 2013 at 01:39:38PM +0200, Michael Gmelin wrote:
 > 
 > > Yes, but it seems like your patch is fixing the not all places in
 > > in6_pcb.c, I think you should modify the code at line 246 as well:
 > > 
 > >                         } else if (t && (reuseport == 0 ||
 > >                             (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 > >                                 return (EADDRINUSE);
 > >                         }
 > > 
 > > so it says
 > >                         } else if (t &&
 > >                             (reuseport & inp_so_options(t)) == 0) {
 > > 		
 > 
 > Good catch! I missed this because I was preparing the patch using
 > r227207 as a reference, but this had been missed there too (fixed
 > later in r233272 by glebius).
 > 
 > > Once 1) has been resolved I can test on a machine running
 > > 9.1-RELEASE later (the patch is small enough to apply it manually).
 > > I will run the "unit test" code from multicast.c I sent earlier and
 > > add IPv6 test cases to it as well.
 > 
 > The updated patch is attached. Thanks.
 > 
 
 Hi,
 
 I adapted the test code, you can find it at
 
 http://blog.grem.de/multicast.c
 
 Test output is:
 
 IPv4 Port 5555:
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use IPv4 Port 5556:
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use IPv4 Port 5557:
   Bind using SO_REUSEADDR x 2...OK   (expected)
   Bind using SO_REUSEADDR x 2...OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use IPv4 Port 5558:
   Bind without socketopts.......OK   (expected)
   Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
   Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
 IPv4 Port 5559:
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind without socketopts.......FAIL (expected): Address already in use
 IPv4 Port 5560:
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind without socketopts.......FAIL (expected): Address already in use
 IPv6 Port 5555:
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use IPv6 Port 5556:
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use IPv6 Port 5557:
   Bind using SO_REUSEADDR x 2...OK   (expected)
   Bind using SO_REUSEADDR x 2...OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use Bind using SO_REUSEADDR.......OK   (expected)
   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 use IPv6 Port 5558:
   Bind without socketopts.......OK   (expected)
   Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
   Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
 IPv6 Port 5559:
   Bind using SO_REUSEADDR.......OK   (expected)
   Bind without socketopts.......FAIL (expected): Address already in use
 IPv6 Port 5560:
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind using SO_REUSEPORT.......OK   (expected)
   Bind without socketopts.......FAIL (expected): Address already in use
 
 So you maintained the old PORT/ADDR behavior, which I think is not such
 a great idea. I would suggest to get another opinion on this, just
 because it's broken now doesn't mean we have to perpetuate it - maybe we
 should compare the behavior with other Unix(like) OSes like the other
 BSDs and Linux to see how their implementations work - usually ported
 software is not changed in that respect, so being compatible is
 valuable.
 
 Besides my rant the code works as designed and seems to resemble the
 behavior before r227207 correctly (I manually applied the patches to
 9.1-RELEASE).
 
 Fun fact: The code in ip6_output.c could have never worked in the first
 place, since it used IN_MULTICAST instead of IN6_IS_ADDR_MULTICAST:
 
 if (IN_MULTICAST(ntohl(in6p->inp_laddr.s_addr)))
 ...
 
 Cheers,
 Michael
 
 
 -- 
 Michael Gmelin

From: Mikolaj Golub <to.my.trociny@gmail.com>
To: Michael Gmelin <freebsd@grem.de>
Cc: Mikolaj Golub <trociny@FreeBSD.org>, bug-followup@FreeBSD.org
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Thu, 27 Jun 2013 23:00:16 +0300

 On Wed, Jun 26, 2013 at 03:03:40PM +0200, Michael Gmelin wrote:
 > Hi,
 > 
 > I adapted the test code, you can find it at
 > 
 > http://blog.grem.de/multicast.c
 > 
 > Test output is:
 > 
 > IPv4 Port 5555:
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use IPv4 Port 5556:
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use IPv4 Port 5557:
 >   Bind using SO_REUSEADDR x 2...OK   (expected)
 >   Bind using SO_REUSEADDR x 2...OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use IPv4 Port 5558:
 >   Bind without socketopts.......OK   (expected)
 >   Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
 >   Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
 > IPv4 Port 5559:
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind without socketopts.......FAIL (expected): Address already in use
 > IPv4 Port 5560:
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind without socketopts.......FAIL (expected): Address already in use
 > IPv6 Port 5555:
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use IPv6 Port 5556:
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use IPv6 Port 5557:
 >   Bind using SO_REUSEADDR x 2...OK   (expected)
 >   Bind using SO_REUSEADDR x 2...OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind using SO_REUSEPORT.......FAIL (NOT expected): Address already in
 > use IPv6 Port 5558:
 >   Bind without socketopts.......OK   (expected)
 >   Bind using SO_REUSEADDR.......FAIL (expected): Address already in use
 >   Bind using SO_REUSEPORT.......FAIL (expected): Address already in use
 > IPv6 Port 5559:
 >   Bind using SO_REUSEADDR.......OK   (expected)
 >   Bind without socketopts.......FAIL (expected): Address already in use
 > IPv6 Port 5560:
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind using SO_REUSEPORT.......OK   (expected)
 >   Bind without socketopts.......FAIL (expected): Address already in use
 > 
 
 Thank you for testing!
 
 > So you maintained the old PORT/ADDR behavior, which I think is not such
 > a great idea. I would suggest to get another opinion on this, just
 > because it's broken now doesn't mean we have to perpetuate it - maybe we
 > should compare the behavior with other Unix(like) OSes like the other
 > BSDs and Linux to see how their implementations work - usually ported
 > software is not changed in that respect, so being compatible is
 > valuable.
 
 It is difficult to talk about portability in the case of SO_REUSEPORT.
 AFAIK, there is no SO_REUSEPORT in Linux and it is recommended to
 always use SO_REUSEADDR for multicast in portable code. It looks like
 in this case we will always have expected behavior with the proposed
 patch.
 
 > Besides my rant the code works as designed and seems to resemble the
 > behavior before r227207 correctly (I manually applied the patches to
 > 9.1-RELEASE).
 > 
 > Fun fact: The code in ip6_output.c could have never worked in the first
 > place, since it used IN_MULTICAST instead of IN6_IS_ADDR_MULTICAST:
 > 
 > if (IN_MULTICAST(ntohl(in6p->inp_laddr.s_addr)))
 > ...
 
 I don't insist on maintaining the old behaviour. But as actually we
 have 2 issues here (regression introduced by me in FreeBSD9 and
 historical behavior that looks wrong), with different priority, I
 would like to fix the issues separately. This way it will be easier to
 track the changes, e.g. when after a year it turns out that the second
 change has broken some other case.
 
 For now I am more concerned about having SO_REUSEADDR regression fixed
 in CURRENT and STABLE9 before 9.2. The patch is under review and I
 plan to commit it next week if it is ok.
 
 The second issue might require more discussion before commiting the
 change.
 
 -- 
 Mikolaj Golub

From: Mark Linimon <linimon@lonesome.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Fri, 28 Jun 2013 14:55:38 -0500

 ----- Forwarded message from Bernd Walter <ticso@cicely7.cicely.de> -----
 
 Date: Fri, 28 Jun 2013 12:15:14 +0200
 From: Bernd Walter <ticso@cicely7.cicely.de>
 To: linimon@freebsd.org
 Cc: freebsd-bugs@freebsd.org, freebsd-net@freebsd.org
 Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled incorrectly
 User-Agent: Mutt/1.5.11
 
 I don't know if this is related or a different bug, but the same
 mentioned commits are suspicious for us.
 We had been running with our own IPv6 software into the same REUSEADDR
 problem and changed to REUSEPORT as this is how it is done in mcastread
 from mcast-tools port.
 Don't know where we originally got the REUSEADDR from, probably a Stevens
 book.
 So far binding works with this change in our software.
 However we only receive packets from network and not packets from
 the host itself.
 We use multicast to notify multiple processes on multiple machines,
 including the machine itself.
 To reproduce:
  - use two hosts
  - start mcastread on each of them on an interface with shared LAN
  - send via mcastsend on one host
  - packets are received on the other host, but not with the mcastread
    on the same host
 
 -- 
 B.Walter <bernd@bwct.de> http://www.bwct.de
 Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.
 
 ----- End forwarded message -----

From: Mark Linimon <linimon@lonesome.com>
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Fri, 28 Jun 2013 14:56:07 -0500

 ----- Forwarded message from Bernd Walter <ticso@cicely7.cicely.de> -----
 
 Date: Fri, 28 Jun 2013 20:58:52 +0200
 From: Bernd Walter <ticso@cicely7.cicely.de>
 To: linimon@freebsd.org
 Cc: freebsd-net@freebsd.org, freebsd-bugs@freebsd.org
 Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled incorrectly
 User-Agent: Mutt/1.5.11
 
 It is unrelated, but I have found the cause for it and filed kern/180065
 together with a working patch.
 The reason is that the packets have no valid checksums when processed
 in ip6_input because of delayed checksum changes.
 
 -- 
 B.Walter <bernd@bwct.de> http://www.bwct.de
 Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.
 
 ----- End forwarded message -----

From: Mikolaj Golub <trociny@FreeBSD.org>
To: bug-followup@FreeBSD.org
Cc: Michael Gmelin <freebsd@grem.de>
Subject: Re: kern/179901: [netinet] [patch] Multicast SO_REUSEADDR handled
 incorrectly
Date: Sun, 30 Jun 2013 10:17:05 +0300

 --EeQfGwPcQSOJBaQU
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 
 On Thu, Jun 27, 2013 at 11:00:16PM +0300, Mikolaj Golub wrote:
 
 > I don't insist on maintaining the old behaviour. But as actually we
 > have 2 issues here (regression introduced by me in FreeBSD9 and
 > historical behavior that looks wrong), with different priority, I
 > would like to fix the issues separately. This way it will be easier to
 > track the changes, e.g. when after a year it turns out that the second
 > change has broken some other case.
 
 Here is a patch for the second issue.
 
 -- 
 Mikolaj Golub
 
 --EeQfGwPcQSOJBaQU
 Content-Type: text/x-diff; charset=us-ascii
 Content-Disposition: attachment; filename="pr179901.2.1.patch"
 
 commit 7cf3a6a95d74ae91c80350fc1ae8e96fe59c3c65
 Author: Mikolaj Golub <trociny@freebsd.org>
 Date:   Sun Jun 30 00:09:20 2013 +0300
 
     A complete duplication of binding should be allowed if on both new and
     duplicated sockets a multicast address is bound and either
     SO_REUSEPORT or SO_REUSEADDR is set.
     
     But actually it works for the following combinations:
     
      * SO_REUSEPORT is set for the fist socket and SO_REUSEPORT for the new;
      * SO_REUSEADDR is set for the fist socket and SO_REUSEADDR for the new;
      * SO_REUSEPORT is set for the fist socket and SO_REUSEADDR for the new;
     
     and fails for this:
     
      * SO_REUSEADDR is set for the fist socket and SO_REUSEPORT for the new.
     
     Fix the last case.
     
     PR:		179901
 
 diff --git a/sys/netinet/in_pcb.c b/sys/netinet/in_pcb.c
 index 3506b74..eb15a38 100644
 --- a/sys/netinet/in_pcb.c
 +++ b/sys/netinet/in_pcb.c
 @@ -554,7 +554,7 @@ in_pcbbind_setup(struct inpcb *inp, struct sockaddr *nam, in_addr_t *laddrp,
  			 * and a multicast address is bound on both
  			 * new and duplicated sockets.
  			 */
 -			if (so->so_options & SO_REUSEADDR)
 +			if ((so->so_options & (SO_REUSEADDR|SO_REUSEPORT)) != 0)
  				reuseport = SO_REUSEADDR|SO_REUSEPORT;
  		} else if (sin->sin_addr.s_addr != INADDR_ANY) {
  			sin->sin_port = 0;		/* yech... */
 diff --git a/sys/netinet6/in6_pcb.c b/sys/netinet6/in6_pcb.c
 index a0a6874..fb84279 100644
 --- a/sys/netinet6/in6_pcb.c
 +++ b/sys/netinet6/in6_pcb.c
 @@ -156,7 +156,7 @@ in6_pcbbind(register struct inpcb *inp, struct sockaddr *nam,
  			 * and a multicast address is bound on both
  			 * new and duplicated sockets.
  			 */
 -			if (so->so_options & SO_REUSEADDR)
 +			if ((so->so_options & (SO_REUSEADDR|SO_REUSEPORT)) != 0)
  				reuseport = SO_REUSEADDR|SO_REUSEPORT;
  		} else if (!IN6_IS_ADDR_UNSPECIFIED(&sin6->sin6_addr)) {
  			struct ifaddr *ifa;
 
 --EeQfGwPcQSOJBaQU--

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179901: commit references a PR
Date: Thu,  4 Jul 2013 18:38:19 +0000 (UTC)

 Author: trociny
 Date: Thu Jul  4 18:38:00 2013
 New Revision: 252710
 URL: http://svnweb.freebsd.org/changeset/base/252710
 
 Log:
   In r227207, to fix the issue with possible NULL inp_socket pointer
   dereferencing, when checking for SO_REUSEPORT option (and SO_REUSEADDR
   for multicast), INP_REUSEPORT flag was introduced to cache the socket
   option.  It was decided then that one flag would be enough to cache
   both SO_REUSEPORT and SO_REUSEADDR: when processing SO_REUSEADDR
   setsockopt(2), it was checked if it was called for a multicast address
   and INP_REUSEPORT was set accordingly.
   
   Unfortunately that approach does not work when setsockopt(2) is called
   before binding to a multicast address: the multicast check fails and
   INP_REUSEPORT is not set.
   
   Fix this by adding INP_REUSEADDR flag to unconditionally cache
   SO_REUSEADDR.
   
   PR:		179901
   Submitted by:	Michael Gmelin freebsd grem.de (initial version)
   Reviewed by:	rwatson
   MFC after:	1 week
 
 Modified:
   head/sys/netinet/in_pcb.c
   head/sys/netinet/in_pcb.h
   head/sys/netinet/ip_output.c
   head/sys/netinet6/in6_pcb.c
   head/sys/netinet6/ip6_output.c
 
 Modified: head/sys/netinet/in_pcb.c
 ==============================================================================
 --- head/sys/netinet/in_pcb.c	Thu Jul  4 18:00:27 2013	(r252709)
 +++ head/sys/netinet/in_pcb.c	Thu Jul  4 18:38:00 2013	(r252710)
 @@ -467,6 +467,23 @@ in_pcb_lport(struct inpcb *inp, struct i
  
  	return (0);
  }
 +
 +/*
 + * Return cached socket options.
 + */
 +short
 +inp_so_options(const struct inpcb *inp)
 +{
 +   short so_options;
 +
 +   so_options = 0;
 +
 +   if ((inp->inp_flags2 & INP_REUSEPORT) != 0)
 +	   so_options |= SO_REUSEPORT;
 +   if ((inp->inp_flags2 & INP_REUSEADDR) != 0)
 +	   so_options |= SO_REUSEADDR;
 +   return (so_options);
 +}
  #endif /* INET || INET6 */
  
  #ifdef INET
 @@ -595,8 +612,7 @@ in_pcbbind_setup(struct inpcb *inp, stru
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t && (reuseport & inp_so_options(t)) == 0) {
  #ifdef INET6
  				if (ntohl(sin->sin_addr.s_addr) !=
  				    INADDR_ANY ||
 
 Modified: head/sys/netinet/in_pcb.h
 ==============================================================================
 --- head/sys/netinet/in_pcb.h	Thu Jul  4 18:00:27 2013	(r252709)
 +++ head/sys/netinet/in_pcb.h	Thu Jul  4 18:38:00 2013	(r252710)
 @@ -442,6 +442,7 @@ struct tcpcb *
  	inp_inpcbtotcpcb(struct inpcb *inp);
  void 	inp_4tuple_get(struct inpcb *inp, uint32_t *laddr, uint16_t *lp,
  		uint32_t *faddr, uint16_t *fp);
 +short	inp_so_options(const struct inpcb *inp);
  
  #endif /* _KERNEL */
  
 @@ -543,6 +544,7 @@ void 	inp_4tuple_get(struct inpcb *inp, 
  #define	INP_PCBGROUPWILD	0x00000004 /* in pcbgroup wildcard list */
  #define	INP_REUSEPORT		0x00000008 /* SO_REUSEPORT option is set */
  #define	INP_FREED		0x00000010 /* inp itself is not valid */
 +#define	INP_REUSEADDR		0x00000020 /* SO_REUSEADDR option is set */
  
  /*
   * Flags passed to in_pcblookup*() functions.
 
 Modified: head/sys/netinet/ip_output.c
 ==============================================================================
 --- head/sys/netinet/ip_output.c	Thu Jul  4 18:00:27 2013	(r252709)
 +++ head/sys/netinet/ip_output.c	Thu Jul  4 18:38:00 2013	(r252710)
 @@ -900,13 +900,10 @@ ip_ctloutput(struct socket *so, struct s
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(inp);
 -				if (IN_MULTICAST(ntohl(inp->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						inp->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						inp->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					inp->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					inp->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(inp);
  				error = 0;
  				break;
 
 Modified: head/sys/netinet6/in6_pcb.c
 ==============================================================================
 --- head/sys/netinet6/in6_pcb.c	Thu Jul  4 18:00:27 2013	(r252709)
 +++ head/sys/netinet6/in6_pcb.c	Thu Jul  4 18:38:00 2013	(r252710)
 @@ -243,8 +243,7 @@ in6_pcbbind(register struct inpcb *inp, 
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t && (reuseport & inp_so_options(t)) == 0) {
  				return (EADDRINUSE);
  			}
  #ifdef INET
 @@ -265,8 +264,8 @@ in6_pcbbind(register struct inpcb *inp, 
  					     INP_IPV6PROTO) ==
  					     (t->inp_vflag & INP_IPV6PROTO))))
  						return (EADDRINUSE);
 -				} else if (t && (reuseport == 0 ||
 -				    (t->inp_flags2 & INP_REUSEPORT) == 0) &&
 +				} else if (t &&
 +				    (reuseport & inp_so_options(t)) == 0 &&
  				    (ntohl(t->inp_laddr.s_addr) != INADDR_ANY ||
  				    (t->inp_vflag & INP_IPV6PROTO) != 0))
  					return (EADDRINUSE);
 
 Modified: head/sys/netinet6/ip6_output.c
 ==============================================================================
 --- head/sys/netinet6/ip6_output.c	Thu Jul  4 18:00:27 2013	(r252709)
 +++ head/sys/netinet6/ip6_output.c	Thu Jul  4 18:38:00 2013	(r252710)
 @@ -1477,13 +1477,10 @@ ip6_ctloutput(struct socket *so, struct 
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(in6p);
 -				if (IN_MULTICAST(ntohl(in6p->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						in6p->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						in6p->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					in6p->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					in6p->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(in6p);
  				error = 0;
  				break;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179901: commit references a PR
Date: Fri, 12 Jul 2013 18:54:56 +0000 (UTC)

 Author: trociny
 Date: Fri Jul 12 18:54:47 2013
 New Revision: 253281
 URL: http://svnweb.freebsd.org/changeset/base/253281
 
 Log:
   MFC r252710:
   
   In r227207, to fix the issue with possible NULL inp_socket pointer
   dereferencing, when checking for SO_REUSEPORT option (and SO_REUSEADDR
   for multicast), INP_REUSEPORT flag was introduced to cache the socket
   option.  It was decided then that one flag would be enough to cache
   both SO_REUSEPORT and SO_REUSEADDR: when processing SO_REUSEADDR
   setsockopt(2), it was checked if it was called for a multicast address
   and INP_REUSEPORT was set accordingly.
   
   Unfortunately that approach does not work when setsockopt(2) is called
   before binding to a multicast address: the multicast check fails and
   INP_REUSEPORT is not set.
   
   Fix this by adding INP_REUSEADDR flag to unconditionally cache
   SO_REUSEADDR.
   
   PR:		179901
   Submitted by:	Michael Gmelin freebsd grem.de (initial version)
   Reviewed by:	rwatson
   Approved by:	re (kib)
 
 Modified:
   stable/9/sys/netinet/in_pcb.c
   stable/9/sys/netinet/in_pcb.h
   stable/9/sys/netinet/ip_output.c
   stable/9/sys/netinet6/in6_pcb.c
   stable/9/sys/netinet6/ip6_output.c
 Directory Properties:
   stable/9/sys/   (props changed)
 
 Modified: stable/9/sys/netinet/in_pcb.c
 ==============================================================================
 --- stable/9/sys/netinet/in_pcb.c	Fri Jul 12 18:52:33 2013	(r253280)
 +++ stable/9/sys/netinet/in_pcb.c	Fri Jul 12 18:54:47 2013	(r253281)
 @@ -466,6 +466,23 @@ in_pcb_lport(struct inpcb *inp, struct i
  
  	return (0);
  }
 +
 +/*
 + * Return cached socket options.
 + */
 +short
 +inp_so_options(const struct inpcb *inp)
 +{
 +   short so_options;
 +
 +   so_options = 0;
 +
 +   if ((inp->inp_flags2 & INP_REUSEPORT) != 0)
 +	   so_options |= SO_REUSEPORT;
 +   if ((inp->inp_flags2 & INP_REUSEADDR) != 0)
 +	   so_options |= SO_REUSEADDR;
 +   return (so_options);
 +}
  #endif /* INET || INET6 */
  
  #ifdef INET
 @@ -594,8 +611,7 @@ in_pcbbind_setup(struct inpcb *inp, stru
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t && (reuseport & inp_so_options(t)) == 0) {
  #ifdef INET6
  				if (ntohl(sin->sin_addr.s_addr) !=
  				    INADDR_ANY ||
 
 Modified: stable/9/sys/netinet/in_pcb.h
 ==============================================================================
 --- stable/9/sys/netinet/in_pcb.h	Fri Jul 12 18:52:33 2013	(r253280)
 +++ stable/9/sys/netinet/in_pcb.h	Fri Jul 12 18:54:47 2013	(r253281)
 @@ -442,6 +442,7 @@ struct tcpcb *
  	inp_inpcbtotcpcb(struct inpcb *inp);
  void 	inp_4tuple_get(struct inpcb *inp, uint32_t *laddr, uint16_t *lp,
  		uint32_t *faddr, uint16_t *fp);
 +short	inp_so_options(const struct inpcb *inp);
  
  #endif /* _KERNEL */
  
 @@ -543,6 +544,7 @@ void 	inp_4tuple_get(struct inpcb *inp, 
  #define	INP_PCBGROUPWILD	0x00000004 /* in pcbgroup wildcard list */
  #define	INP_REUSEPORT		0x00000008 /* SO_REUSEPORT option is set */
  #define	INP_FREED		0x00000010 /* inp itself is not valid */
 +#define	INP_REUSEADDR		0x00000020 /* SO_REUSEADDR option is set */
  
  /*
   * Flags passed to in_pcblookup*() functions.
 
 Modified: stable/9/sys/netinet/ip_output.c
 ==============================================================================
 --- stable/9/sys/netinet/ip_output.c	Fri Jul 12 18:52:33 2013	(r253280)
 +++ stable/9/sys/netinet/ip_output.c	Fri Jul 12 18:54:47 2013	(r253281)
 @@ -899,13 +899,10 @@ ip_ctloutput(struct socket *so, struct s
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(inp);
 -				if (IN_MULTICAST(ntohl(inp->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						inp->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						inp->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					inp->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					inp->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(inp);
  				error = 0;
  				break;
 
 Modified: stable/9/sys/netinet6/in6_pcb.c
 ==============================================================================
 --- stable/9/sys/netinet6/in6_pcb.c	Fri Jul 12 18:52:33 2013	(r253280)
 +++ stable/9/sys/netinet6/in6_pcb.c	Fri Jul 12 18:54:47 2013	(r253281)
 @@ -245,8 +245,7 @@ in6_pcbbind(register struct inpcb *inp, 
  				if (tw == NULL ||
  				    (reuseport & tw->tw_so_options) == 0)
  					return (EADDRINUSE);
 -			} else if (t && (reuseport == 0 ||
 -			    (t->inp_flags2 & INP_REUSEPORT) == 0)) {
 +			} else if (t && (reuseport & inp_so_options(t)) == 0) {
  				return (EADDRINUSE);
  			}
  #ifdef INET
 @@ -267,8 +266,8 @@ in6_pcbbind(register struct inpcb *inp, 
  					     INP_IPV6PROTO) ==
  					     (t->inp_vflag & INP_IPV6PROTO))))
  						return (EADDRINUSE);
 -				} else if (t && (reuseport == 0 ||
 -				    (t->inp_flags2 & INP_REUSEPORT) == 0) &&
 +				} else if (t &&
 +				    (reuseport & inp_so_options(t)) == 0 &&
  				    (ntohl(t->inp_laddr.s_addr) != INADDR_ANY ||
  				    (t->inp_vflag & INP_IPV6PROTO) != 0))
  					return (EADDRINUSE);
 
 Modified: stable/9/sys/netinet6/ip6_output.c
 ==============================================================================
 --- stable/9/sys/netinet6/ip6_output.c	Fri Jul 12 18:52:33 2013	(r253280)
 +++ stable/9/sys/netinet6/ip6_output.c	Fri Jul 12 18:54:47 2013	(r253281)
 @@ -1491,13 +1491,10 @@ ip6_ctloutput(struct socket *so, struct 
  			switch (sopt->sopt_name) {
  			case SO_REUSEADDR:
  				INP_WLOCK(in6p);
 -				if (IN_MULTICAST(ntohl(in6p->inp_laddr.s_addr))) {
 -					if ((so->so_options &
 -					    (SO_REUSEADDR | SO_REUSEPORT)) != 0)
 -						in6p->inp_flags2 |= INP_REUSEPORT;
 -					else
 -						in6p->inp_flags2 &= ~INP_REUSEPORT;
 -				}
 +				if ((so->so_options & SO_REUSEADDR) != 0)
 +					in6p->inp_flags2 |= INP_REUSEADDR;
 +				else
 +					in6p->inp_flags2 &= ~INP_REUSEADDR;
  				INP_WUNLOCK(in6p);
  				error = 0;
  				break;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179901: commit references a PR
Date: Fri, 12 Jul 2013 19:08:41 +0000 (UTC)

 Author: trociny
 Date: Fri Jul 12 19:08:33 2013
 New Revision: 253282
 URL: http://svnweb.freebsd.org/changeset/base/253282
 
 Log:
   A complete duplication of binding should be allowed if on both new and
   duplicated sockets a multicast address is bound and either
   SO_REUSEPORT or SO_REUSEADDR is set.
   
   But actually it works for the following combinations:
   
     * SO_REUSEPORT is set for the fist socket and SO_REUSEPORT for the new;
     * SO_REUSEADDR is set for the fist socket and SO_REUSEADDR for the new;
     * SO_REUSEPORT is set for the fist socket and SO_REUSEADDR for the new;
   
   and fails for this:
   
     * SO_REUSEADDR is set for the fist socket and SO_REUSEPORT for the new.
   
   Fix the last case.
   
   PR:		179901
   MFC after:	1 month
 
 Modified:
   head/sys/netinet/in_pcb.c
   head/sys/netinet6/in6_pcb.c
 
 Modified: head/sys/netinet/in_pcb.c
 ==============================================================================
 --- head/sys/netinet/in_pcb.c	Fri Jul 12 18:54:47 2013	(r253281)
 +++ head/sys/netinet/in_pcb.c	Fri Jul 12 19:08:33 2013	(r253282)
 @@ -554,7 +554,7 @@ in_pcbbind_setup(struct inpcb *inp, stru
  			 * and a multicast address is bound on both
  			 * new and duplicated sockets.
  			 */
 -			if (so->so_options & SO_REUSEADDR)
 +			if ((so->so_options & (SO_REUSEADDR|SO_REUSEPORT)) != 0)
  				reuseport = SO_REUSEADDR|SO_REUSEPORT;
  		} else if (sin->sin_addr.s_addr != INADDR_ANY) {
  			sin->sin_port = 0;		/* yech... */
 
 Modified: head/sys/netinet6/in6_pcb.c
 ==============================================================================
 --- head/sys/netinet6/in6_pcb.c	Fri Jul 12 18:54:47 2013	(r253281)
 +++ head/sys/netinet6/in6_pcb.c	Fri Jul 12 19:08:33 2013	(r253282)
 @@ -156,7 +156,7 @@ in6_pcbbind(register struct inpcb *inp, 
  			 * and a multicast address is bound on both
  			 * new and duplicated sockets.
  			 */
 -			if (so->so_options & SO_REUSEADDR)
 +			if ((so->so_options & (SO_REUSEADDR|SO_REUSEPORT)) != 0)
  				reuseport = SO_REUSEADDR|SO_REUSEPORT;
  		} else if (!IN6_IS_ADDR_UNSPECIFIED(&sin6->sin6_addr)) {
  			struct ifaddr *ifa;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: open->patched 
State-Changed-By: trociny 
State-Changed-When: Sun Jul 14 15:18:10 UTC 2013 
State-Changed-Why:  
The fix is committed. 


Responsible-Changed-From-To: freebsd-net->trociny 
Responsible-Changed-By: trociny 
Responsible-Changed-When: Sun Jul 14 15:18:10 UTC 2013 
Responsible-Changed-Why:  
Will close after MFC r253282. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=179901 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: kern/179901: commit references a PR
Date: Sat, 24 Aug 2013 12:10:18 +0000 (UTC)

 Author: trociny
 Date: Sat Aug 24 12:10:11 2013
 New Revision: 254778
 URL: http://svnweb.freebsd.org/changeset/base/254778
 
 Log:
   MFC r253282:
   
   A complete duplication of binding should be allowed if on both new and
   duplicated sockets a multicast address is bound and either
   SO_REUSEPORT or SO_REUSEADDR is set.
   
   But actually it works for the following combinations:
   
     * SO_REUSEPORT is set for the fist socket and SO_REUSEPORT for the new;
     * SO_REUSEADDR is set for the fist socket and SO_REUSEADDR for the new;
     * SO_REUSEPORT is set for the fist socket and SO_REUSEADDR for the new;
   
   and fails for this:
   
     * SO_REUSEADDR is set for the fist socket and SO_REUSEPORT for the new.
   
   Fix the last case.
   
   PR:		179901
 
 Modified:
   stable/9/sys/netinet/in_pcb.c
   stable/9/sys/netinet6/in6_pcb.c
 Directory Properties:
   stable/9/sys/   (props changed)
 
 Modified: stable/9/sys/netinet/in_pcb.c
 ==============================================================================
 --- stable/9/sys/netinet/in_pcb.c	Sat Aug 24 12:03:24 2013	(r254777)
 +++ stable/9/sys/netinet/in_pcb.c	Sat Aug 24 12:10:11 2013	(r254778)
 @@ -553,7 +553,7 @@ in_pcbbind_setup(struct inpcb *inp, stru
  			 * and a multicast address is bound on both
  			 * new and duplicated sockets.
  			 */
 -			if (so->so_options & SO_REUSEADDR)
 +			if ((so->so_options & (SO_REUSEADDR|SO_REUSEPORT)) != 0)
  				reuseport = SO_REUSEADDR|SO_REUSEPORT;
  		} else if (sin->sin_addr.s_addr != INADDR_ANY) {
  			sin->sin_port = 0;		/* yech... */
 
 Modified: stable/9/sys/netinet6/in6_pcb.c
 ==============================================================================
 --- stable/9/sys/netinet6/in6_pcb.c	Sat Aug 24 12:03:24 2013	(r254777)
 +++ stable/9/sys/netinet6/in6_pcb.c	Sat Aug 24 12:10:11 2013	(r254778)
 @@ -158,7 +158,7 @@ in6_pcbbind(register struct inpcb *inp, 
  			 * and a multicast address is bound on both
  			 * new and duplicated sockets.
  			 */
 -			if (so->so_options & SO_REUSEADDR)
 +			if ((so->so_options & (SO_REUSEADDR|SO_REUSEPORT)) != 0)
  				reuseport = SO_REUSEADDR|SO_REUSEPORT;
  		} else if (!IN6_IS_ADDR_UNSPECIFIED(&sin6->sin6_addr)) {
  			struct ifaddr *ifa;
 _______________________________________________
 svn-src-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/svn-src-all
 To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org"
 
State-Changed-From-To: patched->closed 
State-Changed-By: trociny 
State-Changed-When: Sun Aug 25 19:02:56 UTC 2013 
State-Changed-Why:  
Fixed. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=179901 
>Unformatted:
