From jot.3.brinegar@spamgourmet.com  Fri Jul 25 23:28:33 2003
Return-Path: <jot.3.brinegar@spamgourmet.com>
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id 3B31237B401
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Jul 2003 23:28:33 -0700 (PDT)
Received: from brinegar-computing.com (adsl-64-170-154-150.dsl.sntc01.pacbell.net [64.170.154.150])
	by mx1.FreeBSD.org (Postfix) with SMTP id 9C63843F75
	for <FreeBSD-gnats-submit@freebsd.org>; Fri, 25 Jul 2003 23:28:32 -0700 (PDT)
	(envelope-from jot.3.brinegar@spamgourmet.com)
Received: (qmail 5316 invoked by uid 1000); 26 Jul 2003 06:28:31 -0000
Message-Id: <20030726062831.5315.qmail@brinegar-computing.com>
Date: 26 Jul 2003 06:28:31 -0000
From: "David Brinegar" <jot.3.brinegar@spamgourmet.com>
To: FreeBSD-gnats-submit@freebsd.org
Cc:
Subject: incorrect divisor in /usr/bin/jot -r
X-Send-Pr-Version: 3.113
X-GNATS-Notify:

>Number:         54878
>Category:       bin
>Synopsis:       incorrect divisor in /usr/bin/jot -r
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    dds
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Fri Jul 25 23:30:09 PDT 2003
>Closed-Date:    Mon Nov 27 09:05:17 GMT 2006
>Last-Modified:  Mon Nov 27 09:05:17 GMT 2006
>Originator:     David Brinegar
>Release:        FreeBSD 4.6-RELEASE i386
>Organization:
>Environment:
Any FreeBSD machine.


	
>Description:
	
   src/usr.bin/jot/jot.c uses incorrect divisor.

   revision 1.24, line 278:

      *y = arc4random() / (double)UINT32_MAX;

   will 1 time in 2^32 assign 1.0 to *y, creating a distribution
   of [0,1] instead of the intended [0,1) spread.

   For example one would expect something like the following:

   > jot -w %d -r 1000 1 4 | sort -n | uniq -c
    333 1
    333 2
    334 3

   Internally, jot is assigning *y to 1,2,3, and very rarely 4:

      [1.0,2.0) => 1
      [2.0,3.0) => 2
      [3.0,4.0) => 3
           4.0  => 4

   So this bug creates the remote possiblity of something like:

   > jot -w %d -r 1000 1 4 | sort -n | uniq -c
    333 1
    333 2
    333 3
      1 4


>How-To-Repeat:
	

   jot -w $d -r 0 1 4 | grep 4 | head -1

   and wait a potentially very long time. :-)

>Fix:

	
   src/usr.bin/jot/jot.c revision 1.24, line 278:

      *y = arc4random() / (double)UINT32_MAX;

   should be:

      *y = arc4random() / (1.0 + (double)UINT32_MAX);

   and similar for other revisions, where the divisor should be
   one more than the maximum *random() function return value.

>Release-Note:
>Audit-Trail:

From: David Schultz <das@FreeBSD.ORG>
To: David Brinegar <jot.3.brinegar@spamgourmet.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: bin/54878: incorrect divisor in /usr/bin/jot -r
Date: Wed, 30 Jul 2003 23:15:43 -0700

 On Sat, Jul 26, 2003, David Brinegar wrote:
 >    For example one would expect something like the following:
 > 
 >    > jot -w %d -r 1000 1 4 | sort -n | uniq -c
 >     333 1
 >     333 2
 >     334 3
 > 
 >    Internally, jot is assigning *y to 1,2,3, and very rarely 4:
 > 
 >       [1.0,2.0) => 1
 >       [2.0,3.0) => 2
 >       [3.0,4.0) => 3
 >            4.0  => 4
 [...]
 >    src/usr.bin/jot/jot.c revision 1.24, line 278:
 > 
 >       *y = arc4random() / (double)UINT32_MAX;
 > 
 >    should be:
 > 
 >       *y = arc4random() / (1.0 + (double)UINT32_MAX);
 
 Actually, to be compatible with the non-random behavior, and to
 make the random letter example in the manpage actually work, jot
 needs to treat integers and floating point numbers differently.
 In particular, 'jot -w %d -r 1000 1 4' needs to give integers
 uniformly distrubited over [1,4], whereas 'jot -w %f -r 1000 1 4'
 needs to give floating point numbers uniformly distributed over
 the same range.  As you rightly point out, the integer
 distribution is skewed if you take a number over the floating
 point distribution and round down.
 
 On a related note, jot should be retrofitted to use fmtcheck(3)
 anyway...

From: "David Brinegar" <jot.3.brinegar@spamgourmet.com>
To: "David Schultz  - das@FreeBSD.ORG" <+jot+brinegar+c45043f3f4.das#FreeBSD.ORG@spamgourmet.com>
Cc: FreeBSD-gnats-submit@FreeBSD.ORG
Subject: Re: bin/54878: incorrect divisor in /usr/bin/jot -r
Date: Thu, 31 Jul 2003 09:32:35 -0700

 David Schultz wrote:
 > Actually, to be compatible with the non-random behavior, and to
 > make the random letter example in the manpage actually work, jot
 > needs to treat integers and floating point numbers differently.
 > In particular, 'jot -w %d -r 1000 1 4' needs to give integers
 > uniformly distrubited over [1,4], whereas 'jot -w %f -r 1000 1 4'
 > needs to give floating point numbers uniformly distributed over
 > the same range.
 
 Well, I wonder how many people use the current distribution on
 purpose.  We have a few scripts around here that would need to be
 adjusted to the new range. If someone is using jot to pick 1, 2, or
 3 and after an upgrade it starts picking 4 as well, then the script
 will randomly not work.
 
 I sent a pr to change the man page for these issues, thinking that
 the path of least change is just to point out what is happening so
 that people who haven't already figured it out can get a clue from
 the man page.
 
 Correcting the arc4random() divisor will not impact anyone, as it
 only prevents a problem with arc4random() == 0xFFFFFFFF.
 
 -- 
 David Brinegar

From: David Schultz <das@freebsd.org>
To: jot.3.brinegar@spamgourmet.com
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/54878: incorrect divisor in /usr/bin/jot -r
Date: Thu, 31 Jul 2003 11:26:25 -0700

 On Thu, Jul 31, 2003, jot.3.brinegar@spamgourmet.com wrote:
 > David Schultz wrote:
 > > Actually, to be compatible with the non-random behavior, and to
 > > make the random letter example in the manpage actually work, jot
 > > needs to treat integers and floating point numbers differently.
 > > In particular, 'jot -w %d -r 1000 1 4' needs to give integers
 > > uniformly distrubited over [1,4], whereas 'jot -w %f -r 1000 1 4'
 > > needs to give floating point numbers uniformly distributed over
 > > the same range.
 > 
 > Well, I wonder how many people use the current distribution on
 > purpose.  We have a few scripts around here that would need to be
 > adjusted to the new range. If someone is using jot to pick 1, 2, or
 > 3 and after an upgrade it starts picking 4 as well, then the script
 > will randomly not work.
 > 
 > I sent a pr to change the man page for these issues, thinking that
 > the path of least change is just to point out what is happening so
 > that people who haven't already figured it out can get a clue from
 > the man page.
 > 
 > Correcting the arc4random() divisor will not impact anyone, as it
 > only prevents a problem with arc4random() == 0xFFFFFFFF.
 
 Some people have also invoked jot in ways that anticipate the
 correct behavior, too.  The manpage author was one of them!  The
 program appears to be more significantly broken than I originally
 thought.  Is there a good explanation for the following behavior?
 
 das@HAL9000:~> jot 8 1 2
 1
 1
 1
 1
 2
 2
 2
 2
 das@HAL9000:~> jot -w %d 8 1 2
 1
 1
 1
 1
 1
 1
 1
 1
 das@HAL9000:~> jot -w %f 8 1 2
 1.000000
 1.142857
 1.285714
 1.428571
 1.571429
 1.714286
 1.857143
 2.000000
 das@HAL9000:~> jot -r 8 1 2
 2
 1
 1
 1
 1
 2
 1
 1
 das@HAL9000:~> jot -r -w %d 8 1 2
 1
 1
 1
 1
 1
 1
 1
 1

From: "David Brinegar" <jot.3.brinegar@spamgourmet.com>
To: "David Schultz  - das@freebsd.org" <+jot+brinegar+c45043f3f4.das#freebsd.org@spamgourmet.com>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/54878: incorrect divisor in /usr/bin/jot -r
Date: Thu, 31 Jul 2003 12:52:36 -0700

 David Schultz wrote:
 > Some people have also invoked jot in ways that anticipate the
 > correct behavior, too.  The manpage author was one of them!  The
 > program appears to be more significantly broken than I originally
 > thought.  Is there a good explanation for the following behavior?
 
 I agree with your sentiment.  I just don't know how much such
 corrections would break existing usage or portability.  Perhaps the
 correction should be under a new flag, say -R, so that the -r flag
 is bug for bug compatible.
 
 Explanation of behavior (not necessarily good):
 
    The default -w format is %.0f, which prints a rounded floating
    point value.  This cannot produce a uniform random distribution
    so is probably not suitable for use with -r, as both endpoints of
    the distribution will only be half as likely as other points.
 
    The -r output goes up to but does not include the specified
    endpoint.  jot -r -w %d n 1 4 will not print a 4.  jot -r n 1 4
    will, as explained above, round [3.5-4.0) to 4 even though 4.0 is
    not generated before rounding.
 
 More explanation at:
 
    http://www.freebsd.org/cgi/query-pr.cgi?pr=docs/54879
 
 -- 
 David Brinegar

From: David Schultz <das@freebsd.org>
To: jot.3.brinegar@spamgourmet.com
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/54878: incorrect divisor in /usr/bin/jot -r
Date: Thu, 31 Jul 2003 23:42:21 -0700

 On Thu, Jul 31, 2003, jot.3.brinegar@spamgourmet.com wrote:
 > David Schultz wrote:
 > > Some people have also invoked jot in ways that anticipate the
 > > correct behavior, too.  The manpage author was one of them!  The
 > > program appears to be more significantly broken than I originally
 > > thought.  Is there a good explanation for the following behavior?
 > 
 > I agree with your sentiment.  I just don't know how much such
 > corrections would break existing usage or portability.  Perhaps the
 > correction should be under a new flag, say -R, so that the -r flag
 > is bug for bug compatible.
 
 On the one hand, it's hard to justify breaking POLA, and on the
 other hand, this program has obvious bugs and gross
 inconsistencies between its documented and actual behavior.
 If I had the time for it, I'd probably do something sick and
 twisted and write a replacement called delta(1).  ;-)
 In the mean time, I'm happy to apply your suggested change if
 you like, but that would only make jot a little bit less wrong.
 
 > Explanation of behavior (not necessarily good):
 [...]
 
 Thanks.  I understand the rounding; I was really looking for a
 rationalization of the behavior.  Most likely there isn't one.  :-P

From: "David Brinegar" <jot.3.brinegar@spamgourmet.com>
To: David Schultz <das@freebsd.org>
Cc: FreeBSD-gnats-submit@freebsd.org
Subject: Re: bin/54878: incorrect divisor in /usr/bin/jot -r
Date: Fri, 1 Aug 2003 08:46:17 -0700

 David Schultz wrote:
 > In the mean time, I'm happy to apply your suggested change if
 > you like, but that would only make jot a little bit less wrong.
 
 Great!  Fixing the divisor at least eliminates the one crazy error
 with no easy work around.
 
 Hopefully someone in doc-bug will pick up pr docs/54879 and fix the
 man page so that folks know how to work around the printf stuff.
 
 Thanks again.
 
 -- 
 David Brinegar
State-Changed-From-To: open->suspended 
State-Changed-By: das 
State-Changed-When: Fri Aug 1 09:23:24 PDT 2003 
State-Changed-Why:  
Your suggested change has been committed, thanks!  I'm throwing this 
on the back burner in hopes that someone will take interest in the 
other issues raised in this PR. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=54878 
State-Changed-From-To: suspended->patched 
State-Changed-By: dds 
State-Changed-When: Mon Nov 6 14:03:08 UTC 2006 
State-Changed-Why:  
Revision 1.32 of jot.c and 1.21 of jot.1 fix the problem of the 
random number distribution. 


http://www.freebsd.org/cgi/query-pr.cgi?pr=54878 
Responsible-Changed-From-To: freebsd-bugs->dds 
Responsible-Changed-By: dds 
Responsible-Changed-When: Mon Nov 6 14:06:43 UTC 2006 
Responsible-Changed-Why:  
Patched source. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=54878 
State-Changed-From-To: patched->closed 
State-Changed-By: dds 
State-Changed-When: Mon Nov 27 09:03:41 UTC 2006 
State-Changed-Why:  
MFCd fixes to RELENG_6. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=54878 
>Unformatted:
