From nobody@FreeBSD.org  Thu Nov 13 16:30:47 2008
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 6F01E106568E
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 13 Nov 2008 16:30:47 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21])
	by mx1.freebsd.org (Postfix) with ESMTP id 651628FC1D
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 13 Nov 2008 16:30:47 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from www.freebsd.org (localhost [127.0.0.1])
	by www.freebsd.org (8.14.3/8.14.3) with ESMTP id mADGUlNO097772
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 13 Nov 2008 16:30:47 GMT
	(envelope-from nobody@www.freebsd.org)
Received: (from nobody@localhost)
	by www.freebsd.org (8.14.3/8.14.3/Submit) id mADGUlLv097771;
	Thu, 13 Nov 2008 16:30:47 GMT
	(envelope-from nobody)
Message-Id: <200811131630.mADGUlLv097771@www.freebsd.org>
Date: Thu, 13 Nov 2008 16:30:47 GMT
From: Steven Kreuzer <skreuzer@exit2shell.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: devel/scons has race condition
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         128845
>Category:       ports
>Synopsis:       devel/scons has race condition
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    pgollucci
>State:          closed
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Thu Nov 13 16:40:05 UTC 2008
>Closed-Date:    Thu Dec 25 14:57:23 UTC 2008
>Last-Modified:  Thu Dec 25 15:00:08 UTC 2008
>Originator:     Steven Kreuzer
>Release:        6.3-STABLE
>Organization:
>Environment:
FreeBSD slurry.exit2shell.com 6.3-STABLE FreeBSD 6.3-STABLE #1: Fri May 23 20:22:28 EDT 2008 root@biggierat.sddi.net:/usr/obj/usr/src/sys/SMP amd64
>Description:
Python's subprocess module has a race condition: Popen() constructor has a call to global "_cleanup()" function on whenever a Popen object gets created, and that call causes a check for all pending Popen objects whether their subprocess has exited - i.e. the poll() method is called for every active Popen object.

See http://bugs.python.org/issue1731717 for addition details

SCon's compat/_scons_subprocess.py module is just a copy of a more recent
stock Python subprocess.py modified so it will work with older Python
versions.

The attached patch will add locks around calls to Popen and change
the compat module in a way that the subprocess module is always used, no matter if Python already ships one.

The rationale behind this decision is that there are many Python versions in the wild with different Popen() race condition problems. 
>How-To-Repeat:
Run scons -j8 on a quad core machine
>Fix:


Patch attached with submission follows:

Index: Makefile
===================================================================
RCS file: /usr/share/cvs/freebsd/ports/devel/scons/Makefile,v
retrieving revision 1.33
diff -u -r1.33 Makefile
--- Makefile	29 Oct 2008 00:52:10 -0000	1.33
+++ Makefile	13 Nov 2008 15:39:15 -0000
@@ -7,6 +7,7 @@
 
 PORTNAME=	scons
 PORTVERSION=	1.1.0
+PORTREVISION=	1
 CATEGORIES=	devel python
 MASTER_SITES=	SF
 
--- /dev/null	2008-11-13 11:09:02.000000000 -0500
+++ files/patch-engine-SCons-compat-__init__.py	2008-11-13 11:03:17.064635000 -0500
@@ -0,0 +1,29 @@
+Index: engine/SCons/compat/__init__.py
+===================================================================
+--- engine/SCons/compat/__init__.py (revision 2695)
++++ engine/SCons/compat/__init__.py (working copy)
+@@ -167,11 +167,17 @@
+     del shlex
+     import_as('_scons_shlex', 'shlex')
+ 
+-try:
+-    import subprocess
+-except ImportError:
+-    # Pre-2.4 Python has no subprocess module.
+-    import_as('_scons_subprocess', 'subprocess')
++#try:
++#    import subprocess
++#except ImportError:
++#    # Pre-2.4 Python has no subprocess module.
++#    import_as('_scons_subprocess', 'subprocess')
++
++# Import subprocess unconditionally to avoid possible race conditions in
++# the official subprocess API. If there are API versions without known
++# problems, we can version-check and use the original subprocess module
++# in these cases.
++import_as('_scons_subprocess', 'subprocess')
+ 
+ import sys
+ try:
+
+
--- /dev/null	2008-11-13 11:18:00.000000000 -0500
+++ files/patch-engine-SCons-compat-_scons_subprocess.py	2008-11-13 11:01:02.109982000 -0500
@@ -0,0 +1,33 @@
+Index: engine/SCons/compat/_scons_subprocess.py
+===================================================================
+--- engine/SCons/compat/_scons_subprocess.py (revision 2695)
++++ engine/SCons/compat/_scons_subprocess.py (working copy)
+@@ -581,13 +581,19 @@
+     class object:
+         pass
+ 
++import thread
++lock = thread.allocate_lock()
++
+ class Popen(object):
+     def __init__(self, args, bufsize=0, executable=None,
+                  stdin=None, stdout=None, stderr=None,
+                  preexec_fn=None, close_fds=False, shell=False,
+                  cwd=None, env=None, universal_newlines=False,
+                  startupinfo=None, creationflags=0):
+-        """Create new Popen instance."""
++        """Create new Popen instance.
++        Popen is not thread-safe and is therefore protected with a lock.
++        """
++        lock.acquire()
+         _cleanup()
+ 
+         self._child_created = False
+@@ -655,6 +661,7 @@
+                 self.stderr = os.fdopen(errread, 'rU', bufsize)
+             else:
+                 self.stderr = os.fdopen(errread, 'rb', bufsize)
++        lock.release()
+ 
+ 
+     def _translate_newlines(self, data):


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->pgollucci 
Responsible-Changed-By: edwin 
Responsible-Changed-When: Thu Nov 13 16:40:16 UTC 2008 
Responsible-Changed-Why:  
Over to maintainer (via the GNATS Auto Assign Tool) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=128845 
State-Changed-From-To: open->closed 
State-Changed-By: pgollucci 
State-Changed-When: Thu Dec 25 14:57:22 UTC 2008 
State-Changed-Why:  
Committed. Thanks! 

http://www.freebsd.org/cgi/query-pr.cgi?pr=128845 

From: dfilter@FreeBSD.ORG (dfilter service)
To: bug-followup@FreeBSD.org
Cc:  
Subject: Re: ports/128845: commit references a PR
Date: Thu, 25 Dec 2008 14:57:22 +0000 (UTC)

 pgollucci    2008-12-25 14:57:06 UTC
 
   FreeBSD ports repository
 
   Modified files:
     devel/scons          Makefile 
   Added files:
     devel/scons/files    patch-engine-SCons-compat-__init__.py 
                          patch-engine-SCons-compat-_scons_subprocess.py 
   Log:
   Python's subprocess module has a race condition: Popen() constructor has a call to global "_cleanup()" function on whenever a Popen object gets created, and that call causes a check for all pending Popen objects whether their subprocess has exited - i.e. the poll() method is called for every active Popen object.
   
   See http://bugs.python.org/issue1731717 for addition details
   
   SCon's compat/_scons_subprocess.py module is just a copy of a more recent
   stock Python subprocess.py modified so it will work with older Python
   versions.
   
   The attached patch will add locks around calls to Popen and change
   the compat module in a way that the subprocess module is always used, no matter if Python already ships one.
   
   The rationale behind this decision is that there are many Python versions in the wild with different Popen() race condition problems.
   
   PR:             ports/128845
   Submitted by:   Steven Kreuzer <skreuzer@exit2shell.com>
   Approved by:  araujo (mentor, implicit)
   
   Revision  Changes    Path
   1.35      +1 -0      ports/devel/scons/Makefile
   1.1       +29 -0     ports/devel/scons/files/patch-engine-SCons-compat-__init__.py (new)
   1.1       +33 -0     ports/devel/scons/files/patch-engine-SCons-compat-_scons_subprocess.py (new)
 _______________________________________________
 cvs-all@freebsd.org mailing list
 http://lists.freebsd.org/mailman/listinfo/cvs-all
 To unsubscribe, send any mail to "cvs-all-unsubscribe@freebsd.org"
 
>Unformatted:
