From nobody@FreeBSD.org  Sun Sep  2 10:35:59 2012
Return-Path: <nobody@FreeBSD.org>
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 17B33106566B
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  2 Sep 2012 10:35:59 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id EB91C8FC08
	for <freebsd-gnats-submit@FreeBSD.org>; Sun,  2 Sep 2012 10:35:58 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q82AZwcg017324
	for <freebsd-gnats-submit@FreeBSD.org>; Sun, 2 Sep 2012 10:35:58 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q82AZwdJ017319;
	Sun, 2 Sep 2012 10:35:58 GMT
	(envelope-from nobody)
Message-Id: <201209021035.q82AZwdJ017319@red.freebsd.org>
Date: Sun, 2 Sep 2012 10:35:58 GMT
From: Vitaly Magerya <vmagerya@gmail.com>
To: freebsd-gnats-submit@FreeBSD.org
Subject: [patch] make python curses module work with unicode
X-Send-Pr-Version: www-3.1
X-GNATS-Notify:

>Number:         171246
>Category:       ports
>Synopsis:       [patch] lang/python27: make python curses module work with unicode
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    koobs
>State:          feedback
>Quarter:        
>Keywords:       
>Date-Required:  
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sun Sep 02 10:40:07 UTC 2012
>Closed-Date:    
>Last-Modified:  Fri Nov  1 16:20:00 UTC 2013
>Originator:     Vitaly Magerya
>Release:        FreeBSD 9.1-RC1 amd64
>Organization:
>Environment:
>Description:
I'm having a problem with Python curses module: it
won't work with unicode. Here's a test case:

    import curses
    import locale

    locale.setlocale(locale.LC_ALL, "")

    def run(win):
        win.addstr(u"\u03c0r\u00b2".encode("utf-8"))
        win.getch()

    curses.wrapper(run)

What I expect to see (and what I in fact see on e.g. Linux) is
"&#960;r" (that is, Greek "pi", Latin "r", superscript "2"), what I
actually get is "M-O~@rM-BM-2".

(Note that this is with LC_ALL set to "en_US.UTF-8" everywhere;
I'm seeing the same output in uxterm and putty, with and without
tmux inside of those).

I think this happens because python curses library is linked
with libncurses, not with libncursesw as it is on other platforms:

    $ ldd /usr/local/lib/python2.7/lib-dynload/_curses.so
    /usr/local/lib/python2.7/lib-dynload/_curses.so:
            libncurses.so.8 => /lib/libncurses.so.8 (0x801212000)
            libthr.so.3 => /lib/libthr.so.3 (0x80145f000)
            libc.so.7 => /lib/libc.so.7 (0x80084a000)

In fact, this appears to be intentional; here's a part of
lang/python27/files/patch-setup.py:

@@ -642,7 +642,7 @@
         # use the same library for the readline and curses modules.
         if 'curses' in readline_termcap_library:
             curses_library = readline_termcap_library
-        elif self.compiler.find_library_file(lib_dirs, 'ncursesw'):
+        elif self.compiler.find_library_file(lib_dirs, 'xxxncursesw'):
             curses_library = 'ncursesw'
         elif self.compiler.find_library_file(lib_dirs, 'ncurses'):
             curses_library = 'ncurses'
@@ -1246,12 +1248,13 @@
         # provided by the ncurses library.
         panel_library = 'panel'
         if curses_library.startswith('ncurses'):
-            if curses_library == 'ncursesw':
+            if curses_library == 'xxxncursesw':
                 # Bug 1464056: If _curses.so links with ncursesw,
                 # _curses_panel.so must link with panelw.
                 panel_library = 'panelw'

After some digging through commit logs, I found that this change
originated in [1] as a fix for ports/99496 [2].

Note that this patch is a no-op in 2.7 and in 3.2, so it can be
safely removed: a few lines above the first chunk of the patch
setup.py determines which curses library does readline use
(libncurses.so in FreeBSD), and uses that itself (unless readline
uses libtinfo, in which case python links with libncursesw).

More importantly though, if I remove all that logic and simply
set "curses_library = 'ncursesw'", thus making _curses.so link
with libncursesw.so, nothing breaks: python builds and runs fine,
and my example above works as expected too. Even if I have
devel/ncurses installed (as described in ports/99496), _curses.so
still links with libncursesw.so from base and everything seems
to work.

In short, I'm proposing a patch (it's in the attachment and also
at [3]) against lang/python{27,32} that removes all that logic
and makes _curses.so link with libncursesw from base. As you can
see from redports logs at [4], it builds fine on all releases.
I also tested patched python27 with devel/ncurses installed, and
that works too.

Final note: do contact me if there is something wrong with the
patch. I am determined to make curses work with Unicode.

[1] http://www.freebsd.org/cgi/cvsweb.cgi/ports/lang/python24/files/Attic/patch-setup.py#rev1.8
[2] http://www.freebsd.org/cgi/query-pr.cgi?pr=99496 
[3] http://redports.org/changeset/6571
[4] http://redports.org/buildarchive/20120831205350-31320/
>How-To-Repeat:

>Fix:


Patch attached with submission follows:

Index: /magv/lang/python32/files/patch-setup.py
===================================================================
--- /magv/lang/python32/files/patch-setup.py	(revision 6570)
+++ /magv/lang/python32/files/patch-setup.py	(revision 6571)
@@ -15,5 +15,5 @@
              curses_library = readline_termcap_library
 -        elif self.compiler.find_library_file(lib_dirs, 'ncursesw'):
-+        elif self.compiler.find_library_file(lib_dirs, 'XXXncursesw'):
++        if self.compiler.find_library_file(lib_dirs, 'ncursesw'):
              curses_library = 'ncursesw'
          elif self.compiler.find_library_file(lib_dirs, 'ncurses'):
@@ -32,6 +32,5 @@
          panel_library = 'panel'
          if curses_library.startswith('ncurses'):
--            if curses_library == 'ncursesw':
-+            if curses_library == 'XXXncursesw':
+             if curses_library == 'ncursesw':
                  # Bug 1464056: If _curses.so links with ncursesw,
                  # _curses_panel.so must link with panelw.
Index: /magv/lang/python27/files/patch-setup.py
===================================================================
--- /magv/lang/python27/files/patch-setup.py	(revision 6570)
+++ /magv/lang/python27/files/patch-setup.py	(revision 6571)
@@ -15,5 +15,5 @@
              curses_library = readline_termcap_library
 -        elif self.compiler.find_library_file(lib_dirs, 'ncursesw'):
-+        elif self.compiler.find_library_file(lib_dirs, 'xxxncursesw'):
++        if self.compiler.find_library_file(lib_dirs, 'ncursesw'):
              curses_library = 'ncursesw'
          elif self.compiler.find_library_file(lib_dirs, 'ncurses'):
@@ -50,6 +50,5 @@
          panel_library = 'panel'
          if curses_library.startswith('ncurses'):
--            if curses_library == 'ncursesw':
-+            if curses_library == 'xxxncursesw':
+             if curses_library == 'ncursesw':
                  # Bug 1464056: If _curses.so links with ncursesw,
                  # _curses_panel.so must link with panelw.


>Release-Note:
>Audit-Trail:
Responsible-Changed-From-To: freebsd-ports-bugs->freebsd-python 
Responsible-Changed-By: edwin 
Responsible-Changed-When: Mon Sep 10 09:19:14 UTC 2012 
Responsible-Changed-Why:  
Over to maintainer (via the GNATS Auto Assign Tool) 

http://www.freebsd.org/cgi/query-pr.cgi?pr=171246 
Responsible-Changed-From-To: freebsd-python->koobs 
Responsible-Changed-By: koobs 
Responsible-Changed-When: Thu Oct 24 02:04:55 UTC 2013 
Responsible-Changed-Why:  
I'll take it. 

http://www.freebsd.org/cgi/query-pr.cgi?pr=171246 
State-Changed-From-To: open->feedback 
State-Changed-By: koobs 
State-Changed-When: Thu Oct 24 02:43:50 UTC 2013 
State-Changed-Why:  
Request additional information and tests from submitter 

http://www.freebsd.org/cgi/query-pr.cgi?pr=171246 

From: Kubilay Kocak <koobs.freebsd@gmail.com>
To: bug-followup@FreeBSD.org, vmagerya@gmail.com
Cc:  
Subject: Re: ports/171246: [patch] lang/python27: make python curses module
 work with unicode
Date: Thu, 24 Oct 2013 13:43:19 +1100

 Hi Vitaly,
 
 Firstly, thank you for the detailed report, it makes analysis an
 absolute pleasure.
 
 I'd like some clarity a few things prior to assessment:
 
 1) Did you intend to replace 'elif' with 'if' in the first hunk of each
 section in your patch? If so, why was it necessary? What happens if you
 change it to match upstream sources exactly (ie; revert the patch entirely)
 
 2) You mention that in 2.7 and 3.2, the patch is a no-op. Is this also
 the case in 3.3 and 3.1? If not, why not?
 
 In short, what I'd like to do is try our hand at reverting the existing
 patches completely, in all current Python port versions.
 
 Can you:
 
 - Revert the ncurses patches from setup.py completely
 - In lang/python{26,27,31,32,33)
 - Check redports builds
 - Check python test suite (python -m test.regrtest -vv test_curses)
 - Check your test case
 
 Feel free to start with python{27,33} and let me know what you find.
 
 At best, we find no issues. At worst, we identify issues that can be
 reported and fixed upstream, maintaining a local patch until the next
 release.
 

From: Vitaly Magerya <vmagerya@gmail.com>
To: Kubilay Kocak <koobs.freebsd@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: ports/171246: [patch] lang/python27: make python curses module
 work with unicode
Date: Wed, 30 Oct 2013 23:12:53 +0200

 On 10/24/2013 05:43, Kubilay Kocak wrote:
 > Hi Vitaly,
 
 Hi. Sorry for the delay, I missed your mail, and only discovered
 it accidentally by visiting the bug tracker.
 
 > Firstly, thank you for the detailed report, it makes analysis an
 > absolute pleasure.
 >
 > I'd like some clarity a few things prior to assessment:
 >
 > 1) Did you intend to replace 'elif' with 'if' in the first hunk of each
 > section in your patch? If so, why was it necessary? What happens if you
 > change it to match upstream sources exactly (ie; revert the patch
 entirely)
 
 Yes. Let's take python 2.7 as an example, and walk through the
 piece of it's setup.py that the patches touch.
 
 So, lines 709-719 first execute 'ldd /usr/lib/libreadline.so'
 and then search for 'libn?cursesw?' in the output. On FreeBSD
 'libreadline.so' is linked with '/lib/libncurses.so.8', so the
 script notes that and sets 'readline_termcap_library' variable
 to 'ncurses'.
 
 Then, lines 722-729 go like this:
 
     if 'curses' in readline_termcap_library:
         curses_library = readline_termcap_library
     elif self.compiler.find_library_file(lib_dirs, 'ncursesw'):
         curses_library = 'ncursesw'
     elif self.compiler.find_library_file(lib_dirs, 'ncurses'):
         curses_library = 'ncurses'
     elif self.compiler.find_library_file(lib_dirs, 'curses'):
         curses_library = 'curses'
 
 As you can see, the first clause works, and 'curses_library'
 gets set to the value of 'readline_termcap_library', which is
 'ncurses'. This leads to python shared objects '_curses.so' and
 '_curses_panel.so' to be linked with '-lncurses'.
 
 The patch that we currently have changes 'ncursesw' to 'xxxncursesw'
 in the second clause, but it doesn't matter: that clause is never
 executed. This is why I said it's a no-op.
 
 In previous versions of python the code that detected
 'readline_termcap_library' was less intelligent, and the first
 clause did not work; instead the second clause set curses_library
 to 'ncursesw'. This is where the patch that we current have comes
 in: it effectively disables the second clause, making the third
 clause work, thus setting 'curses_library' to 'ncurses'.
 
 The end result is that libraries that implement python's 'curses'
 module ('_curses.so' and '_curses_panel.so') are linked with
 '-lncurses', not '-lncursesw', which makes Unicode not work.
 
 What my patch does (by changing 'elif' to 'if' in the second
 clause) is to run the second clause unconditionally, thus allowing
 it to set curses_library to 'ncursesw', thus linking '_curses.so'
 and '_curses_panel.so' with '-lncursesw'.
 
 Now, at first I was confused as to why python seems to go to
 great lengths to avoid using '-lncursesw'. The reason for this
 turned out to be another python module, 'readline.so': this
 module must link with '-lreadline', but on FreeBSD 'libreadline.so'
 is itself linked with 'libncurses.so', so if you link '_curses.so'
 with '-lncursesw' and a python program will import both 'readline'
 and 'curses' modules, the python process will end up with both
 'libncurses.so' and 'libncursesw.so' loaded. Since those libraries
 provide identical functions, who knows what may happen...
 
 Well, one thing that may happen is that python may crash at some
 point. This concern seems to be what triggered our current patches
 to be introduced in the first place. Another thing that may
 happen is nothing. Which is what I'm seeing: my code that uses
 curses, even after importing readline, continues to work fine.
 
 Ultimately the problem stems from our 'libreadline.so' being
 linked with 'libncurses.so'. The proper solution (which I would
 prefer to the patch I submitted) is to change the build process
 of ncurses in base to produce a separate 'libtinfo.so' library
 and then make 'libreadline.so' link with that.
 
 > 2) You mention that in 2.7 and 3.2, the patch is a no-op. Is this also
 > the case in 3.3 and 3.1? If not, why not?
 
 I'll need to look into which version does what.
 
 > In short, what I'd like to do is try our hand at reverting the existing
 > patches completely, in all current Python port versions.
 >
 > Can you:
 >
 > - Revert the ncurses patches from setup.py completely
 > - In lang/python{26,27,31,32,33)
 > - Check redports builds
 > - Check python test suite (python -m test.regrtest -vv test_curses)
 > - Check your test case
 >
 > Feel free to start with python{27,33} and let me know what you find.
 
 I'll try to do so shortly to verify my findings, but I'm pretty sure:
 1) python{27,32,33} will build fine
 2) Unicode will not work
 
 > At best, we find no issues. At worst, we identify issues that can be
 > reported and fixed upstream, maintaining a local patch until the next
 > release.

From: Vitaly Magerya <vmagerya@gmail.com>
To: Kubilay Kocak <koobs.freebsd@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: ports/171246: [patch] lang/python27: make python curses module
 work with unicode
Date: Thu, 31 Oct 2013 17:50:03 +0200

 >> 2) You mention that in 2.7 and 3.2, the patch is a no-op. Is this also
 >> the case in 3.3 and 3.1? If not, why not?
 > 
 > I'll need to look into which version does what.
 
 It is a no-op in 2.7, 3.2 and 3.3. The patch is still relevant
 in 2.6 and 3.1.
 
 >> Can you:
 >>
 >> - Revert the ncurses patches from setup.py completely
 >> - In lang/python{26,27,31,32,33)
 >> - Check redports builds
 >> - Check python test suite (python -m test.regrtest -vv test_curses)
 
 BTW, 'python -m test.regrtest -u all -vv test_curses' seems to
 be the correct command; without '-u all' python says that "Use
 of the `curses' resource not enabled".
 
 >> - Check your test case
 >>
 >> Feel free to start with python{27,33} and let me know what you find.
 
 Here are my findings so far:
 1) if you remove the part of the current files/patch-setup.py
    that deals ncurses, python{27,32,33} build fine; see logs at [1]
 2) in all three cases 'test_curses' passes
 3) in all three cases my test case fails
 
 As a side note, it appears that if you have devel/ncurses
 installed, at least python33 links '_curses.so' with 'libncurses.so'
 from base, but '_curses_panel.so' with 'libncurses.so' from
 devel/ncurses. If this happens, 'test_curses' segfaults. One
 more thing to fix, I guess. (This behavior doesn't depend on
 whether or not you removed the patches).
 
 [1] https://redports.org/buildarchive/20131031122501-44919/

From: Vitaly Magerya <vmagerya@gmail.com>
To: Kubilay Kocak <koobs.freebsd@gmail.com>
Cc: bug-followup@FreeBSD.org
Subject: Re: ports/171246: [patch] lang/python27: make python curses module
 work with unicode
Date: Fri, 01 Nov 2013 18:18:50 +0200

 And here's [1] a build log of python{26,31} without the current
 patch and python{27,32,33} with my patch from above. In all these
 cases '_curses.so' and '_curses_panel.so' link with '-lncursesw'.
 
 In all the cases the build works fine (some results are missing
 because redports has some of it's machines down, but I'd wager
 they're OK too), 'test_curses' passes without problems, and my
 test case works fine.
 
 [1] http://redports.org/buildarchive/20131101112500-50583/
>Unformatted:
