Bug 9395 - linking error with python3
: linking error with python3
Status: RESOLVED FIXED
Product: Mageia
Classification: Unclassified
Component: RPM Packages
: Cauldron
: i586 Linux
: Normal Severity: normal
: ---
Assigned To: Shlomi Fish
:
:
:
:
: 10391
:
  Show dependency treegraph
 
Reported: 2013-03-14 22:28 CET by Guillaume Rousse
Modified: 2013-08-22 21:56 CEST (History)
3 users (show)

See Also:
Source RPM: python3
CVE:
Status comment:


Attachments

Description Guillaume Rousse 2013-03-14 22:28:23 CET
Attempting to build a native python module fails because of a missing library flag in linker command:

gcc -pthread -shared -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o -L/usr/lib64 -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so
[..]
home/guillaume/work/mageia/cauldron/python-lxml/BUILD/python-lxml-3.1.0/python3/src/lxml/lxml.objectify.c:26404: undefined reference to « _Py_NoneStruct

The same native python module builds OK with python 2.7:
gcc -pthread -shared -Wl,--as-needed -Wl,--no-undefined -Wl,-z,relro -Wl,-O1 -Wl,--build-id -Wl,--enable-new-dtags build/temp.linux-x86_64-2.7/src/lxml/lxml.objectify.o -L/usr/lib64 -L/usr/lib64 -lxslt -lexslt -lxml2 -lz -lm -lpython2.7 -o build/lib.linux-x86_64-2.7/lxml/objectify.so

The obvious different is the presence of -lpython2.7 flag in second case.

This seems to be an issue in either python3-setuptools or python3 package, as lxml setupinfo.py file only handles the external libraries flags.

Reproducible: 

Steps to Reproduce:
Comment 1 Philippe Makowski 2013-03-15 22:28:18 CET
I don't find where the bug could be in python-setuptools
may be in python3 package ?

if you attempt to build python-lxml with :

 CFLAGS="%{optflags}" %{__python3} setup.py build_ext -i -l python3.2

there is no problem

but I confirm that building it with 

 CFLAGS="%{optflags}" %{__python3} setup.py build

raise the problem, even under mga2
Comment 2 Antoine Pitrou 2013-03-18 11:17:38 CET
Building lxml from source works fine under Ubuntu 12.10, both with Ubuntu's Python 3.2 and with a hand-built Python 3.3. Here is the command line for the latter:

gcc -pthread -shared build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so

Note how it *doesn't* mention "-lpython3something". Also:

$ ldd build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so
	linux-vdso.so.1 =>  (0x00007fff651d7000)
	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fc379564000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc379347000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc378f87000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc378d83000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc378b6c000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc378949000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc37864d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc379b2b000)

However, if I add "-Wl,--no-undefined" to the command line, then linking fails with the same kind of errors as you get:

$ gcc -Wl,--no-undefined -pthread -shared build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so
build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o: dans la fonction « __pyx_tp_new_4lxml_9objectify_PyType »:
/home/antoine/lxml-3.1.0/src/lxml/lxml.objectify.c:26404: référence indéfinie vers « _Py_NoneStruct »

So here's your culprit.
Comment 3 Antoine Pitrou 2013-03-18 11:22:47 CET
Note, if you want to keep "-Wl,--noundefined", you can also use python3-config to add the required library references:

$ python3-config --libs
-lpthread -ldl -lutil -lm -lpython3.2mu
$ python3-config --ldflags
-L/usr/lib/python3.2/config-3.2mu -lpthread -ldl -lutil -lm -lpython3.2mu -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions
Comment 4 Guillaume Rousse 2013-03-18 11:41:05 CET
Sure, the issue comes from our linking policy. But why do setuptools show such different behaviour between two major versions ?
Comment 5 Antoine Pitrou 2013-03-18 12:18:28 CET
AFAICT, it's not setuptools, it's python's distutils.

And, with a hand-made build of Python 2.7, I don't get a different behaviour:

gcc -pthread -shared build/temp.linux-x86_64-2.7-pydebug/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7-pydebug/lxml/objectify.so

Neither do I with Ubuntu's Python 2.7.3. Perhaps there's a patch in Mageia's Python?
Comment 6 Philippe Makowski 2013-03-18 13:31:49 CET
so the problem is in the python3 package
Comment 7 Philippe Makowski 2013-03-19 22:17:08 CET
Antoine, I'm still looking where is the bug, but strangely here what I have under mga2 :
$ python -c 'from distutils.sysconfig import get_config_vars; print(get_config_vars("BLDLIBRARY"))'
['-L. -lpython2.7']
$ python3 -c 'from distutils.sysconfig import get_config_vars; print(get_config_vars("BLDLIBRARY"))'
['-L. -lpython3.2mu']

so they should behave the same way, no ?
Comment 8 Shlomi Fish 2013-06-05 13:15:30 CEST
Just for the record - with a python3.3.2 that I built using:

#!/bin/bash
./configure --prefix="$HOME/apps/python3"

Then I can do:

~/apps/python3/bin/python3 setup.py build

In the lxml tarball and it is working.

Regards,

-- Shlomi Fish
Comment 9 Philippe Makowski 2013-06-05 13:18:26 CEST
and ?
what is your conclusion ?
How to fix the bug ?
Comment 10 Shlomi Fish 2013-06-05 13:23:45 CEST
(In reply to Philippe Makowski from comment #9)
> and ?
> what is your conclusion ?
> How to fix the bug ?

I'm investigating. Currently, I'm doing some cleanups to the python3 package. If you have a suggestion for a patch against the "mgarepo co python3" checkout, that would be appreciated.

Regards,

-- Shlomi Fish
Comment 11 Philippe Makowski 2013-06-05 13:36:23 CEST
of course, if I can find something I will help
this bug as 10102 (witch seems to have same kind of issue but with Python2) as 3348 and 6514 are my targets, but since now with no success
Comment 12 Shlomi Fish 2013-06-11 12:22:32 CEST
Hi Philippe,

(In reply to Philippe Makowski from comment #11)
> of course, if I can find something I will help
> this bug as 10102 (witch seems to have same kind of issue but with Python2)
> as 3348 and 6514 are my targets, but since now with no success

I reported this bug in Python - http://bugs.python.org/issue18142 - and got a comment. It sucks that the tests fail.

Regards,

-- Shlomi Fish
Comment 13 Philippe Makowski 2013-06-11 13:35:16 CEST
Can you try without dont_write_bytecode=1 ?
Comment 14 Shlomi Fish 2013-06-11 13:43:07 CEST
(In reply to Philippe Makowski from comment #13)
> Can you try without dont_write_bytecode=1 ?

What? What do you mean? How and where? Please be more explicit.
Comment 15 Philippe Makowski 2013-06-11 13:55:30 CEST
I see in your python-make-test-output.txt these flags :
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=1, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)

I wander if without dont_write_bytecode=1 results differ or not
Comment 16 Shlomi Fish 2013-06-11 14:09:39 CEST
Hi Philippe,

(In reply to Philippe Makowski from comment #15)
> I see in your python-make-test-output.txt these flags :
> Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
> dont_write_bytecode=1, no_user_site=0, no_site=0, ignore_environment=1,
> verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)
> 
> I wander if without dont_write_bytecode=1 results differ or not

Thanks for the input. I'm now trying to run the tests after doing «unset PYTHONDONTWRITEBYTECODE» in the shell which emits dont_write_bytecode=0.

Regards,

-- Shlomi Fish
Comment 17 Shlomi Fish 2013-06-11 14:15:44 CEST
Thanks for the good insight. After doing

$ unset PYTHONDONTWRITEBYTECODE 

And running "make test" then only the "test_ftp" test fails (not the other ones).

I get this:

<<<<<<
======================================================================
ERROR: test_ftp (test.test_urllib2net.OtherNetworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 2307, in retrfile
    self.ftp.cwd(file)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 591, in cwd
    return self.voidcmd(cmd)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 264, in voidcmd
    return self.voidresp()
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 238, in voidresp
    resp = self.getresp()
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 233, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 Failed to change directory.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 112, in test_ftp
    self._test_urls(urls, self._extra_handlers())
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 218, in _test_urls
    f = urlopen(url, req, TIMEOUT)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 33, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 23, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 469, in open
    response = self._open(req, data)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 487, in _open
    '_open', req)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 447, in _call_chain
    result = func(*args)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 1464, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 2309, in retrfile
    raise URLError('ftp error: %d' % reason) from reason
TypeError: %d format: a number is required, not error_perm

----------------------------------------------------------------------
Ran 15 tests in 14.820s

>>>>>>

Regards,

-- Shlomi Fish
Comment 18 Shlomi Fish 2013-06-11 15:02:35 CEST
(In reply to Shlomi Fish from comment #17)
> Thanks for the good insight. After doing
> 
> $ unset PYTHONDONTWRITEBYTECODE 
> 
> And running "make test" then only the "test_ftp" test fails (not the other
> ones).
> 
> I get this:

OK - an update: apparently, the problem is that with dont_write_bytecode=0 the tests of bdist_rpm don't appear to get run at all (or at least they don't appear in the output). I don't understand where how and why it happens.

Also see:

http://bugs.python.org/issue18188

Regards,

-- Shlomi Fish
Comment 19 Philippe Makowski 2013-06-13 08:58:28 CEST
bdist_rpm is part of test_distutils so if test_distutils succeed then bdist_rpm too I think
Comment 20 Shlomi Fish 2013-06-13 10:51:14 CEST
Hi Philippe,

(In reply to Philippe Makowski from comment #19)
> bdist_rpm is part of test_distutils so if test_distutils succeed then
> bdist_rpm too I think

Please see http://bugs.python.org/issue18188 - an explanation is given there:

<<<<<
> However, I don't see the bdist_rpm anywhere in the output.

Individual test cases are only listed on failure, or when verbose mode is specified. So, if it doesn't appear in the output, it passed.
>>>>>

Regards,

-- Shlomi Fish
Comment 21 Philippe Makowski 2013-06-14 23:30:54 CEST
For information the original report problem is solved by using :
CFLAGS="%{optflags}" %{__python3} setup.py build_ext -i `python3-config --libs`

cf :
http://svnweb.mageia.org/packages/cauldron/python-lxml/current/SPECS/python-lxml.spec?view=markup&pathrev=442226

but I'm not convinced that it is a long term solution
Comment 22 Shlomi Fish 2013-06-15 10:24:12 CEST
(In reply to Philippe Makowski from comment #21)
> For information the original report problem is solved by using :
> CFLAGS="%{optflags}" %{__python3} setup.py build_ext -i `python3-config
> --libs`
> 
> cf :
> http://svnweb.mageia.org/packages/cauldron/python-lxml/current/SPECS/python-
> lxml.spec?view=markup&pathrev=442226
> 
> but I'm not convinced that it is a long term solution

Thanks for the insight.

Regards,

-- Shlomi Fish
Comment 23 Philippe Makowski 2013-06-20 18:41:44 CEST
bug is in the python package (2 and 3), we need to change them to remove "-Wl,--no-undefined" from LDFLAGS in their respective ..../config/Makefile

it should also resolve bug 10102
Comment 24 Shlomi Fish 2013-06-21 10:56:17 CEST
(In reply to Philippe Makowski from comment #23)
> bug is in the python package (2 and 3), we need to change them to remove
> "-Wl,--no-undefined" from LDFLAGS in their respective ..../config/Makefile
> 
> it should also resolve bug 10102

Hi Philippe,

after I apply the patch below to the python3 svn .spec, then I am able to successfully build lxml-3.2.1 using «python3 setup.py build_ext». Without this patch applied, then gcc exits with an error. 

So if there aren't any further objections, I can apply the patch and make a new release (by bumping %mkrel).

Regards,

-- Shlomi Fish

shlomif[rpms]:$mageia/python3$ svn di
Index: SPECS/python3.spec
===================================================================
--- SPECS/python3.spec  (revision 437534)
+++ SPECS/python3.spec  (working copy)
@@ -187,6 +187,9 @@
 export CPPFLAGS="%{optflags} -I/usr/include/ncursesw"
 
 autoreconf -vfi
+# Remove -Wl,--no-undefined in accordance with MGA #9395 :
+# https://bugs.mageia.org/show_bug.cgi?id=9395
+%define _disable_ld_no_undefined 1
 %configure2_5x  --with-threads \
                 --enable-ipv6 \
                 --with-dbmliborder=gdbm \
Comment 25 Philippe Makowski 2013-06-21 14:17:34 CEST
I aggree

and same should be done in python 2
Comment 26 Philippe Makowski 2013-06-21 20:08:20 CEST
another solution can be to not change the build
but only change the Makefile after the build
something like (here for python 2, but it should be the same for python3):

Index: SPECS/python.spec
===================================================================
--- SPECS/python.spec	(révision 445465)
+++ SPECS/python.spec	(copie de travail)

@@ -314,6 +315,10 @@
 
 # fix Makefile to get rid of reference to distcc
 perl -pi -e "/^CC=/ and s/distcc/gcc/" Makefile
+# Remove -Wl,--no-undefined in accordance with MGA #9395 and MGA #10102:
+# https://bugs.mageia.org/show_bug.cgi?id=9395
+# https://bugs.mageia.org/show_bug.cgi?id=10102
+perl -pi -e "/^LDFLAGS=/ and s/-Wl,--no-undefined//" Makefile
 
 # set the install path
 echo '[install_scripts]' >setup.cfg
Comment 27 Guillaume Rousse 2013-06-22 00:10:56 CEST
Either you build python and its modules with this option, either none of them. Tampering with a system designed to ensure ABI compatibility between python core and its extension is not a good idea.
Comment 28 Philippe Makowski 2013-06-22 10:04:13 CEST
Guillaume :
I don't understand well why, but if you say so
I will change this to %define _disable_ld_no_undefined 1
instead

for information Debian provide empty LDFLAGS in the Makefile
Comment 29 Philippe Makowski 2013-06-25 15:19:09 CEST
Shlomi, 

please push your patch, I need it to build other packages

note that I added a macro in python3.macros

http://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/python3.macros?revision=446322&view=markup


thanks
Comment 30 Shlomi Fish 2013-06-25 15:58:17 CEST
(In reply to Philippe Makowski from comment #29)
> Shlomi, 
> 
> please push your patch, I need it to build other packages
> 

Done!

Regards,

-- Shlomi Fish

> note that I added a macro in python3.macros
> 
> http://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/python3.
> macros?revision=446322&view=markup
> 
> 
> thanks
Comment 31 David Walser 2013-08-22 21:56:15 CEST
Fixed in Bug 10391:
http://advisories.mageia.org/MGASA-2013-0252.html

Note You need to log in before you can comment on or make changes to this bug.