I don't find where the bug could be in python-setuptools
may be in python3 package ?

if you attempt to build python-lxml with :

 CFLAGS="%{optflags}" %{__python3} setup.py build_ext -i -l python3.2

there is no problem

but I confirm that building it with 

 CFLAGS="%{optflags}" %{__python3} setup.py build

raise the problem, even under mga2

Building lxml from source works fine under Ubuntu 12.10, both with Ubuntu's Python 3.2 and with a hand-built Python 3.3. Here is the command line for the latter:

gcc -pthread -shared build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so

Note how it *doesn't* mention "-lpython3something". Also:

$ ldd build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so
	linux-vdso.so.1 =>  (0x00007fff651d7000)
	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007fc379564000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc379347000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc378f87000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc378d83000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc378b6c000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc378949000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc37864d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc379b2b000)

However, if I add "-Wl,--no-undefined" to the command line, then linking fails with the same kind of errors as you get:

$ gcc -Wl,--no-undefined -pthread -shared build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.3/lxml/objectify.cpython-33m.so
build/temp.linux-x86_64-3.3/src/lxml/lxml.objectify.o: dans la fonction « __pyx_tp_new_4lxml_9objectify_PyType »:
/home/antoine/lxml-3.1.0/src/lxml/lxml.objectify.c:26404: référence indéfinie vers « _Py_NoneStruct »

So here's your culprit.

CC: (none) => pitrou

Note, if you want to keep "-Wl,--noundefined", you can also use python3-config to add the required library references:

$ python3-config --libs
-lpthread -ldl -lutil -lm -lpython3.2mu
$ python3-config --ldflags
-L/usr/lib/python3.2/config-3.2mu -lpthread -ldl -lutil -lm -lpython3.2mu -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions

Sure, the issue comes from our linking policy. But why do setuptools show such different behaviour between two major versions ?

AFAICT, it's not setuptools, it's python's distutils.

And, with a hand-made build of Python 2.7, I don't get a different behaviour:

gcc -pthread -shared build/temp.linux-x86_64-2.7-pydebug/src/lxml/lxml.objectify.o -L/usr/lib -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7-pydebug/lxml/objectify.so

Neither do I with Ubuntu's Python 2.7.3. Perhaps there's a patch in Mageia's Python?

so the problem is in the python3 package

Source RPM: python-setuptools => python3

Antoine, I'm still looking where is the bug, but strangely here what I have under mga2 :
$ python -c 'from distutils.sysconfig import get_config_vars; print(get_config_vars("BLDLIBRARY"))'
['-L. -lpython2.7']
$ python3 -c 'from distutils.sysconfig import get_config_vars; print(get_config_vars("BLDLIBRARY"))'
['-L. -lpython3.2mu']

so they should behave the same way, no ?

Just for the record - with a python3.3.2 that I built using:

#!/bin/bash
./configure --prefix="$HOME/apps/python3"

Then I can do:

~/apps/python3/bin/python3 setup.py build

In the lxml tarball and it is working.

Regards,

-- Shlomi Fish

Status: NEW => ASSIGNED

and ?
what is your conclusion ?
How to fix the bug ?

(In reply to Philippe Makowski from comment #9)
> and ?
> what is your conclusion ?
> How to fix the bug ?

I'm investigating. Currently, I'm doing some cleanups to the python3 package. If you have a suggestion for a patch against the "mgarepo co python3" checkout, that would be appreciated.

Regards,

-- Shlomi Fish

of course, if I can find something I will help
this bug as 10102 (witch seems to have same kind of issue but with Python2) as 3348 and 6514 are my targets, but since now with no success

Hi Philippe,

(In reply to Philippe Makowski from comment #11)
> of course, if I can find something I will help
> this bug as 10102 (witch seems to have same kind of issue but with Python2)
> as 3348 and 6514 are my targets, but since now with no success

I reported this bug in Python - http://bugs.python.org/issue18142 - and got a comment. It sucks that the tests fail.

Regards,

-- Shlomi Fish

Can you try without dont_write_bytecode=1 ?

(In reply to Philippe Makowski from comment #13)
> Can you try without dont_write_bytecode=1 ?

What? What do you mean? How and where? Please be more explicit.

I see in your python-make-test-output.txt these flags :
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=1, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)

I wander if without dont_write_bytecode=1 results differ or not

Hi Philippe,

(In reply to Philippe Makowski from comment #15)
> I see in your python-make-test-output.txt these flags :
> Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
> dont_write_bytecode=1, no_user_site=0, no_site=0, ignore_environment=1,
> verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)
> 
> I wander if without dont_write_bytecode=1 results differ or not

Thanks for the input. I'm now trying to run the tests after doing «unset PYTHONDONTWRITEBYTECODE» in the shell which emits dont_write_bytecode=0.

Regards,

-- Shlomi Fish

Thanks for the good insight. After doing

$ unset PYTHONDONTWRITEBYTECODE 

And running "make test" then only the "test_ftp" test fails (not the other ones).

I get this:

<<<<<<
======================================================================
ERROR: test_ftp (test.test_urllib2net.OtherNetworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 2307, in retrfile
    self.ftp.cwd(file)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 591, in cwd
    return self.voidcmd(cmd)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 264, in voidcmd
    return self.voidresp()
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 238, in voidresp
    resp = self.getresp()
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/ftplib.py", line 233, in getresp
    raise error_perm(resp)
ftplib.error_perm: 550 Failed to change directory.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 112, in test_ftp
    self._test_urls(urls, self._extra_handlers())
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 218, in _test_urls
    f = urlopen(url, req, TIMEOUT)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 33, in wrapped
    return _retry_thrice(func, exc, *args, **kwargs)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/test/test_urllib2net.py", line 23, in _retry_thrice
    return func(*args, **kwargs)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 469, in open
    response = self._open(req, data)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 487, in _open
    '_open', req)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 447, in _call_chain
    result = func(*args)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 1464, in ftp_open
    fp, retrlen = fw.retrfile(file, type)
  File "/home/shlomif/Download/unpack/prog/python/Python-3.3.2/Lib/urllib/request.py", line 2309, in retrfile
    raise URLError('ftp error: %d' % reason) from reason
TypeError: %d format: a number is required, not error_perm

----------------------------------------------------------------------
Ran 15 tests in 14.820s

>>>>>>

Regards,

-- Shlomi Fish

(In reply to Shlomi Fish from comment #17)
> Thanks for the good insight. After doing
> 
> $ unset PYTHONDONTWRITEBYTECODE 
> 
> And running "make test" then only the "test_ftp" test fails (not the other
> ones).
> 
> I get this:

OK - an update: apparently, the problem is that with dont_write_bytecode=0 the tests of bdist_rpm don't appear to get run at all (or at least they don't appear in the output). I don't understand where how and why it happens.

Also see:

http://bugs.python.org/issue18188

Regards,

-- Shlomi Fish

bdist_rpm is part of test_distutils so if test_distutils succeed then bdist_rpm too I think

Hi Philippe,

(In reply to Philippe Makowski from comment #19)
> bdist_rpm is part of test_distutils so if test_distutils succeed then
> bdist_rpm too I think

Please see http://bugs.python.org/issue18188 - an explanation is given there:

<<<<<
> However, I don't see the bdist_rpm anywhere in the output.

Individual test cases are only listed on failure, or when verbose mode is specified. So, if it doesn't appear in the output, it passed.
>>>>>

Regards,

-- Shlomi Fish

For information the original report problem is solved by using :
CFLAGS="%{optflags}" %{__python3} setup.py build_ext -i `python3-config --libs`

cf :
http://svnweb.mageia.org/packages/cauldron/python-lxml/current/SPECS/python-lxml.spec?view=markup&pathrev=442226

but I'm not convinced that it is a long term solution

(In reply to Philippe Makowski from comment #21)
> For information the original report problem is solved by using :
> CFLAGS="%{optflags}" %{__python3} setup.py build_ext -i `python3-config
> --libs`
> 
> cf :
> http://svnweb.mageia.org/packages/cauldron/python-lxml/current/SPECS/python-
> lxml.spec?view=markup&pathrev=442226
> 
> but I'm not convinced that it is a long term solution

Thanks for the insight.

Regards,

-- Shlomi Fish

bug is in the python package (2 and 3), we need to change them to remove "-Wl,--no-undefined" from LDFLAGS in their respective ..../config/Makefile

it should also resolve bug 10102

(In reply to Philippe Makowski from comment #23)
> bug is in the python package (2 and 3), we need to change them to remove
> "-Wl,--no-undefined" from LDFLAGS in their respective ..../config/Makefile
> 
> it should also resolve bug 10102

Hi Philippe,

after I apply the patch below to the python3 svn .spec, then I am able to successfully build lxml-3.2.1 using «python3 setup.py build_ext». Without this patch applied, then gcc exits with an error. 

So if there aren't any further objections, I can apply the patch and make a new release (by bumping %mkrel).

Regards,

-- Shlomi Fish

shlomif[rpms]:$mageia/python3$ svn di
Index: SPECS/python3.spec
===================================================================
--- SPECS/python3.spec  (revision 437534)
+++ SPECS/python3.spec  (working copy)
@@ -187,6 +187,9 @@
 export CPPFLAGS="%{optflags} -I/usr/include/ncursesw"
 
 autoreconf -vfi
+# Remove -Wl,--no-undefined in accordance with MGA #9395 :
+# https://bugs.mageia.org/show_bug.cgi?id=9395
+%define _disable_ld_no_undefined 1
 %configure2_5x  --with-threads \
                 --enable-ipv6 \
                 --with-dbmliborder=gdbm \

I aggree

and same should be done in python 2

another solution can be to not change the build
but only change the Makefile after the build
something like (here for python 2, but it should be the same for python3):

Index: SPECS/python.spec
===================================================================
--- SPECS/python.spec	(révision 445465)
+++ SPECS/python.spec	(copie de travail)

@@ -314,6 +315,10 @@
 
 # fix Makefile to get rid of reference to distcc
 perl -pi -e "/^CC=/ and s/distcc/gcc/" Makefile
+# Remove -Wl,--no-undefined in accordance with MGA #9395 and MGA #10102:
+# https://bugs.mageia.org/show_bug.cgi?id=9395
+# https://bugs.mageia.org/show_bug.cgi?id=10102
+perl -pi -e "/^LDFLAGS=/ and s/-Wl,--no-undefined//" Makefile
 
 # set the install path
 echo '[install_scripts]' >setup.cfg

Either you build python and its modules with this option, either none of them. Tampering with a system designed to ensure ABI compatibility between python core and its extension is not a good idea.

Guillaume :
I don't understand well why, but if you say so
I will change this to %define _disable_ld_no_undefined 1
instead

for information Debian provide empty LDFLAGS in the Makefile

Shlomi, 

please push your patch, I need it to build other packages

note that I added a macro in python3.macros

http://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/python3.macros?revision=446322&view=markup


thanks

(In reply to Philippe Makowski from comment #29)
> Shlomi, 
> 
> please push your patch, I need it to build other packages
> 

Done!

Regards,

-- Shlomi Fish

> note that I added a macro in python3.macros
> 
> http://svnweb.mageia.org/packages/cauldron/python3/current/SOURCES/python3.
> macros?revision=446322&view=markup
> 
> 
> thanks

Fixed in Bug 10391:
http://advisories.mageia.org/MGASA-2013-0252.html

Status: ASSIGNED => RESOLVED
CC: (none) => luigiwalser
Resolution: (none) => FIXED