Assigning to all packagers collectively, since there seems to be no maintainer for python and python3

CC: (none) => makowski.mageia, marja11
Assignee: bugsquad => pkg-bugs

(In reply to Jan Smout from comment #0)
> Possible resolutions:
> * move %{_libdir}/python%{dirver} to the %files -n %{lib_name} section

I think that it can be done

> * use update-alternatives to allow install of 32 bit python interpreter on
> 64 bit
> 
> The second solution has no use case for me, but might be of interest to
> others
> 
hum, I'm not a big fan of update-alternatives for this case

> Note: to compile on Mageia 5 with updates installed the following patch is
> required from upstream:
> https://hg.python.org/cpython/rev/1ad7c0253abe

Hum, this patch is only for a test, why are you saying this it is needed to "compile", and to compile what ?
we provide 2.7.11 that have this patch
or are you talking about Python 3.4 ?

Created attachment 7876 [details]
New packages (requires, provides and files list)

Would these be ok ?

If yes, I can do the same for Python3 and push this to 5/updates/testing and to cauldron

Hi Phillipe, thank you for your quick reply!


(In reply to Philippe Makowski from comment #2)

> > Note: to compile on Mageia 5 with updates installed the following patch is
> > required from upstream:
> > https://hg.python.org/cpython/rev/1ad7c0253abe
> 
> Hum, this patch is only for a test, why are you saying this it is needed to
> "compile", and to compile what ?

My mistake: technically it is not compiling, but the python test suite is part of the build recipe and that one currently fails without the patch. It complains about a key being too small. At least on my Mageia 5 system I was unable to rebuild the rpm from src without this particular patch.

> we provide 2.7.11 that have this patch
Yeah, I didn't check Cauldron. Could have saved me an hour :-s But hey, I learned some git basics meanwhile ;-)



(In reply to Philippe Makowski from comment #3)

> Would these be ok ?
I think that could work.


I would like to make a few comments before the first test:

* You prefer to create a std lib instead of just moving it to the libpython package. What is the rationale for this? From my point of view libpython.so is not usable without the std lib anyway, so why have a separate package to complete the first? Am I missing something? rpm packaging policy maybe?

* There are a couple of /usr/include in the main packages. Is that normal? I always thought that was something for the devel packages?

* Valgrind is a debugging feature. Should it not be in the devel package also?

* ensurepip is in the main package. Maybe it should also be in the std package. And maybe it doesn't matter because pip install will not be working anyway without the 32bit interpreter... to be tested... 

* The 64 bit std package contains a /usr/lib/python2.7 directory. This seems strange to me. In the spec we see this:
```
# %if "%{_lib}" != "lib"
# %{_prefix}/lib/python%{dirver}
# %endif
```

(In reply to Jan Smout from comment #4)
> I would like to make a few comments before the first test:
> 
> * You prefer to create a std lib instead of just moving it to the libpython
> package. What is the rationale for this? From my point of view libpython.so
> is not usable without the std lib anyway, so why have a separate package to
> complete the first? Am I missing something? rpm packaging policy maybe?

2 reasons , first yes rpm packaging policy
second get not far from what Debian, and Fedora are doing.
Debian is using libpythonX.X-stdlib and Fedora pythonX-libs.

In fact I should split this libpythonX.X-stdlib in to libpythonX.X-stdlib and libpythonX.X-testsuite as Debian did.

 
> * There are a couple of /usr/include in the main packages. Is that normal? I
> always thought that was something for the devel packages?

 "Makefile" and the config-32/64.h file are needed by
 distutils/sysconfig.py:_init_posix()

So may be I should move them in libpythonX.X-stdlib
  
> * Valgrind is a debugging feature. Should it not be in the devel package
> also?

yes you are right

> * ensurepip is in the main package. Maybe it should also be in the std
> package. And maybe it doesn't matter because pip install will not be working
> anyway without the 32bit interpreter... to be tested... 

yes, may be. I will put it in libpythonX.X-stdlib

 
> * The 64 bit std package contains a /usr/lib/python2.7 directory. This seems
> strange to me. In the spec we see this:
> ```
> # %if "%{_lib}" != "lib"
> # %{_prefix}/lib/python%{dirver}
> # %endif
> ```
yes, because we put noarch modules in %{_prefix}/lib/python%{dirver}

So part of the reason this is happening is because the way we package libraries prevents RPM's architecture color prioritization logic from kicking in, since it only kicks in on identically named packages that differ only on the architecture.

Take for example in Fedora, where both 32-bit and 64-bit libpython is in python-libs. When you install "python-libs", the package manager picks the 64-bit one by default, as it matches the primary "architecture color". If you wanted the 32-bit version also, you'd specify "python-libs.i686".

This kicks in some special behavior... Files that exist in both and are identical are ignored on the secondary architecture color (32-bit) and *not* erased on uninstall, files that are on different paths (/usr/lib vs /usr/lib64) are written as normal, and file conflicts are triggered if files that are in both arch colors that are the same paths but are different in content.

For the vast majority of things, this behavior works fine. There are some stuff that requires special handling, but those would not be parallel installable anyway without special handling.

Because we don't do this, we get some very strange behavior at times when we try to parallel install certain libraries.

CC: (none) => ngompa13

> > # %{_prefix}/lib/python%{dirver}
> > # %endif
> > ```
> yes, because we put noarch modules in %{_prefix}/lib/python%{dirver}

Aha. Mystery solved

I think it is ready for updates_testing

(In reply to Philippe Makowski from comment #3)
> If yes, I can do the same for Python3 and push this to 5/updates/testing and
> to cauldron

Err no we won't push such updates to mga5.
That's neither security nor a crash fix
I think it's too dangerous.
If we ever do this, let's do this in current distro, not for previous ones.

CC: (none) => thierry.vignaud

(In reply to Thierry Vignaud from comment #8)
> If we ever do this, let's do this in current distro, not for previous ones.

no problem with that.
Target Milestone: Mageia 6 then

if Jan want, I will send him the srpm for mga5.

Target Milestone: --- => Mageia 6

(In reply to Neal Gompa from comment #6)
> Because we don't do this, we get some very strange behavior at times when we
> try to parallel install certain libraries.

Interesting. This could explain why sometimes I'm having troubles selecting the correct package from the command line and have to revert to rpmdrake

In any case, for the problem at hand I don't see any difficulties. Doing urpmi libpython2.7 should install the correct package

(In reply to Philippe Makowski from comment #9)
> (In reply to Thierry Vignaud from comment #8)
> > If we ever do this, let's do this in current distro, not for previous ones.
> 
> no problem with that.

Too bad I have no time to research pre-existing software in Mageia which could be affected.


> Target Milestone: Mageia 6 then
> 
> if Jan want, I will send him the srpm for mga5.

A good compromise. Thank you

(In reply to Jan Smout from comment #11)
> > if Jan want, I will send him the srpm for mga5.
> 
> A good compromise. Thank you

Jan here for Python2 in Mageia5 : https://makowski.fedorapeople.org/srpms/mga5/

I will work now for Cauldron Python2 and Python3

For all here https://makowski.fedorapeople.org/srpms/mga5/info.txt the new sub packages and the spec https://makowski.fedorapeople.org/srpms/mga5/python.spec

(In reply to Philippe Makowski from comment #12)
> Jan here for Python2 in Mageia5 :
> https://makowski.fedorapeople.org/srpms/mga5/
> 
Thanks.

I recompiled and installed python, libpython, libpython-stdlib, lib64python, lib64python-stdlib.


Off topic I have another issue I never encountered before. I normally put the extra packages in a separate directory, use genhdlist2 and add it as a media (with no-verify-rpm because I don't know how to do otherwise). From there dependencies should resolve themselves. Except that they don't. I still need to manually list all dependent packages. Any thoughts?

(In reply to Jan Smout from comment #13)
> Off topic I have another issue I never encountered before. I normally put
> the extra packages in a separate directory, use genhdlist2 and add it as a
> media (with no-verify-rpm because I don't know how to do otherwise). From
> there dependencies should resolve themselves. Except that they don't. I
> still need to manually list all dependent packages. Any thoughts?

In the final solution, I change the provides/requires
in cauldron we'll have :

python requires lib{64}python2.7
lib{64}python2.7 requires lib{64}python2.7-stdlib
lib{64}python2.7-testsuite requires lib{64}python2.7 
lib{64}python-devel requires lib{64}python2.7

(In reply to Philippe Makowski from comment #14)
> In the final solution, I change the provides/requires
> in cauldron we'll have :
> 
> python requires lib{64}python2.7
> lib{64}python2.7 requires lib{64}python2.7-stdlib
> lib{64}python2.7-testsuite requires lib{64}python2.7 
> lib{64}python-devel requires lib{64}python2.7

Odd. I think it already has that kind of dependency cascade (will send the info in attachment. Note: in my version I removed the %subrel and bumped %rel to 3. For the rest no other deviation from your src.rpm)

Created attachment 7897 [details]
urpmi dependency fail

Here is my config, the output of urpmi --debug and the provides/requires of the relevant packages. Test done using only 64-bit packages

(In reply to Jan Smout from comment #16)
> Created attachment 7897 [details]
> urpmi dependency fail
> 
> Here is my config, the output of urpmi --debug and the provides/requires of
> the relevant packages. Test done using only 64-bit packages

seems your /var/lib/urpmi/mypackages/synthesis.hdlist.cz is not up to date
here an updated mga5 spec file https://makowski.fedorapeople.org/srpms/mga5/python.spec

(In reply to Philippe Makowski from comment #17)
> seems your /var/lib/urpmi/mypackages/synthesis.hdlist.cz is not up to date

$ urpmi.update mypackages
medium "mypackages" is up-to-date

So I redid genhdlist2 on the server and urpmi.update again, but encountered the same problem. Fyi, I also have patched pulseaudio packages which do not have this problem.

> here an updated mga5 spec file
> https://makowski.fedorapeople.org/srpms/mga5/python.spec
That one works for me and I don't see why. The only detectable differences are:
Requires: %{lib_name} = %{version}-%{release}
in the testsuite and the devel package

The mysteries of urpmi...


Anyway. There is only one dependency left: tkinter. I have smem installed which requires tkinter. When upgrading python, tkinter is not upgraded along. I have to specify it manually:

urpmi python tkinter

It complains that there are some files from the stdlib which conflict with tkinter-2.7.9-2.mga5

Sorry to annoy you so much with M5, but I guess that it would be the same for Cauldron?

genhdlist2 will not usually consider a package that was rebuild with the same NVR.
When you update a package, you've to bump either version or releas.
Else you can run "rm -rf media_info" prior to running genhdlist2
(which will be slower on very big repos such as core/release)

(In reply to Thierry Vignaud from comment #19)
> genhdlist2 will not usually consider a package that was rebuild with the
> same NVR.
> When you update a package, you've to bump either version or releas.
> Else you can run "rm -rf media_info" prior to running genhdlist2
> (which will be slower on very big repos such as core/release)

That explains. I indeed had already constructed a spec of my own last week and didn't bother to change the version number when I got the spec from Phillipe. Now I understand better why you work with %subrel before bumping %rel. Sorry for the noise. Let's hope other people may benefit from this discussion.
Thanks for the reply.

(In reply to Jan Smout from comment #18)
> It complains that there are some files from the stdlib which conflict with
> tkinter-2.7.9-2.mga5
> 
Thanks fro the report
do you have the list of files in conflict ?

Created attachment 7917 [details]
conflict with tkinter and python-devel

So I have a conflict with python-devel and tkinter

Friday I was a bit in a hurry. I hope I took the correct packages...

in the spec change to :

%package -n     tkinter
Summary:        A graphical user interface for the Python scripting language
Group:          Development/Python
Requires:       python = %version
Requires:       tcl tk
Conflicts:      python < 2.7.9-2

Status: ASSIGNED => RESOLVED
Resolution: (none) => FIXED

Nope. Still complains about the same files. tkinter is not selected :-/

(In reply to Jan Smout from comment #24)
> Nope. Still complains about the same files. tkinter is not selected :-/

it should, especially if it have a release or subrelease greater than the one installed on your box

Created attachment 7998 [details]
my config + conflicting files

So here is my config

When I uninstall lib64python-devel I can install with 'urpmi python tkinter', but not with 'urpmi python' (tkinter is not selected despite of the Conflicts entry).

So there are 2 problems to be solved: one with tkinter and another with the devel package.

Any ideas?