The dogma of not byte-compiling modules leads to further problems and even more performance glitches: Package python-virtualenv brings its own, buggy implementation of virtualenv, see bug #3350.

Hi, thanks for reporting this bug.
Assigned to the package maintainer.

(I seen you have discussed about this one, on irc, but I didn't understand :) )

Keywords: (none) => Triaged
Assignee: bugsquad => misc
Source RPM: (none) => python

Summary of the discussion on irc:

* There have been earlier discussions about at
  https://qa.mandriva.com/show_bug.cgi?id=50484
  and http://www.mail-archive.com/mageia-dev@mageia.org/msg02043.html
* Maintainer would accept patches implementing a filetrigger

Summary of a discussion I had on #rpm.org this morning:

* Not delivering byte-code files with the packages is odd.
* Changing the database from a filetrigger is not possible, since this is still
  within the transaction.

A filetrigger including a .spec file can be found at
http://bitbucket.org/htgoebel/python-rpm-filetrigger

(In reply to comment #5)
> A filetrigger including a .spec file can be found at
> http://bitbucket.org/htgoebel/python-rpm-filetrigger

Pinging, because nothing has happened with this report for more than 3 months, it still has the status NEW or REOPENED.

@ misc

Please set status to ASSIGNED if you think this bug was assigned correctly. If for work flow reasons you can't do that, then please put OK on the whiteboard instead.

CC: (none) => marja11

Using the file-trigger is a cludge for fixing he real cludge, which is: PYTHONDONTWRITEBYTECODE is set.

To repeat comment #1 PYTHONDONTWRITEBYTECODE *must* *not* *be* *set*! It leads to a lot of problems, see e.g. bug 3358.

Another example for why PYTHONDONTWRITEBYTECODE must not be set: I had *very* wired effects when packaging a programm using www.pyinstaller.org. It took me about 2 hours of hard-core debugging to get it down to the real reason: PYTHONDONTWRITEBYTECODE. See http://www.pyinstaller.org/ticket/527 for more information.

Exaggeration ( like "it lead to a lot of problem like : _1_ problem and _1_ second problem ) is far from being a good way to convince me, quite the contrary, especially when there is a easy work around ( like, you know, using "unset PYTHONDONTWRITEBYTECODE" in the shell )  for people who are skilled enough to see the problem ( and good enough to read the whole discussion especially where I explained the issue I have with the various solutions ).

But thanks for searching for #rpm.org, showing that a file trigger is not gonna do the job.

On a side note,  that's kludge with a k, not cludge.

Re comment #8.

To be frank, your answer sounds arrogant to me.

For other heavy problems, see bug 3350, bug 3358. 

Finding this bug (comment #7) in packaging took *me* two hours. And I'm one of the core developers of pyinstaller and thus pretty skilled in tracking down packaging problems. If somebody else would have been trapped by this bug, *we* would have been the ones fishing in murky waters trying to solve the problem. *We* would have a lot of work and another unresolvable bug in the issue tracker.

*You* just lay back and say: "Convince me!"

This is what I call ignorant and arrogant.

The only accountable reason for setting PYTHONDONTWRITEBYTECODE you gave is: This makes some packages smaller to fit onto the Live-CD.

So what about looking for other solutions, like different packaging for the Live-CD?

Please look at the bottom of this mail to see whether you're the assignee of this  bug, if you don't already know whether you are.


If you're the assignee:

We'd like to know for sure whether this bug was assigned correctly. Please change status to ASSIGNED if it is, or put OK on the whiteboard instead.

If you don't have a clue and don't see a way to find out, then please put NEEDHELP on the whiteboard.

Please assign back to Bug Squad or to the correct person to solve this bug if we were wrong to assign it to you, and explain why.

Thanks :)

**************************** 

@ the reporter and persons in the cc of this bug:

If you have any new information that wasn't given before (like this bug being valid for another version of Mageia, too, or it being solved) please tell us.

@ the reporter of this bug

If you didn't reply yet to a request for more information, please do so within two weeks from now.

Thanks all :-D

This message is a reminder that Mageia 1 is nearing its end of life. 
In approximately 25 days from now, Mageia will stop maintaining and issuing 
updates for Mageia 1. At that time this bug will be closed as WONTFIX (EOL) if it 
remains open with a Mageia 'version' of '1'.

Package Maintainer: If you wish for this bug to remain open because you plan to 
fix it in a currently maintained version, simply change the 'version' to a later 
Mageia version prior to Mageia 1's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we may not 
be able to fix it before Mageia 1 is end of life.  If you would still like to see 
this bug fixed and are able to reproduce it against a later version of Mageia, 
you are encouraged to click on "Version" and change it against that version 
of Mageia.

Although we aim to fix as many bugs as possible during every release's lifetime, 
sometimes those efforts are overtaken by events. Often a more recent Mageia 
release includes newer upstream software that fixes bugs or makes them obsolete.

--
Mageia Bugsquad

This bug persists in Mageia 2.

Version: 1 => 2

This bug persists in Mageia 3.

Hardware: i586 => All
Version: 2 => 3

I found yet another problem cause by setting PYTHONDONTWRITEBYTECODE:

python setup.py bdist_rpm

fails with "File no found: ...*.pyc".

proposal is made for mga 4
https://wiki.mageia.org/en/Feature:PythonByteCompiling

*** Bug 10842 has been marked as a duplicate of this bug. ***

*** Bug 10842 has been marked as a duplicate of this bug. ***

*** Bug 10843 has been marked as a duplicate of this bug. ***

11-15-2011 through 8-13-2013

That is almost 2 years.  Did I arrive in the twilight zone?

Currently PYTHONDONTWRITEBYTECODE continues to be set in /etc/profile.d/30python.sh and /etc/profile.d/30python.csh

... as Hartmut Goebel already pointed out, this should have NEVER been the case.  There is absolutely ZERO argument for doing it.  I manually changed it already and it came back to byte me again (excuse the pun) because it keeps coming back like a bad cold, either through an update or I may have temporarily uninstalled and installed python again.

The suggestion that you can unset it in your .bashrc is ludicrous.  It will not help one iota for a normal user to unset it. It will still be set for root, and the machines CPU cycles will still be wasted time and time again, every time a python package is imported.

Seriously, it is a SINGLE LINE that doesn't belong there.  It is useful for a live CD or other read only filesystem, but an abomination otherwise.

Will somebody PLEASE fix this issue.

I am marking this bug as for Mageia 3 in case there is still no cauldron, and also changing the status to reopened since it never got triaged apparently.

Status: NEW => REOPENED
CC: (none) => jdines
Version: Cauldron => 3

as said in comment 15 :
proposal is made for mga 4
https://wiki.mageia.org/en/Feature:PythonByteCompiling

proposal was accepted, so the change will come with mga4
it is not a SINGLE LINE change, it implies a review of all packages, adopting a new policy, add new rpm macros to do this the right way.

Status: REOPENED => ASSIGNED
Version: 3 => Cauldron

I agree that there is more to be done other than changing the lines in /etc/profile.d/30python.sh and /etc/profile.d/30python.csh to undo the mess caused by setting this and not shipping pyc files, but they have nothing to do with this bug.  This bug is:  'python packages are not compiling the byte-code'

You are thinking of the bug: 'We used to have PYTHONDONTCOMPILEBYTECODE set and not ship pyc files, so some packages on disk still don't have .pyc files in the __pycache__', which is a different bug as yet to be filed.

Perhaps you can explain to me why updating the 30python.sh and 30python.csh files might cause some kind of problem.

As far as the review of all packages, each package can have the pyc files added as part of the next update.  There is no need to solve the problem system-wide in one fell swoop.

Of course, that would be very simple too.  How hard is it, really, to use the find command to look for all the py files on disk in /usr/lib64/python3.3 and import them after unsetting PYTHONDONTWRITEBYTECODE?

Looking in my /usr/lib64/python3.3 directory that particular problem has started to fix itself once I removed the line that sets it.

Maybe I am missing something.  What am I not realizing about this seemingly obvious and simple situation?

(In reply to Jim Dines from comment #21)
Jim, the bug you mean is #10758.

(In reply to Jim Dines from comment #21)

> Looking in my /usr/lib64/python3.3 directory that particular problem has
> started to fix itself once I removed the line that sets it.
> 
yes and now you have files in your system outside of the package management, that not belongs to any package and thus will stay even if you remove the package

> Maybe I am missing something.  What am I not realizing about this seemingly
> obvious and simple situation?
it seems yes


by the way all is now committed in Cauldron and will be there in Mageia 4
this bug, as 10758 will be closed when Mageia 4 final release will be there : https://wiki.mageia.org/en/Mageia_4_Development

>yes and now you have files in your system outside of the package management, >that not belongs to any package and thus will stay even if you remove the package

As will everyone who wasn't willing to wait for a whole new distribution to be released to fix the mistake of setting PYTHONDONTWRITEBYTECODE and not including the .py files in the package.  Of course, who cares / what does that really matter?  A few extra bytes of disk space will be used, that should have been used anyway? What was the alternative? Also, why would they still be on disk if I remove the python package?  They live in the same /usr/lib64/python2.7 and /usr/lib64/python3.3 directories.  Are you saying that if I remove the packages for python2.7 and python3.3 you leave those directories laying around?

Besides, fix the packages so they are correct and they won't be "out of the package" anymore, will they?

Is it that you don't understand how Python works, or are you just digging deeply to find a reason why I am somehow doing something wrong, when it is in fact the Mageia maintainers that have caused the problem?

>it seems yes

And yet it is No.

(In reply to Jim Dines from comment #24)

Jim, while I understand you are angry, please clam down. This is a bug-tracker and not a place for throwing mutt on each other.

I understand that you are dissatisfied that it takes so long. But taking into account, that nobody even cared about this bug until mid of this year, the solution way quite quick, wasn't it?

I'm the original reporter of this bug, and I'm very happy that it is solved at last. And a big thanks to phillippe who finally solved this and worked through all the python-packages to ensure it is fixed.

This will not be fixed for Mag 3, though I do not care. I lived and worked arounf with this for long, so I can easily wait another few month. And Mag 4 will have thei problem solved, see http://svnweb.mageia.org/packages/cauldron/python/releases/2.7.5/9.mga4/SPECS/python.spec?revision=486500&view=markup, approx. line 435.

(In reply to Philippe Makowski from comment #23)

Phillippe, please calm down, too :-) If Jim decided to have this work-around on this system, he may do.

Jim, while I#m using the same workaround, it has one big drawback: If you are uninstalling a package, the .pyc file will be left. While this is not a real problem regarding disk-space, this *may* lead to major problem if versions conflict: the "removed" package can still be imported -- and one may search for this bug quite some time.

Anyway here is the script you are asking for:

  python -m compileall /usr/lib*/python2.7*

(In reply to Hartmut Goebel from comment #25)
> (In reply to Jim Dines from comment #24)
> 
> Jim, while I understand you are angry, please clam down. This is a
> bug-tracker and not a place for throwing mutt on each other.

Well, I *AM* angry, and I was going to apologize, but I just went back and read what I wrote, and I was surprisingly tame.  It is not my intention to cause difficulties, and I would put this whole comment elsewhere, but it seems there is no place to file a bug against a prevailing attitude the way this Bugzilla is configured ;-)

I *DO* want to say that I have been a *LONG* time Linux user, and always have good things to say about Mageia.  I started using Mandrake in the 90s and I have stayed with them through all the Mandriva difficulties, etc.  Overall Mageia is a *GREAT* distribution.  I *ALWAYS* recommend it to people who want a really great distribution.  Seriously.

I have been developing software for more than 30 years, and mistakes happen ... all the time.  That isn't the problem.

I would be a lot less angry if I saw responses like "oops Hartmut" you are right, those should be config files, rather than:

"I'm not sure they really need to be marked as config files though.  Can't sysadmins just drop another file in /etc/profile.d to override it, rather than messing with a packaged file?" (Bug 10758) 

... and:

 "Exaggeration ( like "it lead to a lot of problem like : _1_ problem and _1_ second problem ) is far from being a good way to convince me, quite the contrary, especially when there is a easy work around ( like, you know, using "unset PYTHONDONTWRITEBYTECODE" in the shell )  for people who are skilled enough to see the problem ( and good enough to read the whole discussion especially where I explained the issue I have with the various solutions )."

> 
> I understand that you are dissatisfied that it takes so long. But taking
> into account, that nobody even cared about this bug until mid of this year,
> the solution way quite quick, wasn't it?

I think you are assuming that nobody cared because a bug wasn't filed.  The vast majority of people who care will simply edit the config file in /etc and move on with their life like I did because they are too busy to file a bug every time something is wrong, because as I already said, as software people we encounter that every day.

I cared, but I did as described, until it came back to byte me. (couldn't resist the pun ;-)

There seems to be a prevailing attitude of let the system admin fix it, and BTW they should know to do it magically in a non-standard way, and when they do fix it we will blame them for it.  The audacity of the "it seems yes" comment after all of this is far beyond the pale of acceptability.


> Anyway here is the script you are asking for:
> 
>   python -m compileall /usr/lib*/python2.7*

Thank you very much for that Hartmut!

I really would like to ask you a bunch of questions because you seem very knowledgeable, but this isn't the place.

To the Maintainers, I have changed my mind and I *WILL* apologize for being a bit testy, but I hope you will take it to heart what I said about the overall attitude I have seen on this issue.

To recap:

Mistakes happen.  Every Day.  We all make them.  Own them rather than trying to get someone else to own them.

You guys are doing a great job, and none of this changes my belief that Mageia is a great distribution!

Peace,

Jim D.

Jim, please just note that all will be fixed with Mageia 4
It was a lot of work to be sure that the change will go the right way, without trouble.

I appreciate very much the difficulties involved, and want to thank you for all of the good work you guys are doing.  None of what I wrote is meant to take away from that. As I said, I still think Mageia is one of the best distributions on the planet, and that compliment is not to be taken lightly, since as you have seen, I can be pretty critical at times ;-)

done in mga4

Status: ASSIGNED => RESOLVED
Resolution: (none) => FIXED