Bug 3358 - /usr/bin/virtualenv must not set PYTHONDONTWRITEBYTECODE
Summary: /usr/bin/virtualenv must not set PYTHONDONTWRITEBYTECODE
Status: RESOLVED FIXED
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: 2
Hardware: i586 Linux
Priority: Normal normal
Target Milestone: ---
Assignee: QA Team
QA Contact:
URL:
Whiteboard: has_procedure mga2-32-ok MGA2-64-OK
Keywords: Triaged, validated_update
Depends on:
Blocks:
 
Reported: 2011-11-16 12:59 CET by Hartmut Goebel
Modified: 2013-08-09 19:43 CEST (History)
9 users (show)

See Also:
Source RPM: python-virtualenv
CVE:
Status comment:


Attachments

Description Hartmut Goebel 2011-11-16 12:59:50 CET
http://svnweb.mageia.org/packages/updates/1/python-virtualenv/releases/1.6.4/2.mga2/SOURCES/virtualenv?revision=149032&view=markup

must not set PYTHONDONTWRITEBYTECODE, since this overwrites the user environment and choice. And the user has no chance to enforce his choice.
Hartmut Goebel 2011-11-16 13:00:46 CET

CC: (none) => h.goebel
Source RPM: (none) => python-virtualenv

Comment 1 Manuel Hiebel 2011-11-16 14:46:23 CET
Hi, thanks for reporting this bug.
Assigned to the package maintainer.

Keywords: (none) => Triaged
Assignee: bugsquad => lev

Comment 2 Marja Van Waes 2012-02-20 09:10:17 CET
Pinging, because nothing has happened with this report for more than 3 months, it still has the status NEW or REOPENED.

@ Lev

Same as in the other report:

Please set status to ASSIGNED if you think this bug was assigned correctly. If for work flow reasons you can't do that, then please put OK on the whiteboard instead.

Please reassign the bugs to bugsquad and explain if you think we were wrong to assign them to you.

CC: (none) => marja11

Comment 3 Lev Givon 2012-02-20 17:43:44 CET
By default, virtualenv creates virtual environments using setuptools rather than distribute. This causes virtualenv to complain if  PYTHONDONTWRITEBYTECODE is set (which is the default in Mageia):

The PYTHONDONTWRITEBYTECODE environment variable is not compatible with setuptools. Either use --distribute or unset PYTHONDONTWRITEBYTECODE.

Do we effectively want to require users to explicitly set the variable in order to use virtualenv with its default settings? (If we do, I can add an installation message to the package to warn users to unset the variable themselves.)

Status: NEW => ASSIGNED

Comment 4 Marja Van Waes 2012-02-20 19:11:55 CET
(In reply to comment #3)
> By default, virtualenv creates virtual environments using setuptools rather
> than distribute. This causes virtualenv to complain if  PYTHONDONTWRITEBYTECODE
> is set (which is the default in Mageia):
> 
> The PYTHONDONTWRITEBYTECODE environment variable is not compatible with
> setuptools. Either use --distribute or unset PYTHONDONTWRITEBYTECODE.
> 
> Do we effectively want to require users to explicitly set the variable in order
> to use virtualenv with its default settings? (If we do, I can add an
> installation message to the package to warn users to unset the variable
> themselves.)

@ Lev

Who, besides you, is "we"?

I'll put misc in the cc of this bug, I suppose you mean at least him :)

CC: (none) => misc

Comment 5 Lev Givon 2012-02-20 19:15:32 CET
we == Mageia developers/contributors
Comment 6 Marja Van Waes 2012-02-20 19:29:30 CET
(In reply to comment #5)
> we == Mageia developers/contributors

Thx :)

then maybe ask on Mageia-dev ml? (and/or Mageia-discuss?)
Comment 7 Hartmut Goebel 2012-03-03 17:46:59 CET
Mageia should not set PYTHONDONTWRITEBYTECODE at all. This would solve this issue, too. See bug 3348.
Comment 8 Marja Van Waes 2012-05-26 13:08:49 CEST
Hi,

This bug was filed against cauldron, but we do not have cauldron at the moment.

Please report whether this bug is still valid for Mageia 2.

Thanks :)

Cheers,
marja

Keywords: (none) => NEEDINFO

Comment 9 Hartmut Goebel 2012-05-29 10:50:23 CEST
This is still valid in Mageia 2, see 
http://svnweb.mageia.org/packages/updates/2/python-virtualenv/releases/1.6.4/2.mga2/SOURCES/virtualenv?revision=245591&view=markup


It's a shame this trivial fix takes to long -- even for a voluntary project. Discussion takes more time than the fix.

Version: Cauldron => 2

Sander Lepik 2012-05-29 11:02:09 CEST

Keywords: NEEDINFO => (none)
CC: (none) => sander.lepik
Version: 2 => Cauldron
Whiteboard: (none) => MGA2TOO

Comment 10 Manuel Hiebel 2012-05-29 12:06:28 CEST
(In reply to comment #9)
> This is still valid in Mageia 2, see 
> http://svnweb.mageia.org/packages/updates/2/python-virtualenv/releases/1.6.4/2.mga2/SOURCES/virtualenv?revision=245591&view=markup
> 
> 
> It's a shame this trivial fix takes to long -- even for a voluntary project.
> Discussion takes more time than the fix.

feel free to reopen the discussion on the -dev ml
Comment 11 Hartmut Goebel 2012-05-29 12:27:29 CEST
(In reply to comment #10)

> feel free to reopen the discussion on the -dev ml

Please note: This bug report is about forcefully setting PYTHONDONTWRITEBYTECODE in /usr/bin/virtualenv.

There is nothing to be discussed, as this is a poofen bug:

1) As PYTHONDONTWRITEBYTECODE is already inherited from the normal environment (if set there), there is not need to set PYTHONDONTWRITEBYTECODE in /usr/bin/virtualenv.

2) This is overwriting the users choice of setting or not setting PYTHONDONTWRITEBYTECODE. So this script is forcing the user into using PYTHONDONTWRITEBYTECODE even if he chooses to unset it.

BTW: Hanging the bar high ("reopen discussion on -dev ml") is not a good way for encouraging people to join the community.
Comment 12 Manuel Hiebel 2012-05-29 12:47:50 CEST
well this seems a political decision if I read the comment of lev, so yes the ml is the place for discuss of such things, or we can add more people in cc here, but I don't see who, or lev can fix or not and close the bug.
Comment 13 Hartmut Goebel 2012-05-29 13:01:45 CEST
(In reply to comment #3)
> By default, virtualenv creates virtual environments using setuptools rather
> than distribute. This causes virtualenv to complain if  PYTHONDONTWRITEBYTECODE
> is set (which is the default in Mageia):

This is a different problem and documented in bug 3348, which is about setting PYTHONDONTWRITEBYTECODE as default in Mageia.

This bug report is about forcefully setting PYTHONDONTWRITEBYTECODE in /usr/bin/virtualenv.

> Do we effectively want to require users to explicitly set the variable in order
> to use virtualenv with its default settings?

There is not need for setting the variable in /usr/bin/virtualenv. Please see my comment #11.
Comment 14 Lev Givon 2012-05-30 00:06:13 CEST
Given that the bug has remained open for a while (i.e., not many other folks seem to have weighed in on the matter) and given that virtualenv 1.7 does print an error when virtualenv is run with setuptools and PYTHONDONTWRITEBYTECODE is set, I've modified the cauldron package to not set PYTHONDONTWRITEBYTECODE.
Comment 15 Marja Van Waes 2012-05-30 07:23:08 CEST
(In reply to comment #14)
> Given that the bug has remained open for a while (i.e., not many other folks
> seem to have weighed in on the matter) and given that virtualenv 1.7 does print
> an error when virtualenv is run with setuptools and PYTHONDONTWRITEBYTECODE is
> set, I've modified the cauldron package to not set PYTHONDONTWRITEBYTECODE.

Thanks, Lev.

Version: Cauldron => 2
Summary: /usr/bin/virtualenv must not set => /usr/bin/virtualenv must not set PYTHONDONTWRITEBYTECODE
Whiteboard: MGA2TOO => (none)

Comment 16 Hartmut Goebel 2012-05-30 15:14:58 CEST
(In reply to comment #14)

Thanks.
Dorian Pula 2013-05-15 05:06:56 CEST

CC: (none) => dorian.pula

David Walser 2013-05-20 12:45:18 CEST

CC: (none) => cazzaniga.sandro

Sandro Cazzaniga 2013-05-20 13:04:51 CEST

Assignee: lev => cazzaniga.sandro

Comment 17 Sandro Cazzaniga 2013-05-20 15:13:00 CEST
@Lev: So the package is fixed in cauldron? We just have to push it to 2, now?
Philippe Makowski 2013-06-01 15:20:51 CEST

CC: (none) => makowski.mageia

Comment 18 Philippe Makowski 2013-06-01 16:45:46 CEST
The fix is ok in Mageia 3
Comment 19 Sandro Cazzaniga 2013-06-01 18:01:51 CEST
OK, so it just needs a backport in 2?
Comment 21 Sandro Cazzaniga 2013-06-01 18:24:35 CEST
I'll do it.
Comment 22 Sandro Cazzaniga 2013-06-01 18:32:54 CEST
It's in updates/Testing, 2.
Comment 23 Hartmut Goebel 2013-06-02 08:01:03 CEST
Being the original reported, I confirm this is fixed in Mageia 3.
Comment 24 Sandro Cazzaniga 2013-06-03 17:22:16 CEST
Can you test in mageia 2? :)
Comment 25 Hartmut Goebel 2013-06-06 23:16:41 CEST
Sorry, I alread upgraded all my systems.

Anyway: Since the script causing the patch is no longer part of the RPM, I'm very confident, the problem is solved for Magei 2, too. Just look into /usr/bin/virtualenv.
Comment 26 Hartmut Goebel 2013-07-19 20:15:05 CEST
Just for the records (and to close this bug):

The script causeing the problem ("virtualenv") has been removed in 
rev 434228, see
http://svnweb.mageia.org/packages?view=revision&revision=434228
and
http://svnweb.mageia.org/packages/updates/2/python-virtualenv/current/SPECS/python-virtualenv.spec?r1=296933&r2=434228

So the bug should be fixed.

But I could not spot an updated package on a mirror, so the package "python-virutalenv" still need to be rebuild for Mageia 2.
Comment 27 Philippe Makowski 2013-07-22 21:15:35 CEST
python-virtualenv-1.7.1.2-1.mga2.noarch.rpm is in core/updates_testing
Comment 28 Hartmut Goebel 2013-07-23 11:31:26 CEST
I checked /usr/bin/virtualenv in that RPM and it is okay.

I propose pushing it to core/updates and resolving this bug.
Philippe Makowski 2013-07-23 18:56:45 CEST

Assignee: cazzaniga.sandro => qa-bugs

Comment 29 claire robinson 2013-07-23 20:13:19 CEST
Thanks for testing Harmut. Can you say which arch you tested on i586 or x86_64 please, also if you can test the other one?

Philippe, could you please add a proper advisory.

Thankyou both :)
Comment 30 Philippe Makowski 2013-07-23 20:49:26 CEST
Suggested advisory:
========================

This build fix mga #3358 
python-virtualenv no longer set PYTHONDONTWRITEBYTECODE, and don't overwrites the user environment and choice.
========================


Updated packages in core/updates_testing:
========================
python-virtualenv-1.7.1.2-1.mga2.noarch

Source RPMs: 
python-virtualenv-1.7.1.2-1.mga2.src.rpm
Comment 31 Philippe Makowski 2013-07-23 20:51:20 CEST
(In reply to claire robinson from comment #29)
> Can you say which arch you tested on i586 or
> x86_64 please, also if you can test the other one?

it is a noarch package

> Philippe, could you please add a proper advisory.
done
Comment 32 claire robinson 2013-07-23 20:53:22 CEST
Thankyou. We still check both arches as much as we can. If for nothing else than to ensure they install cleanly.
Comment 33 claire robinson 2013-07-25 17:13:38 CEST
Testing mga2 32, it doesn't seem to work.

Before
------
# cat /usr/bin/virtualenv

#!/bin/bash

PYTHONDONTWRITEBYTECODE= /usr/bin/python -c "import virtualenv; virtualenv.main()" $*

$ cd test
$ virtualenv .
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.

$ ls
bin/  include/  lib/

$ rm -rf *

After
-----
# cat /usr/bin/virtualenv

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'virtualenv==1.7.1.2','console_scripts','virtualenv'
__requires__ = 'virtualenv==1.7.1.2'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('virtualenv==1.7.1.2', 'console_scripts', 'virtualenv')()
    )


$ virtualenv .
The PYTHONDONTWRITEBYTECODE environment variable is not compatible with setuptools. Either use --distribute or unset PYTHONDONTWRITEBYTECODE.

$ ls
$
claire robinson 2013-07-25 17:13:51 CEST

Whiteboard: (none) => feedback

Comment 34 Philippe Makowski 2013-07-25 19:07:36 CEST
your test is not correct, or it is but you don't make the good conclusion
of course in you "After" you have this result, because as said in comment #13 and comment #11, we in Mageia have a global setting for that.
The point is that, before the bug fix the user can't override this global setting now the user can.

your "After" test should be :

After
-----
# cat /usr/bin/virtualenv

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'virtualenv==1.7.1.2','console_scripts','virtualenv'
__requires__ = 'virtualenv==1.7.1.2'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('virtualenv==1.7.1.2', 'console_scripts', 'virtualenv')()
    )


$ virtualenv .
The PYTHONDONTWRITEBYTECODE environment variable is not compatible with setuptools. Either use --distribute or unset PYTHONDONTWRITEBYTECODE.

$ ls
$

$ unset PYTHONDONTWRITEBYTECODE
$ virtualenv .
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.
$ ls
bin/  include/  lib/  lib64@
Comment 35 claire robinson 2013-07-25 19:49:42 CEST
Is it something that should be unset by default Philippe as it currently will cause a regression with this package?
Hartmut Goebel 2013-07-26 10:10:46 CEST

Depends on: (none) => 10842

Sandro Cazzaniga 2013-07-26 10:12:14 CEST

CC: cazzaniga.sandro => (none)

Comment 36 Hartmut Goebel 2013-07-26 10:12:51 CEST
@claire: The regression you are describing in comment #33 is caused by bug 10842 (which I just filed as there was no such bug-report yet).
Philippe Makowski 2013-07-26 10:31:34 CEST

Depends on: 10842 => (none)

Comment 37 Philippe Makowski 2013-07-26 10:32:55 CEST
(In reply to claire robinson from comment #35)
> Is it something that should be unset by default Philippe as it currently
> will cause a regression with this package?

I don't think so
For me it is not really a regression.
Again, before the bug fix the user can't override the global setting PYTHONDONTWRITEBYTECODE now the user can.


What could be eventually done perhaps is to set --distribute as default ?
or let it as it is now with this fix, like it is in Mageia 3.
Comment 38 claire robinson 2013-07-26 11:00:45 CEST
It's really up to you Philippe. As just a simple QA folk, with limited understanding of the issues here, all I see is that it doesn't do with the update what it did do before. 

Our updates to stable releases shouldn't require user interaction, so to me it's a regression but we can defer to your expertise. I can see the positives.

Perhaps a README.update.urpmi would be useful in this case.
Comment 39 Philippe Makowski 2013-07-26 19:49:49 CEST
I made a patch, so by default when PYTHONDONTWRITEBYTECODE is set, it use distribute.

your "After" test should be :

After
-----
# cat /usr/bin/virtualenv

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'virtualenv==1.7.1.2','console_scripts','virtualenv'
__requires__ = 'virtualenv==1.7.1.2'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('virtualenv==1.7.1.2', 'console_scripts', 'virtualenv')()
    )


$ virtualenv .
New python executable in ./bin/python
Installing distribute.........................................................................................................................................................done.
Installing pip.....................done.

$ ls
bin/  include/  lib/
$ rm -rf *
$ ls
$ 

$ unset PYTHONDONTWRITEBYTECODE
$ virtualenv .
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.
$ ls
bin/  include/  lib/
$

Suggested advisory:
========================

This build fix mga #3358 
python-virtualenv no longer set PYTHONDONTWRITEBYTECODE, and don't overwrites the user environment and choice.
========================


Updated packages in core/updates_testing:
========================
python-virtualenv-1.7.1.2-1.1.mga2.noarch

Source RPMs: 
python-virtualenv-1.7.1.2-1.1.mga2.src.rpm
Comment 40 Dave Hodgins 2013-07-28 00:16:47 CEST
Advisory 3358.adv uploaded to svn.

CC: (none) => davidwhodgins
Whiteboard: feedback => (none)

Comment 41 claire robinson 2013-07-29 11:07:14 CEST
At the moment there is no readme.update.urpmi

$ cd test
$ ls
$ virtualenv .
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.
$ ls
bin/  include/  lib/

Update..

$ rm -rf *
$ virtualenv .
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

Even with user workaround..

$ unset PYTHONDONTWRITEBYTECODE
$ virtualenv .
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 5, in <module>
    from pkg_resources import load_entry_point
ImportError: No module named pkg_resources

Philippe, there is obviously an issue here but apart from that, if a user is required to 'unset PYTHONDONTWRITEBYTECODE' to maintain current functionality then that info should be added in a README.update.urpmi
claire robinson 2013-07-29 11:08:32 CEST

Whiteboard: (none) => feedback

Comment 42 Philippe Makowski 2013-07-29 16:12:11 CEST
seems you just hit a new bug, a missing require on python-pkg-resources
and no need for readme.update.urpmi, I would really try to avoid that.
Comment 43 Philippe Makowski 2013-07-29 20:32:47 CEST
I added the missing requires, sorry for the delay, but you made a good catch with your minimal setup, even if it would be strange that someone using virtualenv for real would not have already this package. 
thanks anyway


Suggested advisory:
========================

This build fix mga #3358 
python-virtualenv no longer set PYTHONDONTWRITEBYTECODE, and don't overwrites the user environment and choice.
========================


Updated packages in core/updates_testing:
========================
python-virtualenv-1.7.1.2-1.2.mga2.noarch

Source RPMs: 
python-virtualenv-1.7.1.2-1.2.mga2.src.rpm
claire robinson 2013-07-31 11:57:21 CEST

Whiteboard: feedback => (none)

Comment 44 claire robinson 2013-07-31 12:06:28 CEST
Nice job Philippe. Testing complete mga2 32

Before
------
$ cd test
$ virtualenv .
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.
$ ls
bin/  include/  lib/

After
-----
Confirmed the added require

# urpmi python-virtualenv
To satisfy dependencies, the following packages are going to be installed:
  Package                        Version      Release       Arch    
(medium "Core Release")
  python-pkg-resources           0.6.24       2.mga2        noarch  
(medium "Core Updates Testing")
  python-virtualenv              1.7.1.2      1.2.mga2      noarch  
106KB of additional disk space will be used.
1MB of packages will be retrieved.
Proceed with the installation of the 2 packages? (Y/n) y

$ rm -rf *
$ virtualenv .
New python executable in ./bin/python
Installing distribute.........................................................................................................................................................done.
Installing pip.....................done.
$ ls
bin/  include/  lib/

$ echo $PYTHONDONTWRITEBYTECODE
1
$ unset PYTHONDONTWRITEBYTECODE
$ echo $PYTHONDONTWRITEBYTECODE

$ rm -rf *
$ virtualenv .
New python executable in ./bin/python
Installing setuptools............done.
Installing pip...............done.
$ ls
bin/  include/  lib/

$ echo $PYTHONDONTWRITEBYTECODE

$

So now no user interaction is required and it works with or without PYTHONDONTWRITEBYTECODE set and doesn't overwrite the setting.

Whiteboard: (none) => has_procedure mga2-32-ok

Comment 45 Philippe Makowski 2013-08-07 11:16:53 CEST
are these pushed ?
Comment 46 claire robinson 2013-08-07 11:22:34 CEST
Not yet, it seems
Comment 47 Dave Hodgins 2013-08-07 21:47:55 CEST
Advisory 3358.adv updated on svn for srpm python-virtualenv-1.7.1.2-1.2.mga2

Testing Mageia 2 x86_64 shortly.
Comment 48 Dave Hodgins 2013-08-07 21:59:33 CEST
Testing complete on Mageia 2 x86_64 using procedure from comment 44.

Could someone from the sysadmin team push 3358.adv to updates.

Keywords: (none) => validated_update
Whiteboard: has_procedure mga2-32-ok => has_procedure mga2-32-ok MGA2-64-OK
CC: (none) => sysadmin-bugs

Comment 49 Thomas Backlund 2013-08-09 19:43:07 CEST
Update pushed:
http://advisories.mageia.org/MGAA-2013-0077.html

Status: ASSIGNED => RESOLVED
CC: (none) => tmb
Resolution: (none) => FIXED


Note You need to log in before you can comment on or make changes to this bug.