Bug 32402

Summary: Create a way to clean old leftover python files and directories, when updating python packages
Product: Mageia Reporter: Marc Krämer <mageia>
Component: RPM PackagesAssignee: Python Stack Maintainers <python>
Status: NEW --- QA Contact:
Severity: minor    
Priority: Normal CC: lewyssmith, marja11, yvesbrungard
Version: 9   
Target Milestone: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Source RPM: CVE:
Status comment:

Description Marc Krämer 2023-10-17 21:32:10 CEST
From old versions I can see python does not do a clean update:
find /usr/lib/python*|grep -v 3.10

/usr/lib/python2.7
/usr/lib/python2.7/site-packages
/usr/lib/python2.7/site-packages/BuildManager
/usr/lib/python2.7/site-packages/dot2tex
/usr/lib/python2.7/site-packages/gluster
/usr/lib/python2.7/site-packages/meld
/usr/lib/python2.7/site-packages/pybootchartgui
/usr/lib/python2.7/site-packages/pybootchartgui/__init__.pyc
/usr/lib/python2.7/site-packages/pybootchartgui/batch.pyc
/usr/lib/python2.7/site-packages/pybootchartgui/draw.pyc
/usr/lib/python2.7/site-packages/pybootchartgui/main.pyc
/usr/lib/python2.7/site-packages/pybootchartgui/parsing.pyc
/usr/lib/python2.7/site-packages/pybootchartgui/process_tree.pyc
/usr/lib/python2.7/site-packages/pybootchartgui/samples.pyc
/usr/lib/python2.7/site-packages/xcbgen
/usr/lib/python3.4
/usr/lib/python3.4/site-packages
/usr/lib/python3.5
/usr/lib/python3.5/site-packages
/usr/lib/python3.5/site-packages/MgaRepo
/usr/lib/python3.5/site-packages/MgaRepo/plugins
/usr/lib/python3.5/site-packages/MgaRepo/plugins/__pycache__
/usr/lib/python3.5/site-packages/MgaRepo/plugins/__pycache__/__init__.cpython-35.pyc
/usr/lib/python3.5/site-packages/dnf-plugins
/usr/lib/python3.5/site-packages/dnf-plugins/__pycache__
/usr/lib/python3.5/site-packages/urllib3
/usr/lib/python3.5/site-packages/urllib3/packages
/usr/lib/python3.5/site-packages/urllib3/packages/__pycache__
/usr/lib/python3.5/site-packages/urllib3/packages/__pycache__/six.cpython-35.pyc
/usr/lib/python3.7
/usr/lib/python3.7/site-packages
/usr/lib/python3.7/site-packages/MgaRepo
/usr/lib/python3.7/site-packages/MgaRepo/plugins
/usr/lib/python3.7/site-packages/MgaRepo/plugins/__pycache__
/usr/lib/python3.7/site-packages/MgaRepo/plugins/__pycache__/__init__.cpython-37.pyc
/usr/lib/python3.7/site-packages/dnf-plugins
/usr/lib/python3.7/site-packages/dnf-plugins/__pycache__
/usr/lib/python3.7/site-packages/meld
/usr/lib/python3.7/site-packages/sphinx
/usr/lib/python3.7/site-packages/sphinx/locale
/usr/lib/python3.7/site-packages/sphinx/locale/ar
/usr/lib/python3.7/site-packages/sphinx/locale/ar/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/bn
/usr/lib/python3.7/site-packages/sphinx/locale/bn/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ca
/usr/lib/python3.7/site-packages/sphinx/locale/ca/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/cs
/usr/lib/python3.7/site-packages/sphinx/locale/cs/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/cy
/usr/lib/python3.7/site-packages/sphinx/locale/cy/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/da
/usr/lib/python3.7/site-packages/sphinx/locale/da/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/de
/usr/lib/python3.7/site-packages/sphinx/locale/de/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/el
/usr/lib/python3.7/site-packages/sphinx/locale/el/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/eo
/usr/lib/python3.7/site-packages/sphinx/locale/eo/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/es
/usr/lib/python3.7/site-packages/sphinx/locale/es/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/et
/usr/lib/python3.7/site-packages/sphinx/locale/et/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/eu
/usr/lib/python3.7/site-packages/sphinx/locale/eu/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/fa
/usr/lib/python3.7/site-packages/sphinx/locale/fa/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/fi
/usr/lib/python3.7/site-packages/sphinx/locale/fi/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/fr
/usr/lib/python3.7/site-packages/sphinx/locale/fr/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/he
/usr/lib/python3.7/site-packages/sphinx/locale/he/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/hi
/usr/lib/python3.7/site-packages/sphinx/locale/hi/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/hi_IN
/usr/lib/python3.7/site-packages/sphinx/locale/hi_IN/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/hr
/usr/lib/python3.7/site-packages/sphinx/locale/hr/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/hu
/usr/lib/python3.7/site-packages/sphinx/locale/hu/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/id
/usr/lib/python3.7/site-packages/sphinx/locale/id/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/it
/usr/lib/python3.7/site-packages/sphinx/locale/it/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ja
/usr/lib/python3.7/site-packages/sphinx/locale/ja/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ko
/usr/lib/python3.7/site-packages/sphinx/locale/ko/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/lt
/usr/lib/python3.7/site-packages/sphinx/locale/lt/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/lv
/usr/lib/python3.7/site-packages/sphinx/locale/lv/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/mk
/usr/lib/python3.7/site-packages/sphinx/locale/mk/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/nb_NO
/usr/lib/python3.7/site-packages/sphinx/locale/nb_NO/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ne
/usr/lib/python3.7/site-packages/sphinx/locale/ne/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/nl
/usr/lib/python3.7/site-packages/sphinx/locale/nl/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/pl
/usr/lib/python3.7/site-packages/sphinx/locale/pl/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/pt
/usr/lib/python3.7/site-packages/sphinx/locale/pt/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/pt_BR
/usr/lib/python3.7/site-packages/sphinx/locale/pt_BR/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/pt_PT
/usr/lib/python3.7/site-packages/sphinx/locale/pt_PT/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ro
/usr/lib/python3.7/site-packages/sphinx/locale/ro/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ru
/usr/lib/python3.7/site-packages/sphinx/locale/ru/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/si
/usr/lib/python3.7/site-packages/sphinx/locale/si/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/sk
/usr/lib/python3.7/site-packages/sphinx/locale/sk/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/sl
/usr/lib/python3.7/site-packages/sphinx/locale/sl/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/sr
/usr/lib/python3.7/site-packages/sphinx/locale/sr/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/sr@latin
/usr/lib/python3.7/site-packages/sphinx/locale/sr@latin/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/sv
/usr/lib/python3.7/site-packages/sphinx/locale/sv/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/ta
/usr/lib/python3.7/site-packages/sphinx/locale/ta/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/tr
/usr/lib/python3.7/site-packages/sphinx/locale/tr/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/uk_UA
/usr/lib/python3.7/site-packages/sphinx/locale/uk_UA/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/vi
/usr/lib/python3.7/site-packages/sphinx/locale/vi/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/zh_CN
/usr/lib/python3.7/site-packages/sphinx/locale/zh_CN/LC_MESSAGES
/usr/lib/python3.7/site-packages/sphinx/locale/zh_TW
/usr/lib/python3.7/site-packages/sphinx/locale/zh_TW/LC_MESSAGES
/usr/lib/python3.8
/usr/lib/python3.8/site-packages
/usr/lib/python3.8/site-packages/MgaRepo
/usr/lib/python3.8/site-packages/MgaRepo/plugins
/usr/lib/python3.8/site-packages/MgaRepo/plugins/__pycache__
/usr/lib/python3.8/site-packages/MgaRepo/plugins/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/dnf-plugins
/usr/lib/python3.8/site-packages/dnf-plugins/__pycache__
Comment 1 Jani Välimaa 2023-10-18 01:41:34 CEST
Those leftover files and/or directories were/are not owned by any pkg and thus not removed when updating the pkgs.
Comment 2 Lewis Smith 2023-10-18 11:08:45 CEST
Thanks for your comment, Jani.
So what to do? Put this in ERRATA ?
How to close the bug?

CC: (none) => lewyssmith

Comment 3 Marc Krämer 2023-10-18 11:49:10 CEST
@Jani: e.g. at least usr/lib/python3.7/site-packages/MgaRepo/plugins/__pycache__/__init__.cpython-37.pyc must come from a package.
But why are cached files not put under /var/cache ? And if this is not an option, after uninstalling python the "__pycache__" dirs and the content should be removed.

I would like to see the system more clean instead of having old files lying around and there is no "tool" to find and remove them.
Comment 4 Marja Van Waes 2023-10-20 22:38:27 CEST
(In reply to Marc Krämer from comment #3)
> @Jani: e.g. at least
> usr/lib/python3.7/site-packages/MgaRepo/plugins/__pycache__/__init__.cpython-
> 37.pyc must come from a package.
> But why are cached files not put under /var/cache ? And if this is not an
> option, after uninstalling python the "__pycache__" dirs and the content
> should be removed.
> 
> I would like to see the system more clean instead of having old files lying
> around and there is no "tool" to find and remove them.

Assigning to the python stack maintainers, maybe they know of a way (or can create one) how to handle this more cleanly

Assignee: bugsquad => python
CC: (none) => marja11
Summary: python update does not clean site-packages => Create a way to clean old leftover python files and directories, when updating python packages

Comment 5 papoteur 2023-10-21 08:37:48 CEST
For mgarepo, I see that the difficulty comes from trying to separate mgarepo-ldap package. I think mgarepo-ldap fails to own 
/usr/lib/python3.8/site-packages/MgaRepo/plugins/__pycache__/__init__.cpython-38.pyc
This could be solved by adding 
%pycached %{python3_sitelib}/%{Uname}/plugins/__init__.py

But I wonder why it is useful to have this part separated from the main package.
Perhaps because of python3-ldap requirement. In this case, we can keep the package with only the README.LDAP and the Requires, without other files.

CC: (none) => yvesbrungard

Comment 6 Marc Krämer 2023-10-21 12:43:39 CEST
btw. I don't think this is a python only problem. E.g. for java-jdk:
ls /etc/java/java-*
/etc/java/java-11-openjdk:
java-11-openjdk-11.0.10.0.1-0.0.ea.2.mga8.x86_64/  java-11-openjdk-11.0.15.0.10-1.2.mga8.x86_64/  java-11-openjdk-11.0.18.0.10-1.mga8.x86_64/
java-11-openjdk-11.0.11.0.9-0.1.mga8.x86_64/       java-11-openjdk-11.0.15.0.10-1.mga8.x86_64/    java-11-openjdk-11.0.18.0.10-1.mga9.x86_64/
java-11-openjdk-11.0.13.0.8-2.1.mga8.x86_64/       java-11-openjdk-11.0.17.0.8-1.1.mga8.x86_64/

/etc/java/java-17-openjdk:
java-17-openjdk-17.0.7.0.7-1.mga9.x86_64/  java-17-openjdk-17.0.8.0.7-1.mga9.x86_64/

But I think this one has really wrong config settings....
Comment 7 papoteur 2023-10-21 18:10:20 CEST
Mgarepo should be fixed in cauldron.
Comment 8 katnatek 2023-10-21 19:26:49 CEST
the best way to do this is declaring in %files each folder and file that the python package provide, but could be a pain, I use this on package that fails if we don't clean the previous files

%posttrans
if ls %{python3_sitelib}/|grep %{pypi_name}|grep -v %{version} ; then
	rm -rf %{python3_sitelib}/$(ls %{python3_sitelib}/|grep %{pypi_name}|grep -v %{version})
fi