Advisory: ======================== This update of perl-URPM enables urpmi to parse zstd compressed media metadata thus enabling to switch mga8 to use such zstd compressed metadata ======================== Updated packages in core/updates_testing: ======================== perl-URPM-5.23-1.mga7
Assignee: bugsquad => qa-bugs
Installed and tested without issue. The package perl-URPM was updated along with the rpm package updates. Tested with urpmi and rpmdrake tools. No issues found. ---------------- To satisfy dependencies, the following packages are going to be installed: Package Version Release Arch (medium "Core Updates Testing") lib64rpm8 4.14.2.1 13.mga7 x86_64 perl-URPM 5.23 1.mga7 x86_64 python2-rpm 4.14.2.1 13.mga7 x86_64 python3-rpm 4.14.2.1 13.mga7 x86_64 rpm 4.14.2.1 13.mga7 x86_64 rpm-plugin-ima 4.14.2.1 13.mga7 x86_64 rpm-plugin-syslog 4.14.2.1 13.mga7 x86_64 rpm-plugin-systemd-inhibit 4.14.2.1 13.mga7 x86_64 389KB of disk space will be freed. 996KB of packages will be retrieved. Proceed with the installation of the 8 packages? (Y/n) ---------------- $ uname -a Linux marte 5.4.6-desktop-2.mga7 #1 SMP Mon Dec 23 12:05:27 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux $ rpm -q perl-URPM perl-URPM-5.23-1.mga7
CC: (none) => mageia
Sending this along based on the test in Comment 1. Validating. Advisory in Comment 0.
CC: (none) => andrewsfarm, sysadmin-bugsKeywords: (none) => validated_updateWhiteboard: (none) => MGA7-64-OK
Unvalidating, as I want to test the metadata first But I realized our genhdlist2 only creates gzip hdlists regardless of flags... Only synthesis format changes. genhdlist2 --xml-info-filter ".zstd:zstd -19 -T8" --synthesis-filter ".zstd:zstd -19 -T8" . adding 1 new rpms not available in existing hdlist replacing ./media_info/hdlist.cz with hdlist.cz.tmp replacing ./media_info/synthesis.hdlist.zstd with synthesis.hdlist.zstd.tmp updating ./media_info/MD5SUM $ file media_info/* media_info/hdlist.cz: gzip compressed data, max compression, from Unix, original size modulo 2^32 2053332272 gzip compressed data, unknown method, has CRC, was "", has comment, encrypted, from FAT filesystem (MS-DOS, OS/2, NT), original size modulo 2^32 2053332272 media_info/MD5SUM: ASCII text media_info/synthesis.hdlist.zstd: Zstandard compressed data (v0.8+), Dictionary ID: None So I cant verify the hdlist part, only the synthesis is tested for now... We need to fix genhdlist2. and we need to decide the hdlist/synthersis name, as it currently does not autodetect .zstd LANG=C urpmi.addmedia test /home/tmb/x86_64 adding medium "test" adding 1 new rpms not available in existing hdlist replacing /var/cache/urpmi/partial/synthesis.hdlist.cz with synthesis.hdlist.cz.tmp updating /var/cache/urpmi/partial/MD5SUM Only if I add full media_info stuff: # LANG=C urpmi.addmedia test /home/tmb/x86_64 with media_info/synthesis.hdlist.zstd adding medium "test" or is it simpler to use ".cz" regardless of compression format ?
CC: (none) => tmbKeywords: validated_update => feedback
(In reply to Thomas Backlund from comment #3) > Unvalidating, as I want to test the metadata first > Good to know you have our backs when we are out of our depth, Thomas. Thanks.
(In reply to Thomas Backlund from comment #3) > > But I realized our genhdlist2 only creates gzip hdlists regardless of > flags... > Only synthesis format changes. > yeah, gzip is hardcoded in "sub build {}" I guess we need to add some "--hdlist-filter" flag to override it
(In reply to Thomas Andrews from comment #4) > (In reply to Thomas Backlund from comment #3) > > Unvalidating, as I want to test the metadata first > > > Good to know you have our backs when we are out of our depth, Thomas. > > Thanks. No worries :)
(In reply to Thomas Backlund from comment #3) > We need to fix genhdlist2. > > and we need to decide the hdlist/synthersis name, as it currently does not > autodetect .zstd > > or is it simpler to use ".cz" regardless of compression format ? Looking at current cauldron repo, we have in media_info for core: 20200106-103818-changelog.xml.lzma@ 20200106-103818-files.xml.lzma@ 20200106-103818-hdlist.cz@ 20200106-103818-info.xml.lzma@ 20200106-103818-synthesis.hdlist.cz@ changelog.xml.lzma files.xml.lzma hdlist.cz info.xml.lzma MD5SUM pubkey synthesis.hdlist.cz Should we unify all to zstd compression ? and do we name them .zstd or .cz ?
We don't really care about hdlists. Only synthesis are used by urpmi for deps computations. hdlists are more for heavy querying Cz is just an old convention, the format is autodetected We already switched from gzip to lzma then xz, we don't want to rename them, we've assumptions everywhere.
Created attachment 11447 [details] enable to alter hdlist compressor in genhdlist2 (not really wanted) which show that URPM would need some patching for reading hdlist compressed in anything other than hdlist. I could do it but I doubt it worths it…
ok, so to sum up hdlist is still .cz, still compressed with gzip xml files are still .lzma, still compressed with xz synthesis is also still .cz, now compressed with zstd (for mga8 that is) that way we keep potential breakage to a minimum / non-existant ...
The URPM issue is fixed in http://gitweb.mageia.org/software/rpm/perl-URPM/commit/?id=2afeac43e2a5c0d484100eedd192842d27944558 but as indicated, we don't need to alter hdlist compression, especially as it could seriously slow down generating metadata For comparing for reference: -rw-r--r-- 1 root 267M Gen 7 01:34 Mgz/hdlist.cz -rw-r--r-- 1 root 197M Gen 7 01:32 Mxz/hdlist.cz -rw-r--r-- 1 root 238M Gen 7 01:39 Mzstd/hdlist.cz
(In reply to Thomas Backlund from comment #10) > ok, so to sum up > > hdlist is still .cz, still compressed with gzip > xml files are still .lzma, still compressed with xz > synthesis is also still .cz, now compressed with zstd (for mga8 that is) > > that way we keep potential breakage to a minimum / non-existant ... yes XML files being lzma compressed (really xz) is hardcoded in urpm/xml_info.pm
Keywords: feedback => advisory, validated_update
An update for this issue has been pushed to the Mageia Updates repository. https://advisories.mageia.org/MGAA-2020-0010.html
Status: NEW => RESOLVEDResolution: (none) => FIXED