Bug 2157

a comment here https://mageia.org/wiki/doku.php?do=show&id=iso2%3Atechnical_specification :

sounds interesting but needs to have 2 kinds of metadata in parallel while it's not all integrated - discussions needed

URL: (none) => https://mageia.org/wiki/doku.php?id=iso2:technical_specification
CC: (none) => marja11

I think this would be helpful, yes. 

I only have 2 thing to add :
1) is the format of repomd specified ? I remember seeing developpers of packagekit and zypper being unhappy with a change of format. 

2) can we use createrepo ? is this as fast our current tools ? The slow pace of update and indexes generation on fedora is IMHO  problematic ( slow pace being that we have to wait 2 days to get updates-testing on mirrors ), but I do not know if the problem is their tools or something else.

CC: (none) => misc

@ cjw

can you answer misc's questions?

AFAICT the format is standard but it can be extended.

I don't know enough about all the features that urpmi and GUI package tools provide in mageia. Createrepo provides at least the basic info required for package installation and upgrades. WRT speed we should probably be more concerned about the time it takes to read repomd data (in urpmi/perl-URPM).

tracked at https://bugs.launchpad.net/rpm/+bug/913192

CC: (none) => n3npq

No news

Target Milestone: --- => Mageia 3
Summary: Switch to "standard" rpm metadata for package repositories in Mageia 2 => Switch to "standard" rpm metadata for package repositories
Source RPM: (none) => rpm
Severity: normal => enhancement

I'm against such a move.
yum metadata are bigger and slower to parse.

For updating packages, compare:
1.7M    synthesis.hdlist.cz
4,4M    primary.xml.gz

For retrieving metadata from rpmdrake, compare:
8,7M    files.xml.lzma
11M     filelists.xml.gz

And yum really is quite much slower than urpmi and I suspect the used metada format doesn't help...

As for the comment about recompressing with XZ, that wouldn't change the ration since synthesis would gain from changing compressor (we already support gz, bzip2, xz, ...)

Last but not least, it would have big impacts on perl-URPM, urpmi, rpmdrake and drakx installer.

CC: (none) => thierry.vignaud

Also did someone check generation time?

Opensuse does not uses it on factory but only on the different smaller repositories and stable releases because "createrepo on factory takes hours and we prefer to push out ftp trees quicker".

CC: (none) => pterjan

I think we should close this one as WONTFIX

Source RPM: rpm => rpmtools

(In reply to comment #9)
> I think we should close this one as WONTFIX

@ misc
@ cjw

If there are no objections, I intend to close this report as WONTFIX two weeks from now.

Whoever wants to object: please remove the NEEDINFO keyword and give your motivation

Keywords: (none) => NEEDINFO

keep for future spec

Keywords: NEEDINFO => (none)
Blocks: 1994 => (none)

Hi,

This bug was filed against cauldron, but we do not have cauldron at the moment.

Please report whether this bug is still valid for Mageia 2.

Thanks :)

Cheers,
marja

Keywords: (none) => NEEDINFO

(In reply to comment #11)
> keep for future spec

OK :)

Keywords: NEEDINFO => (none)

As there is no plan to change this soon, I think it should be closed. It can still be reopened later if things change.

CC: (none) => boklm

(In reply to comment #14)
> As there is no plan to change this soon, I think it should be closed. It can
> still be reopened later if things change.

@ Leuhmanu

Sorry, tv and boklm for closing: you're the minority 

Closing as WONTFIX for now

Status: NEW => RESOLVED
Resolution: (none) => WONTFIX

I would like to request the reopening of this bug! 

Switch to "standard" rpm metadata for package repositories.

This will allow us to use YUM and APT-RPM that are in repositories Mageia. In addition to improving support for other managers.

Standardization is important and is a current trend.

CC: (none) => frateraec

Unless someone works on generating those metadata at a decent speed, this is not going to happen.

Currently uploading a package takes a few minutes, it would take a few hours.

Please advise as the other distros rpm do and how to do Mageia justify the impossibility.

As I quoted, opensuse does not do it on their main development repository because it takes several hours.

We may be able to do like them and only to it for stable releases but then it would not be consistent with the development version and not be properly tested.

I don't know how Fedora does it

It could be for Mageia 3 or maybe Mageia 4 stable release?

Reading from an old RH advisory from 2008 there are some options that may help:

=====
As well, this updated package adds a new option flag, "--skip-stat". When
used in conjunction with "--update", createrepo skips calling stat(1) on
files to see if they have changed, and assumes that if the file name
matches, then the repodata can be re-used.

This option has shown a significant increase in performance in the Fedora
build system, cutting down the time it takes createrepo to run, from 40
minutes, to approximately 4.
=====

4 minutes is still quite high but more reasonable.

When uploading a package we have to run it 4 times (i586, x86_64, debug i586, debug x86_64) and currently the total of 4 runs of genhdlist2 takes about 3 minutes (and I had started working on making it faster).

Let's test the performance with that options.

On my home machine on core/release i586 (20115 packages):

First generation with genhdlist2 --clean

real	1m52.755s
user	1m36.790s
sys	0m1.950s

Update with genhdlist2 (and no new package)

real	1m0.889s
user	1m36.080s
sys	0m1.540s

First generation with createrepo

real	10m49.949s
user	6m15.240s
sys	0m12.380s

Update with createrepo --update --skip-stat (and no new package)

real	0m56.652s
user	0m49.190s
sys	0m3.470s


So creating from scratch (which does not happen so often, mostly when we have bugs causing signature problems) takes 5 times longer, for the 4 repositories it would be 44 minutes instead of 7 minutes 30 but updating incrementally (which happens at each package upload) is slightly faster.

Status: RESOLVED => REOPENED
Resolution: WONTFIX => (none)

Actually createrepo --update without --skip-stat is fast too :)

real	0m55.352s
user	0m49.310s
sys	0m3.330s

isn't this fixed already?

CC: (none) => alien

As far as I know, nothing has been done.

i thought we only used the xml and synthesis and not the hdlists themselves?

should we move this feature to mga4 then?

The biggest problem is not the technique. The greatest difficulty is to awaken the enterece in who can do it...

(In reply to AL13N from comment #25)
> i thought we only used the xml and synthesis and not the hdlists themselves?

Which is not the same as repomd.

(In reply to AL13N from comment #26)
> should we move this feature to mga4 then?

It is not clear whether we want this, and whether there is someone who wants to implement this.

There's no such thing as standard metadata for rpm.
IMHO "standard rpm metadata" is a lie as it's not for rpm at all for the higher level package manager (yum, urpmi, ...), each one having its format.
If I were to change format, I would consider switching to libsolv format instead (and to use libsolv as resolver), not yum format.

ok, so better set to WONTFIX then until someone comes along and actually wants to do the work of changing format IF it's actually needed...

Closing. Again.

Status: REOPENED => RESOLVED
Resolution: (none) => WONTFIX

rereading this, i think the stats speak for themselves. time to let the beast rest... forgood.

Could you explain exactly what you mean?

it means that repomd is slower(x5) and bigger than what we have now...

it also means that format-chaning has effects on alot of parts of the distro...

it also means that the primary developer would rather want libsolv rather than repomd, because of his preferences.

iow: not gonna happen.

The main problem is really urpmi and this rpm standardization would be useful to use other package managers. URPMI trying to be friendly often only creates problems. Eg: only show the latest version of the package in repository. Does not downgrade transparently. If you have installed community repositories URPMI hides official packages, etc ... Fortunately now we can use SMART on Mageia 3. I follow the distro since Mandrake 7 and urpmi has always been limited in some points.

perhaps it can be improved instead? (in another bug report)

iinm --oldpackage was ported to urpmi, i think i saw a change about this...

also the latest version, that's just rpmdrake filtering, one could make a patch to show all if one wanted it bad enough...

listen, we can't use repomd, because the whole buildsystem would slow down 5x more..., we could maybe only do repomd for releases, but then it would have bad testing and such...

I understand. So I said the question of urpmi. It is more easier to improve urpmi than to modify the standard. I had already reported the support -- downgrade for urpmi and it has already been partially implemented. Now then is to report more bugs enhancement of urpmi.

https://bugs.mageia.org/show_bug.cgi?id=6655