Bug 32329 - Software improvements for the build system
Summary: Software improvements for the build system
Status: NEW
Alias: None
Product: Infrastructure
Classification: Unclassified
Component: Others (show other bugs)
Version: unspecified
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Sysadmin Team
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-09-27 11:56 CEST by Giuseppe Ghibò
Modified: 2023-12-29 17:21 CET (History)
3 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments

Description Giuseppe Ghibò 2023-09-27 11:56:57 CEST
During the discussion of upgrading the current hardware infrastructure (as of bug #32306), it also emerged the need to improve the current build system also from software point of view. E.g. with the possibility to use or integrate other systems, as well as to use git as alternative to svn.

Some other system, already in use in other distributions, were suggested, e.g. "rpkg" (see https://pagure.io/rpkg), as alternative or complementary to mgarepo (former repsys).

For svn(web) -> git(web) there was already this pending report, that I remind here:

https://bugs.mageia.org/show_bug.cgi?id=20351

We can keep this bug to track the progress on this subject.
Comment 1 Giuseppe Ghibò 2023-09-27 12:03:49 CEST
I do not enter in details, but as mageia usually plans to keep a certain continuity with past releases, the main obstacle of adopting other systems, is that they are based on dnf rather than urpmi, so at the end of the building process the package metadata are generated for dnf, and not for urpmi. So IMHO the main requirement for a hybrid approach is that they should provide at the end of the build process, the metadata also for urpmi.
Bruno Cornec 2023-09-28 01:01:35 CEST

CC: (none) => bruno

Comment 2 katnatek 2023-09-28 03:48:19 CEST
(In reply to Giuseppe Ghibò from comment #1)
> I do not enter in details, but as mageia usually plans to keep a certain
> continuity with past releases, the main obstacle of adopting other systems,
> is that they are based on dnf rather than urpmi, so at the end of the
> building process the package metadata are generated for dnf, and not for
> urpmi. So IMHO the main requirement for a hybrid approach is that they
> should provide at the end of the build process, the metadata also for urpmi.

A package "generated for dnf" works perfectly for urpmi, just have to generate the metadata for urpmi and of course for dnf

I use fedora copr (mock+dnf) to build some of my rpms (the more resource hungry or just to speed the build for the 2 arches) and with exception of packaging errors i don't have any issue using the packages installed for urpmi
Comment 3 katnatek 2023-09-28 03:53:07 CEST
Forget my comment#2 i just missread

BTW make the metadata for urpmi and dnf is a good thing as long as we support dnf as alternative, that will helps also to not have outdated metadata for dnf like happen in some times
Comment 4 Giuseppe Ghibò 2023-09-28 09:31:23 CEST
(In reply to katnatek from comment #2)

> (In reply to Giuseppe Ghibò from comment #1)
> > I do not enter in details, but as mageia usually plans to keep a certain
> > continuity with past releases, the main obstacle of adopting other systems,
> > is that they are based on dnf rather than urpmi, so at the end of the
> > building process the package metadata are generated for dnf, and not for
> > urpmi. So IMHO the main requirement for a hybrid approach is that they
> > should provide at the end of the build process, the metadata also for urpmi.
> 
> A package "generated for dnf" works perfectly for urpmi, just have to
> generate the metadata for urpmi and of course for dnf
> 
> I use fedora copr (mock+dnf) to build some of my rpms (the more resource
> hungry or just to speed the build for the 2 arches) and with exception of
> packaging errors i don't have any issue using the packages installed for
> urpmi

True, of course RPMs do works, but that's for single packages. Installed manually from urpmi pointing directory to the URL of the single rpm file. But for a group of packages generated in a repo, the group of packages would have the 'repodata' metadata dir, but misses the 'media_info' dir for urpmi, because they are not generated. Of course that build system can continue to be able to generate other packages using packages generated from the same repo (because it uses dnf+mock on itself), but the final repo can't be pointed directly to urpmi, because of missed media_info. And this especially true if you don't own the system they are built on. Basically they can't be generated because on that build system doesn't exists the genhdlist2 utils to generate them (which belongs to an infra [urpmi stck] not backported).
Comment 5 katnatek 2023-09-28 22:04:28 CEST
(In reply to Giuseppe Ghibò from comment #4)
I will tell you what i do

I generate the rpms in copr
Download (with script if are a lot) to temporally area
sign rpms
Move to the are from users fetch the rpm
Generate dnf and urpmi metadata

Some like this can be done just instead of download is needed to move rpms ;)
Comment 6 Giuseppe Ghibò 2023-09-28 22:15:03 CEST
(In reply to katnatek from comment #5)

> (In reply to Giuseppe Ghibò from comment #4)
> I will tell you what i do
> 
> I generate the rpms in copr
> Download (with script if are a lot) to temporally area
> sign rpms
> Move to the are from users fetch the rpm
> Generate dnf and urpmi metadata
> 
> Some like this can be done just instead of download is needed to move rpms ;)

yep, you might mirror the rpms locally, work on them, and repush on a 2nd host, with the required metadata, but require an 2nd host...

Another solution would be to work on the tool generating them at source and let (with some hook or something other or porting the required packages) then generate the urpmi metadata during the latest stage of remote building process.
Comment 7 katnatek 2023-09-29 19:28:53 CEST
(In reply to Giuseppe Ghibò from comment #6)
> yep, you might mirror the rpms locally, work on them, and repush on a 2nd
> host, with the required metadata, but require an 2nd host...
> 
Well i do that because i need to do it, but for mageia the build tool could be tweaked to generate the rpms on custom path, so you can skip a step

mock have a --resutldir option allow you make all rpms you are interested  go, by example, to "Core Updates Testing" , other tools maybe need to be configured or adapted to do something like that
Comment 8 Neal Gompa 2023-12-29 17:21:27 CET
Generating urpmi metadata can be done as part of the publishing step to mirrors, which is independent on the build system anyway.

CC: (none) => ngompa13


Note You need to log in before you can comment on or make changes to this bug.