Mageia Bugzilla – Bug 17400
Deploy and configure MirrorBrain to manage mirrors and generate metalinks for DNF
Last modified: 2017-01-17 14:10:16 CET
Per the feature review meeting, MirrorBrain will be deployed to replace the existing mirror management system to enhance urpmi and DNF mirror handling (generating auto-redirecting URLs as well as generating mirrorlists for DNF), as well as helping us identify which packages are used the most to identify and prioritize maintenance of packages in the distribution.
This is also required for the successful completion of the integration of DNF for Mageia.
Steps to Reproduce:
DNF supports metalinks, which offer multiple levels of verification beyond what plain mirrorlists offer, so we should use those with MirrorBrain.
About moving bug 3166 from the "Depends on." field to "See Also:"
If you click on "See Also" you'll read about it:
This allows you to refer to bugs in other installations. You can enter a URL to a bug in the 'Add Bug URLs' field to note that that bug is related to this one. You can enter multiple URLs at once by separating them with a comma.
You should normally use this field to refer to bugs in other installations. For bugs in this installation, it is better to use the Depends on and Blocks fields.
Neal, if this bug doesn't depend on bug 3166, isn't this one then just another duplicate of it?
The reason I did not close this bug as duplicate is, that I assumed something additional would need to be done to mirrorbrain for DNF.
However, if just having mirrorbrain is enough, then this report is indeed a duplicate of bug 3166
Depending on how you look at it, it could be a duplicate (it refers to deploying MirrorBrain), or considered a related bug (MirrorBrain needs to be configured to support RPM-MD metalink data for DNF).
I hesitate to indicate this is a duplicate of bug 3166 because this bug also is the indicator that MirrorBrain deployment+configuration is part of Mageia 6 release engineering requirements (per the feature review linked in the description).
What do you think?
(In reply to Neal Gompa from comment #4)
> (MirrorBrain needs to be
> configured to support RPM-MD metalink data for DNF).
In that case, this is not a duplicate.
It is an additional step that needs to be done, so again setting this bug to depend on bug 3166
Of course, I do hope they'll get solved together, while MirrorBrain is configured for the very first time :-D
Indeed, I hope so too.
As of now, we have RPM-MD repodata being automatically generated on every package upload in Cauldron.
We now need MirrorBrain deployed and configured for our mirror management and generation of metalinks for DNF repodata.
Based on what I see for the Rawhide Metalinks, we need to use v3 Metalinks that point to the repomd.xml files in /repodata folders across each repository.
Preliminarily, I think this is the URL structure we want to have for our metalinks:
releasever: The Mageia release version, set to "cauldron" for Cauldron.
group: The different types of repositories (core, nonfree, tainted)
repo: The repository where the packages live (release, updates, updates_testing, backports, backports_testing)
extra: This part would be added if we're retrieving either debuginfo packages (debug) or source packages (source).
arch: The architecture (i586, x86_64, armv5tl)
* Note that Fedora usually uses $basearch definitions here, and in our case, $basearch would map i586 to i386 and armv5tl to arm. I'm not sure we care here, since unlike Fedora, we provide the full i586 repository as an option to people using 64-bit Mageia rather than a multilib repository. We would probably hardcode the architecture in the metalink URL in repo configuration files for DNF so that both repositories would work.
Some example structures of the system indicated above:
* Cauldron Core Release, x86_64: https://mirrors.mageia.org/metalink?repo=cauldron-core-release&arch=x86_64
* Cauldron Core Updates Testing Debug, i586: https://mirrors.mageia.org/metalink?repo=cauldron-core-updates_testing-debug&arch=i586
* Cauldron Core Release Sources, x86_64: https://mirrors.mageia.org/metalink?repo=cauldron-core-release-source&arch=x86_64
** From what I've seen, the metalink URL requires the arch parameter, but will wind up not using it for this specific case.
For Mageia 6 release onward, we should structure the releases like so:
The only difference to the guide above is that releasever will indicate an actual Mageia release version (such as 5, 6, etc.).
Just a small update...
I now have a mirrorbrain running on dl.mageia.org
(but I haven't pushed it on public dns yet as I want to review my changes after
some sleep to verify I haven't missed anything...)
for now I only use pseudo tree on dl.mageia.org until I get extra disks
Any progress on the MirrorBrain setup?
Have you had a chance to work on this lately?
Ok, so pterjan goes to DC today, so we should hopefully get more disks installed so we get the needed space to finish up infra work.
Was pterjan able to get more disks installed at the datacenter yesterday?
I found an interesting presentation that talked about how the metalinks work in Fedora, which also indicates that the URL can be used to do further constraints.
For example, tacking on "&country=us,ca" would constrain it to mirrors in the US and Canada. "&country=global" returns all mirrors everywhere. "ip=<ip.address>" would permit sending the client IP address so that the mirror manager would use it for GeoIP work.
I don't know how much more work it would be to add, but it'd be cool to have support for later on (after we get the basic mirror system up!).
As we talked about recently, you were working on a workaround or fixing null-rsync to work with our softlinks used in our distrib tree. Have you made any progress on this so that we can have the metalinks set up?
As a workaround I uploaded https://www.mageia.org/mirrorlist/?release=cauldron&arch=x86_64§ion=core&repo=updates_testing&debug=1
but there are message that I didn't get locally and on my test server:
no ideas yet, why they come
Not sure why, but it looks like it's generating URLs properly anyway. Perhaps you should add an error_reporting() function call at the top of your script to silence them?
(In reply to Manuel Hiebel from comment #16)
> As a workaround I uploaded
> but there are message that I didn't get locally and on my test server:
> no ideas yet, why they come
Thanks for a great idea. I'll try to dig in later today. I suspect some double include/require.
From both outputs I see two php server misconfigurations ;).
1: (our) production server should _never_ output any Notice (and Warnings). Outputting Errors is mostly undesired and possibly dangerous as it can increase attack surface.
We should file a bug for that. It would also solve some other issues like the one in bug #15292 #c3 and maybe more.
2: local dev server (in my opinion) should always output Notice, Warnings and Errors.
(In reply to Neal Gompa from comment #17)
> Perhaps you should add an error_reporting() function call at the top of
> your script to silence them?
While I use that sometimes just to be sure I think that modifying php.ini is the proper place to fix that. Of course on dev server the error level should not be above Notice. In our code we manage that in langs basefile.
I fixed it in geoip direclty. Looks ok.
I just committed some improvements to Manuels's mirrorlist but forgot to reference this BR.
It now returns a list of ten mirrors like centos does with priority: shuffled within the country, then continent and finally others.
To drop it's DL load it also excludes our source server (distrib-coffee.ipsl.jussieu.fr) now as a first choice in ISO and doc DL page and entirely from mirrorlist.
Could we have a status about this change? Do we still target Mageia 6 or should we target Mageia 7 now?
It's part of an accepted and implemented feature for Mageia 6, so it would be really nice to get this implemented. AFAIK though, it still requires:
- infra hardware upgrade, as we have many other infra-related issues that are blocked by having too little disk space
- sysadmin time to actually implement mirrorbrain
So I don't know if it will be realistically done before Mageia 6 is released, but I'd keep it on the milestone until it's certain it won't.
Any progress on this? I recall you mentioned you were going to work on it last week...
*** Bug 20130 has been marked as a duplicate of this bug. ***