Bug 17400 - Deploy and configure MirrorBrain to manage mirrors and generate metalinks for DNF
: Deploy and configure MirrorBrain to manage mirrors and generate metalinks for...
Status: NEW
Product: Infrastructure
Classification: Unclassified
Component: BuildSystem
: unspecified
: All Linux
: High Severity: normal
: Mageia 6
Assigned To: Sysadmin Team
:
:
:
:
: 3166 17057
: 4041 16002
  Show dependency treegraph
 
Reported: 2015-12-25 20:37 CET by Neal Gompa
Modified: 2017-01-17 14:10 CET (History)
8 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments

Description Neal Gompa 2015-12-25 20:37:38 CET
Per the feature review meeting[0], MirrorBrain will be deployed to replace the existing mirror management system to enhance urpmi and DNF mirror handling (generating auto-redirecting URLs as well as generating mirrorlists for DNF), as well as helping us identify which packages are used the most to identify and prioritize maintenance of packages in the distribution.

This is also required for the successful completion of the integration of DNF for Mageia.

[0]: http://meetbot.mageia.org/mageia-dev/2015/mageia-dev.2015-11-11-20.09.html

Reproducible: 

Steps to Reproduce:
Comment 1 Neal Gompa 2016-01-30 14:51:19 CET
DNF supports metalinks, which offer multiple levels of verification beyond what plain mirrorlists offer, so we should use those with MirrorBrain.
Comment 2 Marja van Waes 2016-02-09 08:56:07 CET
@ Neal

About moving bug 3166 from the "Depends on." field to "See Also:"

If you click on "See Also" you'll read about it:

________________________________________________________________________________

See Also
    This allows you to refer to bugs in other installations. You can enter a URL to a bug in the 'Add Bug URLs' field to note that that bug is related to this one. You can enter multiple URLs at once by separating them with a comma.

    You should normally use this field to refer to bugs in other installations. For bugs in this installation, it is better to use the Depends on and Blocks fields.

________________________________________________________________________________


Neal, if this bug doesn't depend on bug 3166, isn't this one then just another duplicate of it?
Comment 3 Marja van Waes 2016-02-09 09:00:33 CET
The reason I did not  close this bug as duplicate is, that I assumed something additional would need to be done to mirrorbrain for DNF.

However, if just having mirrorbrain is enough, then this report is indeed a duplicate of bug 3166
Comment 4 Neal Gompa 2016-02-09 09:01:19 CET
@Marja:

Depending on how you look at it, it could be a duplicate (it refers to deploying MirrorBrain), or considered a related bug (MirrorBrain needs to be configured to support RPM-MD metalink data for DNF).

I hesitate to indicate this is a duplicate of bug 3166 because this bug also is the indicator that MirrorBrain deployment+configuration is part of Mageia 6 release engineering requirements (per the feature review linked in the description).

What do you think?
Comment 5 Marja van Waes 2016-02-09 09:13:33 CET
(In reply to Neal Gompa from comment #4)
> @Marja:
> 
> (MirrorBrain needs to be
> configured to support RPM-MD metalink data for DNF).
> 

In that case, this is not a duplicate.

It is an additional step that needs to be done, so again setting this bug to depend on bug 3166

Of course, I do hope they'll get solved together, while MirrorBrain is configured for the very first time :-D
Comment 6 Neal Gompa 2016-02-09 09:56:07 CET
@Marja:

Indeed, I hope so too.
Comment 7 Neal Gompa 2016-03-14 03:31:01 CET
As of now, we have RPM-MD repodata being automatically generated on every package upload in Cauldron.

We now need MirrorBrain deployed and configured for our mirror management and generation of metalinks for DNF repodata.

Based on what I see for the Rawhide Metalinks[0][1][2], we need to use v3 Metalinks that point to the repomd.xml files in /repodata folders across each repository.

Preliminarily, I think this is the URL structure we want to have for our metalinks:

https://mirrors.mageia.org/metalink?repo=<releasever>-<group>-<repo>[-<extra>]&arch=<arch>

releasever: The Mageia release version, set to "cauldron" for Cauldron.

group: The different types of repositories (core, nonfree, tainted)

repo: The repository where the packages live (release, updates, updates_testing, backports, backports_testing)

extra: This part would be added if we're retrieving either debuginfo packages (debug) or source packages (source).

arch: The architecture (i586, x86_64, armv5tl)
* Note that Fedora usually uses $basearch definitions here, and in our case, $basearch would map i586 to i386 and armv5tl to arm. I'm not sure we care here, since unlike Fedora, we provide the full i586 repository as an option to people using 64-bit Mageia rather than a multilib repository. We would probably hardcode the architecture in the metalink URL in repo configuration files for DNF so that both repositories would work.

Some example structures of the system indicated above:

* Cauldron Core Release, x86_64: https://mirrors.mageia.org/metalink?repo=cauldron-core-release&arch=x86_64

* Cauldron Core Updates Testing Debug, i586: https://mirrors.mageia.org/metalink?repo=cauldron-core-updates_testing-debug&arch=i586

* Cauldron Core Release Sources, x86_64: https://mirrors.mageia.org/metalink?repo=cauldron-core-release-source&arch=x86_64
** From what I've seen, the metalink URL requires the arch parameter, but will wind up not using it for this specific case.


[0]: https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1]: https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=x86_64
[2]: https://mirrors.fedoraproject.org/metalink?repo=rawhide-source&arch=x86_64
Comment 8 Neal Gompa 2016-03-14 18:02:25 CET
For Mageia 6 release onward, we should structure the releases like so:

https://mirrors.mageia.org/metalink?repo=mageia-<releasever>-<group>-<repo>[-<extra>]&arch=<arch>

The only difference to the guide above is that releasever will indicate an actual Mageia release version (such as 5, 6, etc.).
Comment 9 Thomas Backlund 2016-03-15 00:08:46 CET
Just a small update...

I now have a mirrorbrain running on dl.mageia.org
(but I haven't pushed it on public dns yet as I want to review my changes after
some sleep to verify I haven't missed anything...)

for now I only use pseudo tree on dl.mageia.org until I get extra disks
installed
Comment 10 Neal Gompa 2016-03-19 20:35:18 CET
@Thomas:

Any progress on the MirrorBrain setup?
Comment 11 Neal Gompa 2016-04-13 03:22:11 CEST
@Thomas:

Have you had a chance to work on this lately?
Comment 12 Thomas Backlund 2016-04-15 09:05:59 CEST
Ok, so pterjan goes to DC today, so we should hopefully get more disks installed so we get the needed space to finish up infra work.
Comment 13 Neal Gompa 2016-04-16 15:23:56 CEST
@Thomas:

Was pterjan able to get more disks installed at the datacenter yesterday?
Comment 14 Neal Gompa 2016-05-01 01:52:10 CEST
I found an interesting presentation[1] that talked about how the metalinks work in Fedora, which also indicates that the URL can be used to do further constraints.

For example, tacking on "&country=us,ca" would constrain it to mirrors in the US and Canada. "&country=global" returns all mirrors everywhere. "ip=<ip.address>" would permit sending the client IP address so that the mirror manager would use it for GeoIP work.

I don't know how much more work it would be to add, but it'd be cool to have support for later on (after we get the basic mirror system up!).

[1]: http://domsch.com/linux/fedora/mirrormanager-fudconf13.odp
Comment 15 Neal Gompa 2016-05-17 13:54:28 CEST
@tmb:

As we talked about recently, you were working on a workaround or fixing null-rsync to work with our softlinks used in our distrib tree. Have you made any progress on this so that we can have the metalinks set up?
Comment 16 Manuel Hiebel 2016-05-30 00:30:32 CEST
As a workaround I uploaded https://www.mageia.org/mirrorlist/?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1

but there are message that I didn't get locally and on my test server:
http://mageia-test.hiebel.eu/mirrorlist/?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1

no ideas yet, why they come
Comment 17 Neal Gompa 2016-05-30 00:53:44 CEST
Not sure why, but it looks like it's generating URLs properly anyway. Perhaps you should add an error_reporting() function call[1] at the top of your script to silence them?

[1]: http://php.net/manual/en/function.error-reporting.php
Comment 18 Filip Komar 2016-05-30 11:29:23 CEST
(In reply to Manuel Hiebel from comment #16)
> As a workaround I uploaded
> https://www.mageia.org/mirrorlist/
> ?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1
> 
> but there are message that I didn't get locally and on my test server:
> http://mageia-test.hiebel.eu/mirrorlist/
> ?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1
> 
> no ideas yet, why they come

Thanks for a great idea. I'll try to dig in later today. I suspect some double include/require.

From both outputs I see two php server misconfigurations ;).

1: (our) production server should _never_ output any Notice (and Warnings). Outputting Errors is mostly undesired and possibly dangerous as it can increase attack surface.
We should file a bug for that. It would also solve some other issues like the one in bug #15292 #c3 and maybe more.

2: local dev server (in my opinion) should always output Notice, Warnings and Errors.


(In reply to Neal Gompa from comment #17)
> Perhaps you should add an error_reporting() function call[1] at the top of
> your script to silence them?

While I use that sometimes just to be sure I think that modifying php.ini is the proper place to fix that. Of course on dev server the error level should not be above Notice. In our code we manage that in langs basefile[0].

[0] http://gitweb.mageia.org/web/www/tree/langs.php
Comment 19 Manuel Hiebel 2016-05-30 13:01:42 CEST
I fixed it in geoip direclty. Looks ok.
Comment 20 Filip Komar 2016-06-05 16:04:22 CEST
I just committed[0] some improvements to Manuels's mirrorlist[1] but forgot to reference this BR.

It now returns a list of ten mirrors like centos does[2] with priority: shuffled within the country, then continent and finally others.

To drop it's DL load it also excludes our source server (distrib-coffee.ipsl.jussieu.fr) now as a first choice in ISO and doc DL page and entirely from mirrorlist.


[0] http://gitweb.mageia.org/web/www/commit/?id=5552b0c89cf71b9b847f9c424255255c40c59877
[1] https://www.mageia.org/mirrorlist/?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1
[2] http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os
Comment 21 Samuel Verschelde 2016-10-10 22:26:38 CEST
Could we have a status about this change? Do we still target Mageia 6 or should we target Mageia 7 now?
Comment 22 Rémi Verschelde 2016-10-12 13:54:41 CEST
It's part of an accepted and implemented feature for Mageia 6, so it would be really nice to get this implemented. AFAIK though, it still requires:
 - infra hardware upgrade, as we have many other infra-related issues that are blocked by having too little disk space
 - sysadmin time to actually implement mirrorbrain

So I don't know if it will be realistically done before Mageia 6 is released, but I'd keep it on the milestone until it's certain it won't.
Comment 23 Neal Gompa 2016-12-12 22:03:29 CET
@Thomas,

Any progress on this? I recall you mentioned you were going to work on it last week...
Comment 24 Doug Laidlaw 2017-01-17 14:10:16 CET
*** Bug 20130 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.