DNF supports metalinks, which offer multiple levels of verification beyond what plain mirrorlists offer, so we should use those with MirrorBrain.

@ Neal

About moving bug 3166 from the "Depends on." field to "See Also:"

If you click on "See Also" you'll read about it:

________________________________________________________________________________

See Also
    This allows you to refer to bugs in other installations. You can enter a URL to a bug in the 'Add Bug URLs' field to note that that bug is related to this one. You can enter multiple URLs at once by separating them with a comma.

    You should normally use this field to refer to bugs in other installations. For bugs in this installation, it is better to use the Depends on and Blocks fields.

________________________________________________________________________________


Neal, if this bug doesn't depend on bug 3166, isn't this one then just another duplicate of it?

The reason I did not  close this bug as duplicate is, that I assumed something additional would need to be done to mirrorbrain for DNF.

However, if just having mirrorbrain is enough, then this report is indeed a duplicate of bug 3166

@Marja:

Depending on how you look at it, it could be a duplicate (it refers to deploying MirrorBrain), or considered a related bug (MirrorBrain needs to be configured to support RPM-MD metalink data for DNF).

I hesitate to indicate this is a duplicate of bug 3166 because this bug also is the indicator that MirrorBrain deployment+configuration is part of Mageia 6 release engineering requirements (per the feature review linked in the description).

What do you think?

(In reply to Neal Gompa from comment #4)
> @Marja:
> 
> (MirrorBrain needs to be
> configured to support RPM-MD metalink data for DNF).
> 

In that case, this is not a duplicate.

It is an additional step that needs to be done, so again setting this bug to depend on bug 3166

Of course, I do hope they'll get solved together, while MirrorBrain is configured for the very first time :-D

Depends on: (none) => 3166

@Marja:

Indeed, I hope so too.

As of now, we have RPM-MD repodata being automatically generated on every package upload in Cauldron.

We now need MirrorBrain deployed and configured for our mirror management and generation of metalinks for DNF repodata.

Based on what I see for the Rawhide Metalinks[0][1][2], we need to use v3 Metalinks that point to the repomd.xml files in /repodata folders across each repository.

Preliminarily, I think this is the URL structure we want to have for our metalinks:

https://mirrors.mageia.org/metalink?repo=<releasever>-<group>-<repo>[-<extra>]&arch=<arch>

releasever: The Mageia release version, set to "cauldron" for Cauldron.

group: The different types of repositories (core, nonfree, tainted)

repo: The repository where the packages live (release, updates, updates_testing, backports, backports_testing)

extra: This part would be added if we're retrieving either debuginfo packages (debug) or source packages (source).

arch: The architecture (i586, x86_64, armv5tl)
* Note that Fedora usually uses $basearch definitions here, and in our case, $basearch would map i586 to i386 and armv5tl to arm. I'm not sure we care here, since unlike Fedora, we provide the full i586 repository as an option to people using 64-bit Mageia rather than a multilib repository. We would probably hardcode the architecture in the metalink URL in repo configuration files for DNF so that both repositories would work.

Some example structures of the system indicated above:

* Cauldron Core Release, x86_64: https://mirrors.mageia.org/metalink?repo=cauldron-core-release&arch=x86_64

* Cauldron Core Updates Testing Debug, i586: https://mirrors.mageia.org/metalink?repo=cauldron-core-updates_testing-debug&arch=i586

* Cauldron Core Release Sources, x86_64: https://mirrors.mageia.org/metalink?repo=cauldron-core-release-source&arch=x86_64
** From what I've seen, the metalink URL requires the arch parameter, but will wind up not using it for this specific case.


[0]: https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
[1]: https://mirrors.fedoraproject.org/metalink?repo=rawhide-debug&arch=x86_64
[2]: https://mirrors.fedoraproject.org/metalink?repo=rawhide-source&arch=x86_64

For Mageia 6 release onward, we should structure the releases like so:

https://mirrors.mageia.org/metalink?repo=mageia-<releasever>-<group>-<repo>[-<extra>]&arch=<arch>

The only difference to the guide above is that releasever will indicate an actual Mageia release version (such as 5, 6, etc.).

Just a small update...

I now have a mirrorbrain running on dl.mageia.org
(but I haven't pushed it on public dns yet as I want to review my changes after
some sleep to verify I haven't missed anything...)

for now I only use pseudo tree on dl.mageia.org until I get extra disks
installed

CC: (none) => tmb

@Thomas:

Any progress on the MirrorBrain setup?

@Thomas:

Have you had a chance to work on this lately?

Ok, so pterjan goes to DC today, so we should hopefully get more disks installed so we get the needed space to finish up infra work.

@Thomas:

Was pterjan able to get more disks installed at the datacenter yesterday?

I found an interesting presentation[1] that talked about how the metalinks work in Fedora, which also indicates that the URL can be used to do further constraints.

For example, tacking on "&country=us,ca" would constrain it to mirrors in the US and Canada. "&country=global" returns all mirrors everywhere. "ip=<ip.address>" would permit sending the client IP address so that the mirror manager would use it for GeoIP work.

I don't know how much more work it would be to add, but it'd be cool to have support for later on (after we get the basic mirror system up!).

[1]: http://domsch.com/linux/fedora/mirrormanager-fudconf13.odp

@tmb:

As we talked about recently, you were working on a workaround or fixing null-rsync to work with our softlinks used in our distrib tree. Have you made any progress on this so that we can have the metalinks set up?

As a workaround I uploaded https://www.mageia.org/mirrorlist/?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1

but there are message that I didn't get locally and on my test server:
http://mageia-test.hiebel.eu/mirrorlist/?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1

no ideas yet, why they come

Not sure why, but it looks like it's generating URLs properly anyway. Perhaps you should add an error_reporting() function call[1] at the top of your script to silence them?

[1]: http://php.net/manual/en/function.error-reporting.php

(In reply to Manuel Hiebel from comment #16)
> As a workaround I uploaded
> https://www.mageia.org/mirrorlist/
> ?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1
> 
> but there are message that I didn't get locally and on my test server:
> http://mageia-test.hiebel.eu/mirrorlist/
> ?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1
> 
> no ideas yet, why they come

Thanks for a great idea. I'll try to dig in later today. I suspect some double include/require.

From both outputs I see two php server misconfigurations ;).

1: (our) production server should _never_ output any Notice (and Warnings). Outputting Errors is mostly undesired and possibly dangerous as it can increase attack surface.
We should file a bug for that. It would also solve some other issues like the one in bug #15292 #c3 and maybe more.

2: local dev server (in my opinion) should always output Notice, Warnings and Errors.


(In reply to Neal Gompa from comment #17)
> Perhaps you should add an error_reporting() function call[1] at the top of
> your script to silence them?

While I use that sometimes just to be sure I think that modifying php.ini is the proper place to fix that. Of course on dev server the error level should not be above Notice. In our code we manage that in langs basefile[0].

[0] http://gitweb.mageia.org/web/www/tree/langs.php

I fixed it in geoip direclty. Looks ok.

I just committed[0] some improvements to Manuels's mirrorlist[1] but forgot to reference this BR.

It now returns a list of ten mirrors like centos does[2] with priority: shuffled within the country, then continent and finally others.

To drop it's DL load it also excludes our source server (distrib-coffee.ipsl.jussieu.fr) now as a first choice in ISO and doc DL page and entirely from mirrorlist.


[0] http://gitweb.mageia.org/web/www/commit/?id=5552b0c89cf71b9b847f9c424255255c40c59877
[1] https://www.mageia.org/mirrorlist/?release=cauldron&arch=x86_64&section=core&repo=updates_testing&debug=1
[2] http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os

Could we have a status about this change? Do we still target Mageia 6 or should we target Mageia 7 now?

It's part of an accepted and implemented feature for Mageia 6, so it would be really nice to get this implemented. AFAIK though, it still requires:
 - infra hardware upgrade, as we have many other infra-related issues that are blocked by having too little disk space
 - sysadmin time to actually implement mirrorbrain

So I don't know if it will be realistically done before Mageia 6 is released, but I'd keep it on the milestone until it's certain it won't.

@Thomas,

Any progress on this? I recall you mentioned you were going to work on it last week...

*** Bug 20130 has been marked as a duplicate of this bug. ***

CC: (none) => laidlaws

*** Bug 22381 has been marked as a duplicate of this bug. ***

See Also: https://bugs.mageia.org/show_bug.cgi?id=22381 => (none)
CC: (none) => peixe80

I came to this bug via a duplicate.  Unsubscribing.

I've just obsoleted MirrorBrain in Cauldron as it hasn't been updated (or developed) since 2015. The alternative seems to be MirrorManager2, by Fedora infrastructure team. Either this bug needs to be updated or WONTFIX and a new bug.

Hi,
This is High priority bug for a good reason.

Making Mageia even better than ever is best direction.
In order to do right thing, this bug should be examined and fixed as soon as possible.

Packagers, please make the status to Assigned when you are working on this.
Feel free to reassign the bug if bad-triaged. Also, if bug is old, please close it.

On October 1st 2020, we will drop priority to normal.

(In reply to Olav Vitters from comment #27)
> I've just obsoleted MirrorBrain in Cauldron as it hasn't been updated (or
> developed) since 2015. The alternative seems to be MirrorManager2, by Fedora
> infrastructure team. Either this bug needs to be updated or WONTFIX and a
> new bug.

Closing as duplicate of bug 3166, which now has the summary:

use MirrorManager2 for download management (for better mirror choosing) and also configure it for DNF

*** This bug has been marked as a duplicate of bug 3166 ***

Resolution: (none) => DUPLICATE
Status: NEW => RESOLVED