(In reply to Marc Krämer from comment #0)
> Main mirror
> distrib-coffee.ipsl.jussieu.fr
> 
> doesn't fetch updated packages as e.g. 6/core firefox-52.5.2-1.mga6
> 
> kernel.org mirror is ok

It's a disaster http://mirrors.mageia.org/status

We really need MirrorBrain

I didn't check whether all "red" mirrors sync with distrib-coffee, but know *21* of the mirrors do, according to http://mirrors.mageia.org/

@ Nanar

Even after you kick distrib-coffee, given the little bandwidth of it, should we maybe send a mail to the mirrors-announce ml requesting to sync with a different mirror than distrib-coffee? WDYT?

Thanks for distrib-coffee, btw, and for Sophie :-)

CC: (none) => marja11, nanardon
Summary: On of the main mirrors does not sync => distrib-coffee doesn't sync, over 20 other mirrors sync with it.... should we write mirrors-announce?

Which mirror with a reasonable bandwidth does work

CC: (none) => bjarne.thomsen

at least kernel.org. But many mirrors use distrib-coffee as main mirror to sync

mirrorservice.org at Kent University in UK is a very fast first tier mirror, however it does not sync quite as often as d-c normally does.

How do we get a solution here?
Our main problem is the distribution between the mirrors, they are organized hierarchically without any failover etc. many of the second tier mirrors use distrib-coffee, so we have kind of single point of failure. I think we should have a bunch of first tier mirrors and the second tier mirrors should not stick to one of them. Maybe this can be solved by an rotating DNS entry or another redirect technique. For http-mirrors this is easy via http redirect.

If we use mirrorbrain, as far as I've read, this does only solve the problem for the user, but not for our mirror infrastructure, where 90% of the mirrors are out of sync.

Sorry for not having told so in this bug report:

Nanar wrote me yesterday to tell that distrib-coffee is down because of a hardware problem. They are working on replacing the failing component and hope the mirror will be back up next week. 
All of the old hardware will be replaced, soon. He apologizes for the inconvenience.

I really think one of our sysadmins should notify the maintainers of other mirrors via https://ml.mageia.org/l/info/mirrors-announce and encourage to sync with a different tier 1 mirror while distrib-coffee is down.

CC'ing tmb

CC: (none) => tmb
Summary: distrib-coffee doesn't sync, over 20 other mirrors sync with it.... should we write mirrors-announce? => distrib-coffee doesn't sync, over 20 other mirrors sync with it.... Please notify mirrors-announce ml

Announcement sent:

Hi,

Tier 1 mirror distrib-coffee.ipsl.jussieu.fr is down due to hardware breakage and will be available again next week at the earliest.

To avoid long breakages for our end-users, we request that mirrors that have been mirroring from distrib-coffee switch to another Tier 1.

Currentlly that is:

    o rsync://mageia.c3sl.ufpr.br/mageia/
      located in Curitiba (Brasil)
    o rsync://mirrors.kernel.org/mirrors/mageia/
      located in USA and Europe
    o rsync://ftp.acc.umu.se/mirror/mageia/
      located in Umea (Sweden)
    o rsync://mirror.math.princeton.edu/pub/mageia/
      located in Princeton (USA)
    o rsync://distro.ibiblio.org/mageia/
      located in Durham (USA)

--
Best regards

Mageia.org sysadm team

Thanks for the notification, Thomas.

Keeping this report open, even if there is nothing more we can do on our side, so that users searching our Bugzilla about the issue will more easily find it.

CC'ing isadora and doktor5000 for the forums, and atelier team  for the blog, because it might be good to add a message about the issue in the forums and/or the blog.

CC: (none) => atelier-bugs, doktor5000, isis2000
Summary: distrib-coffee doesn't sync, over 20 other mirrors sync with it.... Please notify mirrors-announce ml => distrib-coffee and all mirrors syncing with are broken.

Nanar just wrote that new storage is installed, but empty. 

distrib-coffee is syncing now, but it'll take a while before all files are back.

It would have been better for it to sync it before putting it back online. I assume all dependent mirrors have cleaned the contents of all distribs.

On http://ftp-stud.hs-esslingen.de/pub/Mirrors/Mageia/distrib/
we currently have only version 1.

This is temporary, but still not very nice...

Just for the record.

(In reply to Marc Krämer from comment #10)
> It would have been better for it to sync it before putting it back online. I
> assume all dependent mirrors have cleaned the contents of all distribs.

Unless they did something smart to keep that from happening!

> 
> On http://ftp-stud.hs-esslingen.de/pub/Mirrors/Mageia/distrib/
> we currently have only version 1.

and now 1 and 5 :-(

Locally, all media for EOL Mageia releases and also all stable ISOs are in an exclude list, so they won't get wiped. I'll add all stable media (& SRPMS) "release" directories to the exclude list, too. That is still not smart enough, though.

Is there something similar to "rsync --list-only" that recursively shows all remote directories? If so, it could be used to compare the remote directory tree with the local one and, if they don't match, to stop the rsync script before really syncing.

My script does a check of the files count in a dummy run and switches to another mirror if the files count has dropped significantly.
If none of the mirrors checked look sane it does not sync.
This avoids my local mirror getting wiped in these situations.
It's not foolproof as there are legitimate reasons for sudden drops in the files counts, so it does need some manual intervention occasionally.

CC: (none) => zen25000

(In reply to Barry Jackson from comment #12)
> My script does a check of the files count in a dummy run and switches to
> another mirror if the files count has dropped significantly.

A link to the script in this bug repport  would be helpfull.

Perhaps you should put in on Github.

CC: (none) => cooker

(In reply to Johnny A. Solbu from comment #13)
> (In reply to Barry Jackson from comment #12)
> > My script does a check of the files count in a dummy run and switches to
> > another mirror if the files count has dropped significantly.

Thanks of reminding us of your script
> 
> A link to the script in this bug repport  would be helpfull.

Thanks, Barry, for giving the link on IRC, so I won't need to search my mails to see whether I can find it ;-)

http://mtf.duckdns.org/pub/linux/barjac/soft/cronsync

> 
> Perhaps you should put in on Github.

+1

(In reply to Marja van Waes from comment #14)
> (In reply to Johnny A. Solbu from comment #13)
> > (In reply to Barry Jackson from comment #12)
> > > My script does a check of the files count in a dummy run and switches to
> > > another mirror if the files count has dropped significantly.
> 
> Thanks of reminding us of your script
> > 
> > A link to the script in this bug repport  would be helpfull.
> 
> Thanks, Barry, for giving the link on IRC, so I won't need to search my
> mails to see whether I can find it ;-)
> 
> http://mtf.duckdns.org/pub/linux/barjac/soft/cronsync
> 
> > 
> > Perhaps you should put in on Github.
> 
> +1

https://github.com/barjac/cronsync

That was pulled straight off my server as it is currently in use.

This has been fixed since quite a while now, closing.

Resolution: (none) => FIXED
Status: NEW => RESOLVED