Bug 22190 - distrib-coffee and all mirrors syncing with it are broken.
Summary: distrib-coffee and all mirrors syncing with it are broken.
Status: RESOLVED FIXED
Alias: None
Product: Infrastructure
Classification: Unclassified
Component: Others (show other bugs)
Version: unspecified
Hardware: All Linux
Priority: Normal critical
Target Milestone: ---
Assignee: Sysadmin Team
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-12-13 13:45 CET by Marc Krämer
Modified: 2018-01-24 09:13 CET (History)
10 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments

Description Marc Krämer 2017-12-13 13:45:11 CET
Main mirror
distrib-coffee.ipsl.jussieu.fr

doesn't fetch updated packages as e.g. 6/core firefox-52.5.2-1.mga6

kernel.org mirror is ok
Comment 1 Marja Van Waes 2017-12-14 10:13:56 CET
(In reply to Marc Krämer from comment #0)
> Main mirror
> distrib-coffee.ipsl.jussieu.fr
> 
> doesn't fetch updated packages as e.g. 6/core firefox-52.5.2-1.mga6
> 
> kernel.org mirror is ok

It's a disaster http://mirrors.mageia.org/status

We really need MirrorBrain

I didn't check whether all "red" mirrors sync with distrib-coffee, but know *21* of the mirrors do, according to http://mirrors.mageia.org/

@ Nanar

Even after you kick distrib-coffee, given the little bandwidth of it, should we maybe send a mail to the mirrors-announce ml requesting to sync with a different mirror than distrib-coffee? WDYT?

Thanks for distrib-coffee, btw, and for Sophie :-)

CC: (none) => marja11, nanardon
Summary: On of the main mirrors does not sync => distrib-coffee doesn't sync, over 20 other mirrors sync with it.... should we write mirrors-announce?

Comment 2 Bjarne Thomsen 2017-12-14 11:54:23 CET
Which mirror with a reasonable bandwidth does work

CC: (none) => bjarne.thomsen

Comment 3 Marc Krämer 2017-12-14 14:09:37 CET
at least kernel.org. But many mirrors use distrib-coffee as main mirror to sync
Comment 4 Barry Jackson 2017-12-14 14:24:20 CET
mirrorservice.org at Kent University in UK is a very fast first tier mirror, however it does not sync quite as often as d-c normally does.
Comment 5 Marc Krämer 2017-12-16 14:59:17 CET
How do we get a solution here?
Our main problem is the distribution between the mirrors, they are organized hierarchically without any failover etc. many of the second tier mirrors use distrib-coffee, so we have kind of single point of failure. I think we should have a bunch of first tier mirrors and the second tier mirrors should not stick to one of them. Maybe this can be solved by an rotating DNS entry or another redirect technique. For http-mirrors this is easy via http redirect.

If we use mirrorbrain, as far as I've read, this does only solve the problem for the user, but not for our mirror infrastructure, where 90% of the mirrors are out of sync.
Comment 6 Marja Van Waes 2017-12-16 17:13:17 CET
Sorry for not having told so in this bug report:

Nanar wrote me yesterday to tell that distrib-coffee is down because of a hardware problem. They are working on replacing the failing component and hope the mirror will be back up next week. 
All of the old hardware will be replaced, soon. He apologizes for the inconvenience.

I really think one of our sysadmins should notify the maintainers of other mirrors via https://ml.mageia.org/l/info/mirrors-announce and encourage to sync with a different tier 1 mirror while distrib-coffee is down.

CC'ing tmb

CC: (none) => tmb
Summary: distrib-coffee doesn't sync, over 20 other mirrors sync with it.... should we write mirrors-announce? => distrib-coffee doesn't sync, over 20 other mirrors sync with it.... Please notify mirrors-announce ml

Comment 7 Thomas Backlund 2017-12-16 17:43:53 CET
Announcement sent:

Hi,

Tier 1 mirror distrib-coffee.ipsl.jussieu.fr is down due to hardware breakage and will be available again next week at the earliest.

To avoid long breakages for our end-users, we request that mirrors that have been mirroring from distrib-coffee switch to another Tier 1.

Currentlly that is:

    o rsync://mageia.c3sl.ufpr.br/mageia/
      located in Curitiba (Brasil)
    o rsync://mirrors.kernel.org/mirrors/mageia/
      located in USA and Europe
    o rsync://ftp.acc.umu.se/mirror/mageia/
      located in Umea (Sweden)
    o rsync://mirror.math.princeton.edu/pub/mageia/
      located in Princeton (USA)
    o rsync://distro.ibiblio.org/mageia/
      located in Durham (USA)

--
Best regards

Mageia.org sysadm team
Comment 8 Marja Van Waes 2017-12-16 18:16:33 CET
Thanks for the notification, Thomas.

Keeping this report open, even if there is nothing more we can do on our side, so that users searching our Bugzilla about the issue will more easily find it.

CC'ing isadora and doktor5000 for the forums, and atelier team  for the blog, because it might be good to add a message about the issue in the forums and/or the blog.

CC: (none) => atelier-bugs, doktor5000, isis2000
Summary: distrib-coffee doesn't sync, over 20 other mirrors sync with it.... Please notify mirrors-announce ml => distrib-coffee and all mirrors syncing with are broken.

Marja Van Waes 2017-12-16 18:16:46 CET

Summary: distrib-coffee and all mirrors syncing with are broken. => distrib-coffee and all mirrors syncing with it are broken.

Comment 9 Marja Van Waes 2017-12-16 18:32:17 CET
Nanar just wrote that new storage is installed, but empty. 

distrib-coffee is syncing now, but it'll take a while before all files are back.
Comment 10 Marc Krämer 2017-12-16 21:03:09 CET
It would have been better for it to sync it before putting it back online. I assume all dependent mirrors have cleaned the contents of all distribs.

On http://ftp-stud.hs-esslingen.de/pub/Mirrors/Mageia/distrib/
we currently have only version 1.

This is temporary, but still not very nice...

Just for the record.
Comment 11 Marja Van Waes 2017-12-18 08:39:31 CET
(In reply to Marc Krämer from comment #10)
> It would have been better for it to sync it before putting it back online. I
> assume all dependent mirrors have cleaned the contents of all distribs.

Unless they did something smart to keep that from happening!

> 
> On http://ftp-stud.hs-esslingen.de/pub/Mirrors/Mageia/distrib/
> we currently have only version 1.

and now 1 and 5 :-(

Locally, all media for EOL Mageia releases and also all stable ISOs are in an exclude list, so they won't get wiped. I'll add all stable media (& SRPMS) "release" directories to the exclude list, too. That is still not smart enough, though.

Is there something similar to "rsync --list-only" that recursively shows all remote directories? If so, it could be used to compare the remote directory tree with the local one and, if they don't match, to stop the rsync script before really syncing.
Comment 12 Barry Jackson 2017-12-18 10:59:22 CET
My script does a check of the files count in a dummy run and switches to another mirror if the files count has dropped significantly.
If none of the mirrors checked look sane it does not sync.
This avoids my local mirror getting wiped in these situations.
It's not foolproof as there are legitimate reasons for sudden drops in the files counts, so it does need some manual intervention occasionally.

CC: (none) => zen25000

Comment 13 Johnny A. Solbu 2017-12-18 22:07:59 CET
(In reply to Barry Jackson from comment #12)
> My script does a check of the files count in a dummy run and switches to
> another mirror if the files count has dropped significantly.

A link to the script in this bug repport  would be helpfull.

Perhaps you should put in on Github.

CC: (none) => cooker

Comment 14 Marja Van Waes 2017-12-19 07:57:29 CET
(In reply to Johnny A. Solbu from comment #13)
> (In reply to Barry Jackson from comment #12)
> > My script does a check of the files count in a dummy run and switches to
> > another mirror if the files count has dropped significantly.

Thanks of reminding us of your script
> 
> A link to the script in this bug repport  would be helpfull.

Thanks, Barry, for giving the link on IRC, so I won't need to search my mails to see whether I can find it ;-)

http://mtf.duckdns.org/pub/linux/barjac/soft/cronsync

> 
> Perhaps you should put in on Github.

+1
Comment 15 Barry Jackson 2017-12-19 14:13:25 CET
(In reply to Marja van Waes from comment #14)
> (In reply to Johnny A. Solbu from comment #13)
> > (In reply to Barry Jackson from comment #12)
> > > My script does a check of the files count in a dummy run and switches to
> > > another mirror if the files count has dropped significantly.
> 
> Thanks of reminding us of your script
> > 
> > A link to the script in this bug repport  would be helpfull.
> 
> Thanks, Barry, for giving the link on IRC, so I won't need to search my
> mails to see whether I can find it ;-)
> 
> http://mtf.duckdns.org/pub/linux/barjac/soft/cronsync
> 
> > 
> > Perhaps you should put in on Github.
> 
> +1

https://github.com/barjac/cronsync

That was pulled straight off my server as it is currently in use.
Comment 16 Marja Van Waes 2018-01-24 09:13:37 CET
This has been fixed since quite a while now, closing.

Status: NEW => RESOLVED
Resolution: (none) => FIXED


Note You need to log in before you can comment on or make changes to this bug.