Bug 26967 - Mirroring going stale?
Summary: Mirroring going stale?
Status: RESOLVED WORKSFORME
Alias: None
Product: Infrastructure
Classification: Unclassified
Component: Others (show other bugs)
Version: unspecified
Hardware: All Linux
Priority: High major
Target Milestone: ---
Assignee: Sysadmin Team
QA Contact:
URL: http://mirrors.mageia.org/status
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-17 12:13 CEST by Morgan Leijström
Modified: 2020-10-14 21:11 CEST (History)
3 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments
Most mga7 mirrors are 31 h behind (342.18 KB, image/png)
2020-10-11 23:02 CEST, Morgan Leijström
Details

Description Morgan Leijström 2020-07-17 12:13:18 CEST
Almost all mirrors are behind, showing as orange and lightgreen.
There are as few green as there are red.
Morgan Leijström 2020-07-17 12:13:37 CEST

Priority: Normal => High

Olav Vitters 2020-07-23 20:50:13 CEST

CC: (none) => olav

Comment 1 Morgan Leijström 2020-07-24 10:30:05 CEST
Seems OK to day and mostly yesterday too - only Cauldron lags a bit but that is natural.

Two and three days ago however it was not good: at 2020-07-20 11:40 more than half of the mga7 mirrors was red, stamped 2020-07-18 06:15. All of the rest was light green, except distrib-cofee, and ftp.free which were green.
Comment 2 Aurelien Oudelet 2020-09-19 18:08:52 CEST
Hi,
This is High priority bug for a good reason.

Making Mageia even better than ever is best direction.
In order to do right thing, this bug should be examined and fixed as soon as possible.

Packagers, please make the status to Assigned when you are working on this.
Feel free to reassign the bug if bad-triaged. Also, if bug is old, please close it.

On October 1st 2020, we will drop priority to normal.
Comment 3 Morgan Leijström 2020-10-11 23:02:49 CEST
Created attachment 11928 [details]
Most mga7 mirrors are 31 h behind
Comment 4 Dave Hodgins 2020-10-12 00:10:24 CEST
Looks like http://mirrors.mageia.org/mirrors/ftp.acc.umu.se has stopped syncing
from wherever it syncs from, and the others sync directly or indirectly from it.

Anyone know who to contact for that mirror?

CC: (none) => davidwhodgins

Comment 5 Morgan Leijström 2020-10-12 00:12:58 CEST
Currently it is running, just fetched a testing update from umu.se.

But are 90% of mirrors syncing from umu?! (north end of Sweden)
Comment 6 Morgan Leijström 2020-10-12 00:20:31 CEST
According to map http://mirrors.mageia.org/report
umu do not sync from anything(?), and only feed 5 mirrors.
Comment 7 Dave Hodgins 2020-10-12 00:56:31 CEST
You're right. There are too many with that same time, including
mageia.c3sl.ufpr.br, a tier 1 mirror and ftp.nluug.nl which syncs from
mirrors.kernel.org, according to http://mirrors.mageia.org/, and kernel.org
is up-to-date.

I'm guessing now that at least some of the mirrors have changed which mirror
they are syncing from, but have not updated mirrors.mageia.org.

Makes figuring out where the problem is that much harder.

Would be nice if http://mirrors.mageia.org/status showed tier 1 mirrors first.
Comment 8 Dave Hodgins 2020-10-12 06:58:20 CEST
It's up-to-date now.
Comment 9 Dave Hodgins 2020-10-12 06:59:17 CEST
Closing
Comment 10 Morgan Leijström 2020-10-12 08:04:33 CEST
Well, it happens rather frequently - examples see comment 0 and comment 2.
The fact it works for the moment i think do not mean the problem is solved.

Do we have a list of contacts for all mirrors?
We should ping them to note the problem and update/fill in what source they use.
Comment 11 Morgan Leijström 2020-10-12 14:01:24 CEST
Still more than 2/3 are orange, including the ones that sync from umu accordign to map.  Umu.se is still green


Most "oranges" are now stamped 2020-10-11 20:18 which is interesting because yesterday the the page said last checked 2020-10-11 22:40 with orange times 2020-10-10 15:43.

Why did the check at 22:40 not pick up the stamp from 20:18, almost two and a half hour.

I wonder if it is the checking or mirroring that have problems...?
Comment 12 Dave Hodgins 2020-10-14 02:21:50 CEST
Mostly, the checking. The timestamp on a mirror only gets updated if it found
a change the last time it synced.

IIRC, http://mirrors.mageia.org/status checks the mirrors every 20 minutes.

If a package change is uploaded to rsync.mageia.org, then every tier 1 mirror
is instantly out of date until they next sync.

If it's been at least 12 hours since a package was built, and a new package
finishes being built just before the status check, then every tier 1 mirror
is instantly more than 12 hours old.

Similarly, if it's been 2 days since a package was built, which happens at
times on the stable release such as Mageia 7, then as soon as a new update
gets uploaded to rsync.mageia.org, all tier 1 mirrors will be shown as red,
old or broken. That happens at times on cauldron too, usually after version
freeze.

The frequency of checking for updates on mirrors that are not tier 1 mirrors
is up to the mirror admin, so they are only up-to-date if rsync.mageia.org
has not been updated for the combined time of the sync of each tier above
them.

The X mirrors no longer support http (which http://mirrors.mageia.org/status
uses) as they've chosen to only support https. The urpmi tools do handle https,
it's just the status report that doesn't.

The status tool makes it easier to spot when a mirror has stopped syncing, but
it's info has to be interpreted carefully.
Comment 13 Morgan Leijström 2020-10-14 09:24:23 CEST
Ah.

Good explanation.

Do you think a text like that could be put at the status page?
Comment 14 Dave Hodgins 2020-10-14 10:50:22 CEST
I think that would be a bit much, on an already crowded page. I don't think
there are that many people who regularly check it that haven't figured out
how it works already.
Comment 15 Dave Hodgins 2020-10-14 10:53:07 CEST
Also see https://pkgsubmit.mageia.org/ to know when a packages were last
added to rsync.mageia.org.
Comment 16 Morgan Leijström 2020-10-14 12:18:58 CEST
Well a link [Explanations] could at the top there, opening a simple text page...
Comment 17 Dave Hodgins 2020-10-14 20:55:52 CEST
I've passed on a request to our atelier group asking them to add the
description, either on that page or on a new page, as they choose.
Comment 18 Morgan Leijström 2020-10-14 21:11:56 CEST
Thumbs up :)

Some servers are very long behind, but that have been known for a while.

I know people have been pinged about it. Hard to reach mirror admins.

Closing.  Reopening if recurring with real broad mirror delay.

Status: NEW => RESOLVED
Resolution: (none) => WORKSFORME


Note You need to log in before you can comment on or make changes to this bug.