Bug 30089 - Please stop letting distro.ibiblio.org be a 1st tier mirror, and remove it from the mirrorlist.
Summary: Please stop letting distro.ibiblio.org be a 1st tier mirror, and remove it fr...
Status: REOPENED
Alias: None
Product: Websites
Classification: Unclassified
Component: Other (show other bugs)
Version: trunk
Hardware: All Linux
Priority: High major
Target Milestone: ---
Assignee: Sysadmin Team
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-02-23 22:31 CET by Pierre Fortin
Modified: 2024-01-01 21:09 CET (History)
10 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments

Description Pierre Fortin 2022-02-23 22:31:26 CET
Description of problem: Cauldron install fails due to missing compssUsers.pl


Version-Release number of selected component (if applicable):

Cauldron appeared to successfully download from distro.ibiblio.org  However, this one file downloaded an html permission denied message instead.

http://distro.ibiblio.org/mageia/distrib/cauldron/x86_64/media/media_info/compssUsers.pl

Downloaded compssUsers.pl from another site and install works now.

How reproducible: Always 


Steps to Reproduce:
1. mirror files from distro.ibiblio.org
2. 
3.
Pierre Fortin 2022-02-23 22:43:13 CET

Summary: Cauldron install failed due to missing => Cauldron install failed due to missing compssUsers.pl

Comment 1 sturmvogel 2022-02-24 07:54:39 CET
This mirror is completely out of sync. Last update was 2021.
Comment 2 sturmvogel 2022-02-24 07:57:23 CET
See also here
https://mirrors.mageia.org/status
Comment 3 Thomas Andrews 2022-02-24 14:30:14 CET
Unfortunately, Mageia has no control over the operation of individual mirrors. That is the domain of the owners of each particular mirror.

However, this bug presents us with an opportunity. I believe the focus of this bug should be changed to request that a link to https://mirrors.mageia.org/status be added to the top of the http://mirrors.mageia.org/ page, as sturmvogel proposed on the Discuss mailing list.

In my opinion, it's long overdue.

CC: (none) => andrewsfarm

Filip Komar 2022-02-25 22:48:00 CET

CC: (none) => filip.komar

Comment 4 Jybz 2022-04-05 13:03:28 CEST
Mageia has no control on mirrors, it's true. But Mageia has control on official mirror list are can add/remove any of them.
An administrator is required for such task.

CC: (none) => j.biernacki+mga

Comment 5 Thomas Andrews 2022-04-05 15:18:30 CEST
Yes, it has been done before. See Bug 23588. 

But, it should not be done lightly. Last I knew, ibiblio was a Tier 1 mirror, and has been with Mageia from the beginning. That should not be forgotten. 

I think one of the sysadmins needs to reach out to the ibiblio site admin, if it hasn't already been done, to learn what's going on before summarily dropping it from the list.
Comment 6 Dave Hodgins 2022-04-05 19:48:39 CEST
Re-assigning to the sysadmin team

Assignee: atelier-bugs => sysadmin-bugs
CC: (none) => davidwhodgins

Comment 7 Dave Hodgins 2022-04-05 19:51:40 CEST
Also note that it's only a partial mirror, (no debug or testing repos).
Dave Hodgins 2022-04-05 19:53:37 CEST

Summary: Cauldron install failed due to missing compssUsers.pl => distro.ibiblio.org is only a partial mirror, and stopped syncing 2021-11-30

Comment 8 Pierre Fortin 2022-05-19 17:56:15 CEST
Closing

Status: NEW => RESOLVED
Resolution: (none) => OLD

Comment 9 Dave Hodgins 2022-05-19 18:16:47 CEST
Still needs to be dealt with. It's only a partial mirror and has not
synced updates since 2021-Feb-25 02:16:29

That's 15 months behind currently. It should not be included in mirrorlist,
as any new user whom that mirror has been selected will not see any new updates.

Status: RESOLVED => REOPENED
Resolution: OLD => (none)

Comment 10 Thomas Andrews 2022-12-08 23:14:23 CET
During testing of the first round of the Mageia 9 beta1 isos, after an install when I was setting up a specific mirror, I saw that the ibiblio mirror is still on the list of US mirrors. It's probably also somewhere on the list of mirrors on the netinstall isos, and I wouldn't be surprised if some users are getting connected to it through $MIRRORLIST.

We really need to remove this mirror from those two lists, and from $MIRRORLIST. Users are going to have no end of troubles if that doesn't happen.
Morgan Leijström 2022-12-09 16:54:34 CET

Severity: normal => major
Priority: Normal => release_blocker
CC: (none) => fri

Comment 11 Thomas Andrews 2023-03-02 17:49:41 CET
So, we have released the Mageia 9 beta1 isos to the public. Each of those isos contains a provision somewhere along the line where the user has the chance to set up a specific mirror if he or she so chooses, or they can use MIRRORLIST. On each of the iso installs where I chose a specific mirror, ibiblio was still there on the list of possibles, sometimes presented at the top of the list. And after checking, I see that mirror is still many months out of date.

I'm not going to re-iterate why this is very bad for Mageia. I was originally willing to give them a chance to correct the problem, but it has now gone on for far too long. It should be obvious to anyone that the ibiblio mirror needs to be removed from the list.

But unfortunately, that's not enough. The map at https://mirrors.mageia.org/report indicates that there are three additional mirrors that use the ibiblio mirror as a source. They are, of course, equally out of date, and equally useless.

At the very least, beyond removing ibiblio we need to inform the admins of those three mirrors that they need to switch to a different source, and why, and if they don't change, remove them from the list as well.

And as far as I'm concerned, this MUST be done before the final release of Mageia 9. The longer we wait, the worse the consequences to Mageia's reputation will be.
Comment 12 Thomas Andrews 2023-05-05 00:18:50 CEST
I see that quietly, the ibiblio mirror has been removed from the list of Cauldron mirrors on the beta2 isos, and the list of specific mirrors in MCC after the install. Thank you.

Reducing the priority from Mageia 9 release blocker, but leaving it open for Mageia 8 users that might fall into this trap to find.

Priority: release_blocker => Normal

Comment 13 Morgan Leijström 2023-11-08 15:29:58 CET
Trying today in mga9, in MCC, adding a specific mirror:
It fetch list of mirrors and in that list users can select this defunct ibiblio.

Priority: Normal => High

Comment 14 Frédéric "LpSolit" Buclin 2023-11-08 17:41:39 CET
Why not add code in MCC so that it can check how up-to-date a mirror is, and hide it from the list if it's older than X days? This way, the list would be dynamic, and if a mirror is out-of-sync only momentarily, it doesn't require a sysadmin to remove it from the list.

To avoid the installer and MCC having to query each mirror to know which ones are up-to-date, this information could be stored in one single place somewhere on the mageia.org server, which the installer and MCC would download first, before listing mirrors to users. This way, it would also be possible to add new mirrors after the ISO have been published.
Comment 15 katnatek 2023-11-08 19:40:06 CET
I think we must add as require a current mail to active sysadmin in https://wiki.mageia.org/en/Mirrors_policy

As alternative to comment#14 I suggest make the mirrolist api produce a list just with trusted mirrors
Comment 16 Marja Van Waes 2023-11-25 12:48:47 CET
There have been several attempts by several Mageians to contact ibiblio about their seriously outdated mirror, but nothing helped.

https://ml.mageia.org/l/arc/qa-discuss/2023-11/msg00017.html


It hasn't synced at all since September 17.

Users who install Mageia still get it as mirror.

@ Dave Hodgins,

in https://bugs.mageia.org/show_bug.cgi?id=23588#c11 you said:

> I finally figured out how to remove a mirror from the list and have now
> removed
> mirrors-usa.go-parts.com

How did you do that?

CC: (none) => manuel.mageia, marja11
Summary: distro.ibiblio.org is only a partial mirror, and stopped syncing 2021-11-30 => Please stop letting distro.ibiblio.org be a 1st tier mirror, and remove it from the mirrorlist.

Comment 17 Marja Van Waes 2023-11-25 13:59:40 CET
(In reply to Marja Van Waes from comment #16)

> 
> @ Dave Hodgins,
> 
> in https://bugs.mageia.org/show_bug.cgi?id=23588#c11 you said:
> 
> > I finally figured out how to remove a mirror from the list and have now
> > removed
> > mirrors-usa.go-parts.com
> 
> How did you do that?

Found it, but I can't blank fields in https://mirrors.mageia.org/mirrors/distro.ibiblio.org like you did for mirrors-usa.go-parts.com. :-(
Comment 18 Manuel Hiebel 2023-11-25 16:20:46 CET
(In reply to Marja Van Waes from comment #16)
> There have been several attempts by several Mageians to contact ibiblio
> about their seriously outdated mirror, but nothing helped.
> 
> https://ml.mageia.org/l/arc/qa-discuss/2023-11/msg00017.html
> 
> 
> It hasn't synced at all since September 17.
> 
> Users who install Mageia still get it as mirror.
> 
> @ Dave Hodgins,
> 
> in https://bugs.mageia.org/show_bug.cgi?id=23588#c11 you said:
> 
> > I finally figured out how to remove a mirror from the list and have now
> > removed
> > mirrors-usa.go-parts.com
> 
> How did you do that?

Marja you added me, but I cannot do something.
I guess @sysadmin needs to do it directly in the database
Comment 19 Marja Van Waes 2023-11-25 17:08:52 CET
(In reply to Manuel Hiebel from comment #18)

> 
> Marja you added me, but I cannot do something.
> I guess @sysadmin needs to do it directly in the database


I had hoped you'd have some power ;-)

In the past I had the power to edit

puppet/deployment/main_mirror/files/mirror/mirror.readme

So I tried to remove distro.ibiblio.org from the list of mirrors that other mirrors are advised to sync with, but I lost the power :-(
Comment 20 Dave Hodgins 2023-11-25 17:19:11 CET
After a spammer added a bunch of mirrors with the "mirror links" going to
spam web sites, the sysadmins tightened down the options on the mirror update
page.

Only the sysadmins can do it now, and my understanding is that it must
be done by running sql to remove the entry from the database.

When I did it, there was a not obvious way in the website that I found. That's
been removed. I don't remember now exactly how I got to the screen where
things like the latitude/longtitude could be changed. By updating all of the
fields to be blank, it removed the entry.
katnatek 2023-12-13 22:22:34 CET

See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=32607

katnatek 2023-12-13 22:26:00 CET

See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=32620

Comment 21 katnatek 2023-12-14 02:20:49 CET
Consider also https://mirror.rabisu.com/mageia/ as nasty mirrors
Comment 22 Guillaume Bedot 2023-12-14 19:03:14 CET
(In reply to Dave Hodgins from comment #20)
> After a spammer added a bunch of mirrors with the "mirror links" going to
> spam web sites, the sysadmins tightened down the options on the mirror update
> page.

Maybe not enough, if I go to https://mirrors.mageia.org/mirrors/mirrors-usa.go-parts.com, there is an "update information" button.

But there is no button for distro.ibiblio.org indeed.

CC: (none) => geex+mageia

Comment 23 Guillaume Bedot 2023-12-14 19:19:40 CET
This one is also very out-of-date :
http://mirror.isoc.org.il/pub/mageia/ 09-Apr-2022
Comment 24 Thomas Andrews 2023-12-14 20:18:19 CET
(In reply to katnatek from comment #21)
> Consider also https://mirror.rabisu.com/mageia/ as nasty mirrors

According to https://mirrors.mageia.org/report that one syncs from ibiblio.
Comment 25 Thomas Andrews 2023-12-14 20:22:37 CET
(In reply to Guillaume Bedot from comment #23)
> This one is also very out-of-date :
> http://mirror.isoc.org.il/pub/mageia/ 09-Apr-2022

That one is in Israel. At the moment, they may have other things to think about than syncing with Mageia.
Comment 26 Morgan Leijström 2023-12-14 21:10:00 CET
It have been stuck *much* longer then last local war period...
Comment 27 katnatek 2023-12-15 00:41:07 CET
(In reply to Thomas Andrews from comment #24)
> (In reply to katnatek from comment #21)
> > Consider also https://mirror.rabisu.com/mageia/ as nasty mirrors
> 
> According to https://mirrors.mageia.org/report that one syncs from ibiblio.

Yes, but I did send a mail about the situation of ibiblio  to rabisu and not get any answer and not look they took any action about that
Comment 28 Pascal Terjan 2023-12-31 19:26:50 CET
I looked at our logs and it seems the problem is that ibiblio is syncing from a different machine than the one we have in the config:

Dec 28 10:36:03 duvel.mageia.org rsyncd[4040100]: connect from jobbot0.ibiblio.org (152.19.134.28)
Dec 28 10:36:03 duvel.mageia.org rsyncd[4040100]: rsync denied on module mageia from jobbot0.ibiblio.org (152.19.134.28)
Dec 28 14:36:01 duvel.mageia.org rsyncd[750662]: connect from jobbot0.ibiblio.org (152.19.134.28)
Dec 28 14:36:01 duvel.mageia.org rsyncd[750662]: rsync denied on module mageia from jobbot0.ibiblio.org (152.19.134.28)


Our config only allows the IP of jobbot1.ibiblio.org, I added jobbot0.ibiblio.org in https://gitweb.mageia.org/infrastructure/puppet/commit/?id=45107263ba400154ba3b78d563e63780b5e7bfc3 and hopefully it will be enough for it to get up to date again

CC: (none) => pterjan

Comment 29 katnatek 2024-01-01 01:15:55 CET
(In reply to Pascal Terjan from comment #28)
> I looked at our logs and it seems the problem is that ibiblio is syncing
> from a different machine than the one we have in the config:
> 
> Dec 28 10:36:03 duvel.mageia.org rsyncd[4040100]: connect from
> jobbot0.ibiblio.org (152.19.134.28)
> Dec 28 10:36:03 duvel.mageia.org rsyncd[4040100]: rsync denied on module
> mageia from jobbot0.ibiblio.org (152.19.134.28)
> Dec 28 14:36:01 duvel.mageia.org rsyncd[750662]: connect from
> jobbot0.ibiblio.org (152.19.134.28)
> Dec 28 14:36:01 duvel.mageia.org rsyncd[750662]: rsync denied on module
> mageia from jobbot0.ibiblio.org (152.19.134.28)
> 
> 
> Our config only allows the IP of jobbot1.ibiblio.org, I added
> jobbot0.ibiblio.org in
> https://gitweb.mageia.org/infrastructure/puppet/commit/
> ?id=45107263ba400154ba3b78d563e63780b5e7bfc3 and hopefully it will be enough
> for it to get up to date again

Start to work, green squares are there, some still are red, but soon this will be solved.
Comment 30 Marja Van Waes 2024-01-01 01:18:04 CET
(In reply to Pascal Terjan from comment #28)
> I looked at our logs and it seems the problem is that ibiblio is syncing
> from a different machine than the one we have in the config:
> 
<snip>
> 
> Our config only allows the IP of jobbot1.ibiblio.org, I added
> jobbot0.ibiblio.org in
> https://gitweb.mageia.org/infrastructure/puppet/commit/
> ?id=45107263ba400154ba3b78d563e63780b5e7bfc3 and hopefully it will be enough
> for it to get up to date again

Thanks! It looks good, Mageia 8 on that mirror is fully up-to-date now, so is Mageia 9 armv7h and aarch64. https://mirrors.mageia.org/status

I assume you can revert https://gitweb.mageia.org/infrastructure/puppet/commit/?id=03d63ba9e247a3482469a9d26647a03b218f36f4 before January 2 (UTC) :-)
Comment 31 katnatek 2024-01-01 17:42:44 CET
Now ibiblio and rabisu are in sync , but I note some things in mirrors status

Lot of Mageia 9 i586 marked in red with 2023-12-30 09:24, I think they sync with http://ftp.klid.dk/ftp/mageia/ that is in sync process

http://mirror.isoc.org.il/pub/mageia/ 2022-04-08 22:01 for mageia 8 x86_64 and aarch64
Comment 32 r howard 2024-01-01 21:07:15 CET
(In reply to katnatek from comment #31)
> Now ibiblio and rabisu are in sync , but I note some things in mirrors status
> 
> Lot of Mageia 9 i586 marked in red with 2023-12-30 09:24, I think they sync
> with http://ftp.klid.dk/ftp/mageia/ that is in sync process
> 
> http://mirror.isoc.org.il/pub/mageia/ 2022-04-08 22:01 for mageia 8 x86_64
> and aarch64

The email address of the mirror admin at mirror.isoc.org.il is mirrormaster@isoc.org.il if someone wants to contact them regarding their mageia directories being out of sync.

CC: (none) => rihoward1

Comment 33 Thomas Andrews 2024-01-01 21:09:41 CET
Ran a couple of acid tests on ibiblio:

I had a MGA8 Plasma install left on my HP pavilion 15. I upgraded that this afternoon using urpmi, then I used drakrpm-edit-media through MCC to switch from the resulting MIRRORLIST to the ibiblio mirror. There were error messages about the missing testing and debug repos, but that was to be expected since this is only a partial mirror. Then I used it to install Frozen Bubble, which drew in a number of dependencies. That was successful.

Then I tried a new install on some unused hard drive space of another system, using the nonfree netinstall iso. That failed, with a message that the installer program wasn't available from the ibiblio mirror. 

@Pascal: Could there be yet another of our config settings that needs to be changed so that ibiblio gets the installer?

The last test was with the CI, where I specified the ibiblio https mirror as supplemental media. That install was also successful, eventually booting to a fully updated working desktop.

@David: "note that it's only a partial mirror, (no debug or testing repos)" When you think about it a bit, that's not so bad, considering that most of our users don't do anything with those repos anyway. However, it is a bit disconcerting to see those error messages when it is first set up, especially on a Tier 1 mirror.

Note You need to log in before you can comment on or make changes to this bug.