Bug 14980 - installer removes third-party sources/media from urpmi.cfg when upgrading
Summary: installer removes third-party sources/media from urpmi.cfg when upgrading
Status: RESOLVED WONTFIX
Alias: None
Product: Mageia
Classification: Unclassified
Component: Installer (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal enhancement
Target Milestone: ---
Assignee: Thierry Vignaud
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 15013
  Show dependency treegraph
 
Reported: 2015-01-07 21:40 CET by David Walser
Modified: 2023-09-24 00:19 CEST (History)
8 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments
patch that saves a backup of urpmi.cfg (540 bytes, patch)
2015-02-07 17:41 CET, Angelo Naselli
Details | Diff
stage2 failing with error about urpmi.cfg.bak (974.24 KB, image/png)
2015-02-08 22:36 CET, Marja Van Waes
Details

Description David Walser 2015-01-07 21:40:10 CET
When upgrading using the installer, the sources in urpmi.cfg are removed, which for the previous distro version media makes sense, but if you have additional third-party sources in there (like the Google Chrome repo for instance), it shouldn't remove that.

Reproducible: 

Steps to Reproduce:
David Walser 2015-01-07 21:40:59 CET

CC: (none) => ennael1

claire robinson 2015-01-08 10:22:57 CET

CC: (none) => eeeemail

claire robinson 2015-01-12 09:35:31 CET

Blocks: (none) => 15013

Florian Hubold 2015-01-21 23:26:27 CET

CC: (none) => doktor5000

Comment 1 Rémi Verschelde 2015-02-03 21:20:02 CET
I wonder if that's something we can easily fix. Users might have all kind of setup in urpmi.cfg, with some or all of the Mageia media, sometimes with duplicates, etc.
How would we identify media that we want to remove, and those that we don't? Grepping for "distrib/$initialversion" in the URL might work, but it's a bit of a dirty hack.

CC: (none) => remi

Comment 2 David Walser 2015-02-03 21:30:02 CET
Dirty or not, it'd be better to delete too few than too many.
Comment 3 Florian Hubold 2015-02-04 22:20:55 CET
Why not simply set them to ignored instead of deleting them? 

Or at least inform the user that this is going to happen beforehand and that he needs to re-add (all/third party) media back (or re-enable them, depending on the outcome of this bug)?
Comment 4 David Walser 2015-02-04 23:05:43 CET
If any development efforts are going to be spent on this, let's get a proper fix
Angelo Naselli 2015-02-05 21:22:48 CET

CC: (none) => anaselli

Comment 5 Thierry Vignaud 2015-02-06 09:27:30 CET
(In reply to David Walser from comment #0)
> When upgrading using the installer, the sources in urpmi.cfg are removed,
> which for the previous distro version media makes sense,

Actually it makes sense for most repos.
Repos that are release agnostic are very rare (as in google chrome example as they ship most libs statically linked and only dynamically link to libs that are sure to be everywhere such as glibc)

(In reply to Florian Hubold from comment #3)
> Why not simply set them to ignored instead of deleting them? 

B/c we already have a huge list of media in urpm-edit-sources and it will just become unreadable

And adding a warning would just make the installer harder to understand to most end users...

AFAIC this is not a bug and should just be closed...
Comment 6 Thierry Vignaud 2015-02-06 09:31:41 CET
And let's do that

Status: NEW => RESOLVED
Resolution: (none) => WONTFIX

Comment 7 Angelo Naselli 2015-02-06 09:37:48 CET
Thierry i totally agree -as i said during the meeting-, 'though during last night meeting people voted for making a backup of at least urpmi.cfg before deleting
all the medias.
see http://meetbot.mageia.org/mageia-dev/2015/mageia-dev.2015-02-05-19.39.html
Comment 8 David Walser 2015-02-06 21:08:12 CET
The idea of only deleting media that use the Mageia GPG key was brought up in the meeting and sounds like the right solution.

Making assumptions about whether third party media that people have added is release-dependent makes no sense.  If it is, let the admin/user figure that out.  Don't just blindly delete everything.

Resolution: WONTFIX => (none)
Status: RESOLVED => REOPENED

Comment 9 Thierry Vignaud 2015-02-06 21:50:07 CET
It's not.
It may break urpmi...
Eg we can imagine a mga5 package has new deps that are also provided by a package from the repo but has non explicit file conflicts, ...

I've seen urpmi looping forever when upgrading b/c both N & N+1 media were set up.

On the other hand, the upgrade could fail b/c a package could not be removed b/c a 3rd party package has outdated dependencies no more backed in new media...
Comment 10 Angelo Naselli 2015-02-06 21:55:38 CET
>On the other hand, the upgrade could fail b/c a package could not be removed b/c 
> a 3rd party package has outdated dependencies no more backed in new media...
But that also with the media removed.
I think the solution of a urpmi.cfg backup could be a good compromise at least for the incoming mga5
Comment 11 Rémi Verschelde 2015-02-06 21:59:33 CET
I think the discussion is not about keeping third-party repos *enabled* during the upgrade, but to restore them after the upgrade.
I don't know how it works currently, but maybe urpmi can use a "clean" urpmi.cfg file during the upgrade, and restore the previous one after the install (after having replaced the old official repos by the new ones too).
Comment 12 Angelo Naselli 2015-02-06 22:05:09 CET
well maybe it's better to remove our repo to disable third party ones and
add our new medias from mirrors. 
At the final checkup after installation the urpm-edit-media view could
be provided to let the user enabling the repos back the ones he wanted to.
We cannot know which could be enabled and which not. I have 3 private repos
and google-chrome one if you enable all of them you will enable mga4 private repos. But you cannot know that just from the name...
Comment 13 Rémi Verschelde 2015-02-06 22:13:37 CET
(In reply to Angelo Naselli from comment #12)
> At the final checkup after installation the urpm-edit-media view could
> be provided to let the user enabling the repos back the ones he wanted to.

That sounds good. So the procedure would be:
- Remove Mageia N official repos
- Disable all other repos
- Add Mageia N+1 official repos
- Upgrade...
- After the upgrade, bring up urpm-edit-media and let the user do their job.
Comment 14 Angelo Naselli 2015-02-06 22:17:57 CET
>- After the upgrade, bring up urpm-edit-media and let the user do their job.
or adding a "tab" for that configuration too, since there is a page with all the configuration to be checked
Comment 15 David Walser 2015-02-06 22:45:06 CET
Thanks Angelo, that makes a lot of sense, especially when there's something remotely similar that allows you to check core vs. nonfree vs. tainted before the installation, having something that allows you to do the same with your non-native media afterward would probably not be too confusing.
Comment 16 Angelo Naselli 2015-02-07 17:41:55 CET
Created attachment 5870 [details]
patch that saves a backup of urpmi.cfg

Just a guess since i gave a quick look to the code... and i cannot do any tests until next Thursday. But any comments is welcome so that i won't do any useless tests :)
Comment 17 Marja Van Waes 2015-02-08 19:37:56 CET
Sorry, Angelo, I'm giving up on trying to test your patch.

I've created many drakx-installing-stage2 tarballs, but no matter what I do, after every upgrade install ddebug.log keeps saying that version 16.54 was used.

First I replaced the drakx-installer-stage2 rpm that was in my local mirror, with a copy of the drakx-installer-stage2 rpm that I built locally from the new tarball containing your patch. However, I should probably have done the following:

Then I copied new mdkinst.sqfs from the new package into /my/mageia/mirror/distrib/cauldron/x86_64/install/stage2/ instead
However, the VERSION file in the same directory as where mdkinst.sqfs came from, still said "16.54", no matter which crazy versions the tarball and the package had that I had made, so I don't know which version I was really testing :-(

Anyway, on not one of my numerous test I found a /etc/urpmi/urpmi.cfg.bak

The only mention I see of urpmi.cfg, is 

* wrote config file [/mnt/etc/urpmi/urpmi.cfg]
(after * starting step `choosePackages')

CC: (none) => marja11

Comment 18 Marja Van Waes 2015-02-08 22:36:07 CET
Created attachment 5871 [details]
stage2 failing with error about urpmi.cfg.bak

Maybe my mistake was that I hadn't git added and committed the changed files :-/

Anyway, I tried again, now after adding and committing.

Package building failed on some localisation stuff, but I found a nice mdkinst.sqfs and VERSION (with correct version) in BUILDROOT, which I copied to my local mirror.

After that, boot-nonfree.iso install failed when loading stage2:

String found where operator expected at /usr/bin/libDrakX/install/media.pm line 896, near ""$::prefix/etc/urpmi/urpmi.cfg" "$::prefix/etc/urpmi/urpmi.cfg.bak""
     (Missing operator before "$::prefix/etc/urpmi/urpmi.cfg.bak"?)

See attached screenshot
Comment 19 Angelo Naselli 2015-02-08 23:05:26 CET
marja sorry i was wrong in the patch (not tested as said)
eval { cp_af "$::prefix/etc/urpmi/urpmi.cfg", "$::prefix/etc/urpmi/urpmi.cfg.bak" };

I forgot the "," between two parameters... maybe
Comment 20 Colin Guthrie 2015-02-08 23:15:13 CET
(In reply to Angelo Naselli from comment #16)
> Created attachment 5870 [details]
> patch that saves a backup of urpmi.cfg
> 
> Just a guess since i gave a quick look to the code... and i cannot do any
> tests until next Thursday. But any comments is welcome so that i won't do
> any useless tests :)

Seems like a reasonable change (except the missing comma!)

I still think that in an ideal world, we should do the following:

1. Take original urpmi.cfg and update any official media to be MGA(N+1) (basically delete all media with our key and add back in official media again.
2. Write that to disk
3. Bind mount a temporary file over the top of the written file.
4. Write nicer config to that tmp file which *only* has official media and thus not any unknown sources defined.

After we're done, the bind mount would be umounted and the user's config will be revealed again.

Either that or just make installer urpmi use a different config.

All that said, this is a bit beyond my perl skills so I think your solution is a good one for now!
Comment 21 Florian Hubold 2015-02-09 00:00:57 CET
(In reply to Thierry Vignaud from comment #5)
> 
> (In reply to Florian Hubold from comment #3)
> > Why not simply set them to ignored instead of deleting them? 
> 
> And adding a warning would just make the installer harder to understand to
> most end users...

It could simply be added to release notes, which endusers are expected to read, and which are already shown in the installer IIRC.

So the solution from Angelo or as proposed by Colin would totally suffice, to only use official media during installer and restore the other repos afterwards.
Comment 22 Angelo Naselli 2015-02-09 09:56:16 CET
on Comment #20 the question is, does that work? :)

Colin I think we can fix this bug this way and open a new one after a discussion
during a meeting when mga5 is out. 
The way to take is clear, the way to implement could be chosen later, but imo
very soon after mga5 is out. This kind of changes has to be tested early :D
Comment 23 Marja Van Waes 2015-02-09 14:01:59 CET
Sorry, Angelo, with the additional comma stage2 now starts fine, but I still don't have a urpmi.cfg.bak after reboot
Comment 24 Angelo Naselli 2015-02-13 23:30:31 CET
@Marja that means that the patch was not good, it was a guess as i said. I will try to study better this issue if colin has not fixed yet.
Rémi Verschelde 2015-03-09 21:44:25 CET

Blocks: (none) => 14069

Rémi Verschelde 2015-03-09 21:48:12 CET

Blocks: 14069 => (none)

Comment 25 Dave Hodgins 2015-03-09 22:07:40 CET
Note added to https://wiki.mageia.org/en/Mageia_5_Release_Notes#Upgrading_from_Mageia_4

Severity lowered to enhancement, for Mageia 6, as it's too late to be
making changes to the installer, that are not for release blocker bugs.

Severity: normal => enhancement
CC: (none) => davidwhodgins

Comment 26 katnatek 2023-09-23 03:27:58 CEST
What about allow to configure third party repositories in the installer, of course giving a warning of do on user risk.

BTW we are on Mageia 9 and i no think that the change is implemented but maybe i'm wrong

Hardware: i586 => All

katnatek 2023-09-23 19:19:28 CEST

CC: (none) => j.alberto.vc

Comment 27 Dave Hodgins 2023-09-23 20:34:20 CEST
I agree with comments 5 and 9. Upgrading is difficult enough to get right which
is part of the reason it requires such thorough testing prior to release.

People are warned to remove the third party repos and the packages installed
from them in
https://wiki.mageia.org/en/How_to_choose_the_right_Mageia_upgrade_method#Preparations

Depending on where it fails, a failed upgrade may leave the system un-bootable,
or it may be left in a state where it's not possible to start a desktop
environment.

It's too risky in my opinion, not to remove the third party repos.
Comment 28 katnatek 2023-09-23 20:55:50 CEST
(In reply to Dave Hodgins from comment #27)
> I agree with comments 5 and 9. Upgrading is difficult enough to get right
> which
> is part of the reason it requires such thorough testing prior to release.
> 
> People are warned to remove the third party repos and the packages installed
> from them in
> https://wiki.mageia.org/en/
> How_to_choose_the_right_Mageia_upgrade_method#Preparations
> 
> Depending on where it fails, a failed upgrade may leave the system
> un-bootable,
> or it may be left in a state where it's not possible to start a desktop
> environment.
> 
> It's too risky in my opinion, not to remove the third party repos.

Well i migrate mageia 8 to mageia 9 in times of b1 with the blogdrake repositories for mageia 9 with urpmi and not find any issue, but is a small repo and is not a representative testing

So closing this as OLD

Status: REOPENED => RESOLVED
Resolution: (none) => OLD

Comment 29 David Walser 2023-09-23 22:24:12 CEST
Changing to WONTFIX as it sounds like the bug is still valid.  I guess the moral is don't use the installer to upgrade.

Resolution: OLD => WONTFIX

Comment 30 katnatek 2023-09-24 00:14:50 CEST
(In reply to David Walser from comment #29)
> Changing to WONTFIX as it sounds like the bug is still valid.  I guess the
> moral is don't use the installer to upgrade.

At less if you like/need software from sources external to mageia and you like to have some adventure ;)

The installer is good for upgrades without external sources
Comment 31 katnatek 2023-09-24 00:19:29 CEST
(In reply to katnatek from comment #30)
> (In reply to David Walser from comment #29)
> > Changing to WONTFIX as it sounds like the bug is still valid.  I guess the
> > moral is don't use the installer to upgrade.
> 
> At less if you like/need software from sources external to mageia and you
> like to have some adventure ;)
> 
> The installer is good for upgrades without external sources

And the installer is also a good way of reduce downloads if you now how to add to urpmi sources for the upgrade, specially if you have to upgrade more than one system

Note You need to log in before you can comment on or make changes to this bug.