Bug 24362 - Change default package downloader to wget
Summary: Change default package downloader to wget
Status: NEW
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: release_blocker major
Target Milestone: Mageia 9
Assignee: Mageia tools maintainers
QA Contact:
URL:
Whiteboard: MGA8TOO
Keywords: FOR_ERRATA9, IN_ERRATA8
: 30075 31049 (view as bug list)
Depends on:
Blocks:
 
Reported: 2019-02-15 04:05 CET by Dave Hodgins
Modified: 2022-12-03 20:19 CET (History)
8 users (show)

See Also:
Source RPM: urpmi
CVE:
Status comment:


Attachments

Description Dave Hodgins 2019-02-15 04:05:24 CET
For the last couple of hours, while testing cauldron installs it fails
to set up the online media at the end of the installation, despite the online
media being added at the beginning of the install, and continues to fail
trying to retrieve the $mirrorlist when running drakrpm-edit-media after
booting into the installed system. Using the options/global options to
change the downloader to wget works around the problem.

In addition, there have been problems in the past with some mirrors, when
using aria2c, that go away after switching to wget.

While aria2c may be more efficient, when the mirror allows multiple connections,
it's safer to default to wget.
Comment 1 David Walser 2019-02-15 14:20:55 CET
I agree, I've always found that wget works better than aria2c and curl as the urpmi downloader.
Comment 2 Manuel Hiebel 2019-02-15 23:09:58 CET
or we use aria2 with metalink (needs mirrorbrain or co)
Comment 3 Dave Hodgins 2019-02-15 23:40:41 CET
(In reply to Manuel Hiebel from comment #2)
> or we use aria2 with metalink (needs mirrorbrain or co)

Which we don't currently have set up. Switching the default from aria2c to
wget is a very simple change that works with what we have now.
Comment 4 Sébastien Morin 2019-03-25 20:49:34 CET
Still valid with Mga7-beta3 (March 24th)

Adding mirrors for the different media always ends up with an error message saying the media couldn't be downloaded, which is misleading since they all seem absolutely usable right away (run # urpmi --auto-update in a terminal to see they are up-to-date).

Pushing the priority up.

Keywords: (none) => 7beta3
CC: (none) => sebsweb
Priority: Normal => High

Comment 5 mtm spm 2020-06-29 13:59:04 CEST
How do I change drakconf to use wget instead of aria ?
With aria (likely due to my company paranoid proxy) the installation timeout and is impossible to install any new software from GUI.

CC: (none) => mtm_spm

Comment 6 Dave Hodgins 2020-06-29 14:38:26 CEST
Using rpmdrake, select Options, then select Media Manager.

That will run drakrpm-edit-media.

In drakrpm-edit-media (aka Media Manager), select Options, then Global Options.
Use the selection box for the "Download program to use" and select wget.
Note that wget must be installed for the option to use it to be shown.
Comment 7 Dave Hodgins 2020-06-29 14:44:22 CEST
If you're using a text only install, edit the global options in
/etc/urpmi/urpmi.cfg and add a line with ...
  downloader: wget

For example, in my system ...
# head -n 6 /etc/urpmi/urpmi.cfg
{
  downloader: wget
  resume: 1
  verify-rpm: 1
  xml-info: always
}
Comment 8 Aurelien Oudelet 2020-09-19 18:08:50 CEST
Hi,
This is High priority bug for a good reason.

Making Mageia even better than ever is best direction.
In order to do right thing, this bug should be examined and fixed as soon as possible.

Packagers, please make the status to Assigned when you are working on this.
Feel free to reassign the bug if bad-triaged. Also, if bug is old, please close it.

On October 1st 2020, we will drop priority to normal.
Comment 9 Morgan Leijström 2021-03-01 22:08:17 CET
This really is night and day; repeated failures trying to update and install new packages, with default for a couple hours tries now and then!

( Additionally it fail with unspecified error in the popup, should rise a bug for that too; user is completely lost! )

Changed to wget and no failures.

Circumstances: live xfce 64 with persistence on Dell M4400 laptop, wifi, mirror umu.se

Keywords: 7beta3 => 8final
CC: (none) => fri

Comment 10 Morgan Leijström 2021-03-02 01:38:19 CET
BTW, default is now not aria, but curl

(Mageia 8 final Live Plasma and Xfce checked)

Summary: Change default downloader from aria2c to wget => Change default package downloader from wget

Dave Hodgins 2021-03-02 03:43:35 CET

Summary: Change default package downloader from wget => Change default package downloader to wget

Comment 11 Morgan Leijström 2021-03-02 09:17:54 CET
Doh, thanks, it was late night, deleted wrong set of words... :)
Comment 12 Morgan Leijström 2021-03-04 21:20:24 CET
Not being able to update or install new packages is severe and hard to fis for new users who dont know.  The error messages are not helpful at all, as they complain about the packages instead!

I believe the problem is triggered by a combination of wifi harware and router.
(and possibly versions of drivers, protocol...)
Basically always fail on my Dell M4400, never on Thinkpad T400, same router.

IMO it deserves a visual place, so in Errata now:
https://wiki.mageia.org/en/Mageia_8_Errata#Downloading_software

Keywords: (none) => IN_ERRATA8

Comment 13 Morgan Leijström 2021-04-07 23:24:27 CEST
Another reference: in Bug 28744 report.bug contain a few curl failures, but maybe not enough to point finger at curl.
Comment 14 Morgan Leijström 2021-04-14 22:59:50 CEST
On my affected laptop Dell Precision M4400, both wget and also aria2 works perfectly.

Without changing from curl it is impossible to update a mga8 install at all, it does not get through the initial batch of gcc, rpm etc packages.
Morgan Leijström 2021-04-14 23:00:42 CEST

Keywords: 8final => (none)
Whiteboard: (none) => MGA7TOO, MGA8TOO

Comment 15 Morgan Leijström 2021-12-07 14:47:36 CET
FWIW same problem on my "new" laptop Thinkpad T510
So default setting is failing on 2 of 4 laptops here...

Two recent questions on this from users on our forum:
https://forums.mageia.org/en/viewtopic.php?t=14417

Source RPM: urpmi-8.114-2.mga7.src.rpm => urpmi
Whiteboard: MGA7TOO, MGA8TOO => MGA8TOO
See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=28539

Comment 16 Morgan Leijström 2021-12-07 14:57:25 CET
Added error message example and urpmi --clean tip to errata.
Morgan Leijström 2021-12-08 16:35:01 CET

Target Milestone: --- => Mageia 9

Comment 17 Thierry Vignaud 2022-01-17 00:59:46 CET
I'm not against it but I think I remember that at some point we were defaulting to wget which didn't work for some people.

I tried to do a quick search in history.
Prior to 2009, drakx was using its own downloader:
https://gitweb.mageia.org/software/drakx/commit/?id=f1c5acfa0a593ef6ac6cac6fa8d7fbaf7e8600af

Urpmi always defaulted to curl if installed:
https://gitweb.mageia.org/software/rpm/urpmi/commit/?id=c9fdb6d4e5b72adba69f81692ee12c2e20366a85

CC: (none) => thierry.vignaud

Comment 18 Dave Hodgins 2022-01-17 01:11:41 CET
As per the start of this bug report, it was aria2c that had the most problems,
then curl, with wget being much more reliable.
Comment 19 Frank Griffin 2022-01-17 01:48:20 CET
I'm not really partial to any of them, provided they work.  If you want to fix curl (or that's an MGA problem rather than a curl problem), that's fine with me.

If we're updating the download support, I did have a previous bug report saying that an install referencing a hostname for the repository did a DNS lookup for each reference rather than do it once and cache the IP address, which would be much more efficient.  Not to mention that it would seem to circumvent this problem, since curl is complaining midstream about no longer being able to resolve the repository hostname.

CC: (none) => ftg

Comment 20 Martin Whitaker 2022-01-17 23:04:27 CET
I suspect the main problem with curl is that by default it doesn't retry on transient errors. Using the --retry option could help.

CC: (none) => mageia

Comment 21 Frank Griffin 2022-01-18 03:18:06 CET
Actually, the error reported by curl was that it couldn't resolve a hostname that it had used countless times before in the same install.

Coincidentally, in bug#28539, we see that the install stage1 can't resolve a hostname even though it has been passed a valid DNS server address and installed it in /etc/resolv.conf.  This could be some anomaly between whatever the stage 1 installer and perl in stage 2 uses for DNS as opposed to the standard resolver.
Comment 22 Morgan Leijström 2022-01-18 08:01:56 CET
(In reply to Martin Whitaker from comment #20)
> I suspect the main problem with curl is that by default it doesn't retry on
> transient errors.

That default sounds really stupid...!

> Using the --retry option could help.

Is it possible to set the current drak tools to use curl with --retry?

(I could try using it so then for a while as a test)
Comment 23 Martin Whitaker 2022-01-18 09:55:17 CET
You can use

  urpmi --downloader curl --retry 5 ...

(you only need the --downloader option if you have configured urpmi to use another method)
Comment 24 Morgan Leijström 2022-01-18 15:34:28 CET
thanks, will try that.

Any way to command curl ---retry for the updater app or drakrpm?
Comment 25 Frank Griffin 2022-01-18 16:49:24 CET
According to TV in bug#28539 if you're using the legacy boot image (the one with a command line where you can enter parameters) saying "linux downloader=wget".
Comment 26 Morgan Leijström 2022-01-18 16:54:26 CET
I am (also) chasing the problem users have when using mgaapplet and GUI tools for regular updating and program install.
Comment 27 Martin Whitaker 2022-01-18 17:05:04 CET
'man urpmi.cfg' tells me you can add the --retry option in /etc/urpmi/urpmi.cfg. I've not tried it.
Comment 28 Morgan Leijström 2022-01-19 14:34:20 CET
Tested drakrpm with curl attempting to making it retry by editing /etc/urpmi/urpmi.cfg, per 'man urpmi.cfg' :

Adding the line
    retry
After the line specifying the downloader does not work.
Checking by launching urpmi, urpmi spits out that it do not recognise that option.

Adopting to style of the other options, I changed it to
    retry: 3
Then it did not complain, but it also did not help
(using MCC -> update, like normal users - did not try using plain urpmi yet.)
Could also be that it drops connection so often that retries is futile:
When running, it takes just a second to fail all kernel related packages, not much at all got retrieved.
I guess curl use some protocol/dialect incompatible with something or a combination of things between internet and me. 

Both wget and aria2 works, every time I tried.  curl never.

I issued
# urpmi -clean
in between tests.

Interestingly, this machine worked mostly OK with default curl using our previous routers, but not the new ones, which apparently make this problem worse.  Could also be some change upstream lately.

__Equipment:

Laptop: Asus A717, wifi Atheros QCA6174, driver ath10k_pci (mga8 x86_64 Plasma)

Router(s): TP-link Deco S4, on latest fw (20210607), mesh of 3 units, laptop not using the one connected to fibre modem.  Administration app say all are well.

Fibre modem: hard to reach, can check if interest...

Fibre operator: IP-Only installed fibre, modem, their station

Internet supplier: Bredband2 (maybe too cheap, lost connection due to power failure (no UPS) almost all of monday due to storm, while others around using other suppliers had no problem.  No working fallback, nor excuse. so guess they generally don't aim for reliability...)
Comment 29 Morgan Leijström 2022-01-19 20:51:09 CET
(In reply to Martin Whitaker from comment #23)
> You can use
> 
>   urpmi --downloader curl --retry 5 ...

Unfortunately retries do not help.

Small packages work either way, but large like kernel is impossible for me on all machines using curl, also with --retry


$ sudo urpmi --downloader curl kernel-desktop-5.10.16-1.mga8
-> "bad rpms"

$ sudo urpmi --clean
$ sudo urpmi --downloader curl --retry 5 kernel-desktop-5.10.16-1.mga8
-> "bad rpms"

$ sudo urpmi --clean
$ sudo urpmi --downloader wget kernel-desktop-5.10.16-1.mga8
-> success

This test was on my wired ethernet workhorse.
Before "upgrading" (duh!) the router, curl worked here.
Comment 30 Morgan Leijström 2022-01-21 22:27:20 CET
Acer Aspire One laptop *do* work reliably per default settings.
Running updated 64-bit mga8, Xfce
wifi: Qualcomm Atheros AR9285, driver ath9k
CPU&GPU: Atom N450

Another one on the *failing* list is Thinkpad T43, with
Booted 32-bit mga8 Xfce live
wifi: Intel PRO/Wireless 2200BG (Calexico2), driver ipw2200
CPU Pentium M
Morgan Leijström 2022-01-22 18:45:28 CET

See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=26827

Comment 31 Morgan Leijström 2022-02-01 00:51:42 CET
Trying a netinstall now it of course fail for the same reason...
Is there a way today to trick mga8 netinstall to use wget or aria?
Comment 32 Dave Hodgins 2022-02-22 20:59:22 CET
*** Bug 30075 has been marked as a duplicate of this bug. ***

CC: (none) => kmmos1

Comment 33 Martin Whitaker 2022-10-30 10:47:17 CET
*** Bug 31049 has been marked as a duplicate of this bug. ***

CC: (none) => westel

Comment 34 Frank Griffin 2022-11-02 19:12:30 CET
This is still happening in the installer during fresh installs.  Again, it's a 6 RC from curl trying to download media.cfg using the same hostname that stage 1 used to load stage 2.

Elevating to release-blocker since not being able to run the install is sort of critical.

Priority: High => release_blocker

Comment 35 Frank Griffin 2022-11-02 19:34:02 CET
I had previously thought that the problem was that stage 2 was losing knowledge of the DNS server which DHCP passed to stage 1, but I've just tried an HTTP install from my in-house cauldron mirror using an IP address rather than a hostname and it fails the same way but with a curl error 7 (TCP connect failed) trying to get media.cfg.  So the problem is not necessarily anything in curl but instead that stage 2 is losing the network completely by the time it looks for media.cfg.
Comment 36 Frank Griffin 2022-11-12 19:14:26 CET
Just as a followup, since I couldn't get a netinstall to work, I downloaded the MGA9 Alpha and did a DVD install.  The wifi works perfectly.
Lewis Smith 2022-11-18 13:54:23 CET

Keywords: (none) => FOR_ERRATA9

Comment 37 Morgan Leijström 2022-11-25 21:38:57 CET
Still valid mga9 alpha1 live plasma, curl 7.86.0-1
Comment 38 Martin Whitaker 2022-12-03 18:59:02 CET
Changing the default to wget would only partially fix the original complaint, as urpmi will still use aria2c to download metadata. You have to explicitly set the downloader to override that. And it's aria2c that causes me the most problems with my nearest mirror (mirrorservice.org). That's not to say I'm against changing the default though.

However, as there's been no progress on this, I've modified the stage 2 installer to let the user select a specific downloader, both when adding additional media at the beginning, and when configuring update media at the end. I've also allowed the user to select a specific mirror when configuring update media, rather than always using the nearest mirror.
Comment 39 Morgan Leijström 2022-12-03 20:19:21 CET
Great, thanks
I note to docteam ML for documenting

Note You need to log in before you can comment on or make changes to this bug.