Bug 24362 - Change default package downloader to wget
Summary: Change default package downloader to wget
Status: NEW
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: release_blocker major
Target Milestone: Mageia 9
Assignee: Mageia tools maintainers
QA Contact:
Whiteboard: MGA8TOO
: 30075 31049 (view as bug list)
Depends on:
Reported: 2019-02-15 04:05 CET by Dave Hodgins
Modified: 2023-02-05 18:22 CET (History)
9 users (show)

See Also:
Source RPM: urpmi
Status comment:


Description Dave Hodgins 2019-02-15 04:05:24 CET
For the last couple of hours, while testing cauldron installs it fails
to set up the online media at the end of the installation, despite the online
media being added at the beginning of the install, and continues to fail
trying to retrieve the $mirrorlist when running drakrpm-edit-media after
booting into the installed system. Using the options/global options to
change the downloader to wget works around the problem.

In addition, there have been problems in the past with some mirrors, when
using aria2c, that go away after switching to wget.

While aria2c may be more efficient, when the mirror allows multiple connections,
it's safer to default to wget.
Comment 1 David Walser 2019-02-15 14:20:55 CET
I agree, I've always found that wget works better than aria2c and curl as the urpmi downloader.
Comment 2 Manuel Hiebel 2019-02-15 23:09:58 CET
or we use aria2 with metalink (needs mirrorbrain or co)
Comment 3 Dave Hodgins 2019-02-15 23:40:41 CET
(In reply to Manuel Hiebel from comment #2)
> or we use aria2 with metalink (needs mirrorbrain or co)

Which we don't currently have set up. Switching the default from aria2c to
wget is a very simple change that works with what we have now.
Comment 4 Sébastien Morin 2019-03-25 20:49:34 CET
Still valid with Mga7-beta3 (March 24th)

Adding mirrors for the different media always ends up with an error message saying the media couldn't be downloaded, which is misleading since they all seem absolutely usable right away (run # urpmi --auto-update in a terminal to see they are up-to-date).

Pushing the priority up.

Keywords: (none) => 7beta3
Priority: Normal => High
CC: (none) => sebsweb

Comment 5 mtm spm 2020-06-29 13:59:04 CEST
How do I change drakconf to use wget instead of aria ?
With aria (likely due to my company paranoid proxy) the installation timeout and is impossible to install any new software from GUI.

CC: (none) => mtm_spm

Comment 6 Dave Hodgins 2020-06-29 14:38:26 CEST
Using rpmdrake, select Options, then select Media Manager.

That will run drakrpm-edit-media.

In drakrpm-edit-media (aka Media Manager), select Options, then Global Options.
Use the selection box for the "Download program to use" and select wget.
Note that wget must be installed for the option to use it to be shown.
Comment 7 Dave Hodgins 2020-06-29 14:44:22 CEST
If you're using a text only install, edit the global options in
/etc/urpmi/urpmi.cfg and add a line with ...
  downloader: wget

For example, in my system ...
# head -n 6 /etc/urpmi/urpmi.cfg
  downloader: wget
  resume: 1
  verify-rpm: 1
  xml-info: always
Comment 8 Aurelien Oudelet 2020-09-19 18:08:50 CEST
This is High priority bug for a good reason.

Making Mageia even better than ever is best direction.
In order to do right thing, this bug should be examined and fixed as soon as possible.

Packagers, please make the status to Assigned when you are working on this.
Feel free to reassign the bug if bad-triaged. Also, if bug is old, please close it.

On October 1st 2020, we will drop priority to normal.
Comment 9 Morgan Leijström 2021-03-01 22:08:17 CET
This really is night and day; repeated failures trying to update and install new packages, with default for a couple hours tries now and then!

( Additionally it fail with unspecified error in the popup, should rise a bug for that too; user is completely lost! )

Changed to wget and no failures.

Circumstances: live xfce 64 with persistence on Dell M4400 laptop, wifi, mirror umu.se

Keywords: 7beta3 => 8final
CC: (none) => fri

Comment 10 Morgan Leijström 2021-03-02 01:38:19 CET
BTW, default is now not aria, but curl

(Mageia 8 final Live Plasma and Xfce checked)

Summary: Change default downloader from aria2c to wget => Change default package downloader from wget

Dave Hodgins 2021-03-02 03:43:35 CET

Summary: Change default package downloader from wget => Change default package downloader to wget

Comment 11 Morgan Leijström 2021-03-02 09:17:54 CET
Doh, thanks, it was late night, deleted wrong set of words... :)
Comment 12 Morgan Leijström 2021-03-04 21:20:24 CET
Not being able to update or install new packages is severe and hard to fis for new users who dont know.  The error messages are not helpful at all, as they complain about the packages instead!

I believe the problem is triggered by a combination of wifi harware and router.
(and possibly versions of drivers, protocol...)
Basically always fail on my Dell M4400, never on Thinkpad T400, same router.

IMO it deserves a visual place, so in Errata now:

Keywords: (none) => IN_ERRATA8

Comment 13 Morgan Leijström 2021-04-07 23:24:27 CEST
Another reference: in Bug 28744 report.bug contain a few curl failures, but maybe not enough to point finger at curl.
Comment 14 Morgan Leijström 2021-04-14 22:59:50 CEST
On my affected laptop Dell Precision M4400, both wget and also aria2 works perfectly.

Without changing from curl it is impossible to update a mga8 install at all, it does not get through the initial batch of gcc, rpm etc packages.
Morgan Leijström 2021-04-14 23:00:42 CEST

Whiteboard: (none) => MGA7TOO, MGA8TOO
Keywords: 8final => (none)

Comment 15 Morgan Leijström 2021-12-07 14:47:36 CET
FWIW same problem on my "new" laptop Thinkpad T510
So default setting is failing on 2 of 4 laptops here...

Two recent questions on this from users on our forum:

Source RPM: urpmi-8.114-2.mga7.src.rpm => urpmi
Whiteboard: MGA7TOO, MGA8TOO => MGA8TOO
See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=28539

Comment 16 Morgan Leijström 2021-12-07 14:57:25 CET
Added error message example and urpmi --clean tip to errata.
Morgan Leijström 2021-12-08 16:35:01 CET

Target Milestone: --- => Mageia 9

Comment 17 Thierry Vignaud 2022-01-17 00:59:46 CET
I'm not against it but I think I remember that at some point we were defaulting to wget which didn't work for some people.

I tried to do a quick search in history.
Prior to 2009, drakx was using its own downloader:

Urpmi always defaulted to curl if installed:

CC: (none) => thierry.vignaud

Comment 18 Dave Hodgins 2022-01-17 01:11:41 CET
As per the start of this bug report, it was aria2c that had the most problems,
then curl, with wget being much more reliable.
Comment 19 Frank Griffin 2022-01-17 01:48:20 CET
I'm not really partial to any of them, provided they work.  If you want to fix curl (or that's an MGA problem rather than a curl problem), that's fine with me.

If we're updating the download support, I did have a previous bug report saying that an install referencing a hostname for the repository did a DNS lookup for each reference rather than do it once and cache the IP address, which would be much more efficient.  Not to mention that it would seem to circumvent this problem, since curl is complaining midstream about no longer being able to resolve the repository hostname.

CC: (none) => ftg

Comment 20 Martin Whitaker 2022-01-17 23:04:27 CET
I suspect the main problem with curl is that by default it doesn't retry on transient errors. Using the --retry option could help.

CC: (none) => mageia

Comment 21 Frank Griffin 2022-01-18 03:18:06 CET
Actually, the error reported by curl was that it couldn't resolve a hostname that it had used countless times before in the same install.

Coincidentally, in bug#28539, we see that the install stage1 can't resolve a hostname even though it has been passed a valid DNS server address and installed it in /etc/resolv.conf.  This could be some anomaly between whatever the stage 1 installer and perl in stage 2 uses for DNS as opposed to the standard resolver.
Comment 22 Morgan Leijström 2022-01-18 08:01:56 CET
(In reply to Martin Whitaker from comment #20)
> I suspect the main problem with curl is that by default it doesn't retry on
> transient errors.

That default sounds really stupid...!

> Using the --retry option could help.

Is it possible to set the current drak tools to use curl with --retry?

(I could try using it so then for a while as a test)
Comment 23 Martin Whitaker 2022-01-18 09:55:17 CET
You can use

  urpmi --downloader curl --retry 5 ...

(you only need the --downloader option if you have configured urpmi to use another method)
Comment 24 Morgan Leijström 2022-01-18 15:34:28 CET
thanks, will try that.

Any way to command curl ---retry for the updater app or drakrpm?
Comment 25 Frank Griffin 2022-01-18 16:49:24 CET
According to TV in bug#28539 if you're using the legacy boot image (the one with a command line where you can enter parameters) saying "linux downloader=wget".
Comment 26 Morgan Leijström 2022-01-18 16:54:26 CET
I am (also) chasing the problem users have when using mgaapplet and GUI tools for regular updating and program install.
Comment 27 Martin Whitaker 2022-01-18 17:05:04 CET
'man urpmi.cfg' tells me you can add the --retry option in /etc/urpmi/urpmi.cfg. I've not tried it.
Comment 28 Morgan Leijström 2022-01-19 14:34:20 CET
Tested drakrpm with curl attempting to making it retry by editing /etc/urpmi/urpmi.cfg, per 'man urpmi.cfg' :

Adding the line
After the line specifying the downloader does not work.
Checking by launching urpmi, urpmi spits out that it do not recognise that option.

Adopting to style of the other options, I changed it to
    retry: 3
Then it did not complain, but it also did not help
(using MCC -> update, like normal users - did not try using plain urpmi yet.)
Could also be that it drops connection so often that retries is futile:
When running, it takes just a second to fail all kernel related packages, not much at all got retrieved.
I guess curl use some protocol/dialect incompatible with something or a combination of things between internet and me. 

Both wget and aria2 works, every time I tried.  curl never.

I issued
# urpmi -clean
in between tests.

Interestingly, this machine worked mostly OK with default curl using our previous routers, but not the new ones, which apparently make this problem worse.  Could also be some change upstream lately.


Laptop: Asus A717, wifi Atheros QCA6174, driver ath10k_pci (mga8 x86_64 Plasma)

Router(s): TP-link Deco S4, on latest fw (20210607), mesh of 3 units, laptop not using the one connected to fibre modem.  Administration app say all are well.

Fibre modem: hard to reach, can check if interest...

Fibre operator: IP-Only installed fibre, modem, their station

Internet supplier: Bredband2 (maybe too cheap, lost connection due to power failure (no UPS) almost all of monday due to storm, while others around using other suppliers had no problem.  No working fallback, nor excuse. so guess they generally don't aim for reliability...)
Comment 29 Morgan Leijström 2022-01-19 20:51:09 CET
(In reply to Martin Whitaker from comment #23)
> You can use
>   urpmi --downloader curl --retry 5 ...

Unfortunately retries do not help.

Small packages work either way, but large like kernel is impossible for me on all machines using curl, also with --retry

$ sudo urpmi --downloader curl kernel-desktop-5.10.16-1.mga8
-> "bad rpms"

$ sudo urpmi --clean
$ sudo urpmi --downloader curl --retry 5 kernel-desktop-5.10.16-1.mga8
-> "bad rpms"

$ sudo urpmi --clean
$ sudo urpmi --downloader wget kernel-desktop-5.10.16-1.mga8
-> success

This test was on my wired ethernet workhorse.
Before "upgrading" (duh!) the router, curl worked here.
Comment 30 Morgan Leijström 2022-01-21 22:27:20 CET
Acer Aspire One laptop *do* work reliably per default settings.
Running updated 64-bit mga8, Xfce
wifi: Qualcomm Atheros AR9285, driver ath9k
CPU&GPU: Atom N450

Another one on the *failing* list is Thinkpad T43, with
Booted 32-bit mga8 Xfce live
wifi: Intel PRO/Wireless 2200BG (Calexico2), driver ipw2200
CPU Pentium M
Morgan Leijström 2022-01-22 18:45:28 CET

See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=26827

Comment 31 Morgan Leijström 2022-02-01 00:51:42 CET
Trying a netinstall now it of course fail for the same reason...
Is there a way today to trick mga8 netinstall to use wget or aria?
Comment 32 Dave Hodgins 2022-02-22 20:59:22 CET
*** Bug 30075 has been marked as a duplicate of this bug. ***

CC: (none) => kmmos1

Comment 33 Martin Whitaker 2022-10-30 10:47:17 CET
*** Bug 31049 has been marked as a duplicate of this bug. ***

CC: (none) => westel

Comment 34 Frank Griffin 2022-11-02 19:12:30 CET
This is still happening in the installer during fresh installs.  Again, it's a 6 RC from curl trying to download media.cfg using the same hostname that stage 1 used to load stage 2.

Elevating to release-blocker since not being able to run the install is sort of critical.

Priority: High => release_blocker

Comment 35 Frank Griffin 2022-11-02 19:34:02 CET
I had previously thought that the problem was that stage 2 was losing knowledge of the DNS server which DHCP passed to stage 1, but I've just tried an HTTP install from my in-house cauldron mirror using an IP address rather than a hostname and it fails the same way but with a curl error 7 (TCP connect failed) trying to get media.cfg.  So the problem is not necessarily anything in curl but instead that stage 2 is losing the network completely by the time it looks for media.cfg.
Comment 36 Frank Griffin 2022-11-12 19:14:26 CET
Just as a followup, since I couldn't get a netinstall to work, I downloaded the MGA9 Alpha and did a DVD install.  The wifi works perfectly.
Lewis Smith 2022-11-18 13:54:23 CET

Keywords: (none) => FOR_ERRATA9

Comment 37 Morgan Leijström 2022-11-25 21:38:57 CET
Still valid mga9 alpha1 live plasma, curl 7.86.0-1
Comment 38 Martin Whitaker 2022-12-03 18:59:02 CET
Changing the default to wget would only partially fix the original complaint, as urpmi will still use aria2c to download metadata. You have to explicitly set the downloader to override that. And it's aria2c that causes me the most problems with my nearest mirror (mirrorservice.org). That's not to say I'm against changing the default though.

However, as there's been no progress on this, I've modified the stage 2 installer to let the user select a specific downloader, both when adding additional media at the beginning, and when configuring update media at the end. I've also allowed the user to select a specific mirror when configuring update media, rather than always using the nearest mirror.
Comment 39 Morgan Leijström 2022-12-03 20:19:21 CET
Great, thanks
I note to docteam ML for documenting
Comment 40 Morgan Leijström 2022-12-25 18:40:41 CET
The downloader selection option seem not yet implemented?
netinstall dated 2022-12-24, 	Stage 2 mdkinst.sqfs 2022-12-21 

 -  Or what is the secret find the settings?
Comment 41 Martin Whitaker 2022-12-29 10:34:03 CET
(In reply to Morgan Leijström from comment #40)
> The downloader selection option seem not yet implemented?
> netinstall dated 2022-12-24, 	Stage 2 mdkinst.sqfs 2022-12-21 
>  -  Or what is the secret find the settings?

When using netinstall you aren't offered the option of adding additional media, so the choice of downloader will only appear when adding on-line media after installation is complete. Did you get that far?

If you want to change the downloader used by netinstall itself, that needs to be done in stage 1, not stage 2. That's supported by the "downloader=" boot command line option, as described in bug 28539.
Comment 42 Morgan Leijström 2022-12-29 13:25:25 CET
Ah thanks.

No chance here to get netinstall installation that far as to adding additional media as there are too plenty errors with the default downloader...

Stage 2 asks what media to use (i .e tainted etc.)
That would be a good place to ask about downloader :)

Will try "downloader=" option  later.

If we by this bug decide to default to wget, the default of stage 1 IMO should also be wget.
Comment 43 Morgan Leijström 2023-01-09 12:01:16 CET
 "downloader=wget" works for me

(In reply to Martin Whitaker from comment #38)
> Changing the default to wget would only partially fix the original
> complaint, as urpmi will still use aria2c to download metadata.
> You have to explicitly set the downloader to override that. 

Shouldn't we then raise separate a bug (depending on this bug) about using same default for metadata as for rpms?

Really confusing otherwise.

> And it's aria2c that causes
> me the most problems with my nearest mirror (mirrorservice.org). 

For me as default the metadata seem to never fail, so here aria2 (and wget) works, but not curl - it fails in the first rpm download/install transaction.
I have not tried different mirrors, using only umu.se.

If anyone have time, It could be interesting to try all mirrors with all three downloadrers and put the result in an table - i winder if it would be the same result for all users or it also depends on some bug in any box on the line or ISP traffic shaping methods.

What method is used to retrieve stage2, and can that be user altered if necessary?
Comment 44 Herman Viaene 2023-01-09 12:09:45 CET
As far as I understand this, we all seem to be barking up the wrong tree. The issue is not the installation procedure or drakconf, but urpmi, right???

CC: (none) => herman.viaene

Comment 45 Morgan Leijström 2023-01-09 12:26:41 CET
In a default unaltered system, aria2c is used to download synthesis files, and curl is used to download rpm files.

That discrepancy itself is confusing, and should get fixed.

If I understand correctly, only if user explicitly set default downloader (i.e in GUI) that downloader is used, in all cases.

The above I think is for a separate bug.


urpmi can use different downloaders: aria2, curl, wget 

This bug is about what downloader it should use per default - for metadata & rpm files, for all installers, (and for netinstaller stage2 ?)

It seems wget is most reliable for most people.
The current default, curl, seem to be worst, and some have problems with aria2.
Comment 46 Martin Whitaker 2023-01-17 18:00:55 CET
(In reply to Morgan Leijström from comment #43)
> For me as default the metadata seem to never fail, so here aria2 (and wget)
> works, but not curl - it fails in the first rpm download/install transaction.
> I have not tried different mirrors, using only umu.se.

Are you using HTTP or FTP? If HTTP, that's probably bug 31339.
Comment 47 Morgan Leijström 2023-01-17 20:33:40 CET
http. Added comment in bug 31339.
Comment 48 Martin Whitaker 2023-01-21 13:30:32 CET
As Thierry wrote in comment 17 that he wasn't against it, I have made the change. wget will now be preferred over curl if it is installed. aria2 will still be preferred for metadata unless a downloader is explicitly selected. As I don't know why metadata is treated as a special case, I am unwilling to change that aspect.

This can be tested when urpmi-8.129-1.mga9 and drakx-installer-stage2-18.54-2.mga9 reach the mirrors.
Comment 49 Martin Whitaker 2023-01-23 20:28:42 CET
From https://ml.mageia.org/l/arc/discuss/2023-01/msg00091.html, it turns out there is a good reason to keep curl as the default. If the user makes changes to /etc/wgetrc, that can conflict with the wget options urpmi uses, and thus prevent urpmi from downloading any files.

curl has an option to skip reading its global configuration fila, and urpmi uses that option. wget does not have such an option.

So I think we'll need to revert the change I made.
Comment 50 Morgan Leijström 2023-01-23 20:34:55 CET
(In reply to Martin Whitaker from comment #49)
> curl has an option to skip reading its global configuration fila, and urpmi
> uses that option. wget does not have such an option.

$ LC_ALL=C wget --help | grep config
       --config=FILE               specify config file to use
       --no-config                 do not read any config file
Comment 51 Martin Whitaker 2023-01-23 20:43:49 CET
Grrr. I searched for mentions of wgetrc!
Comment 52 Frank Griffin 2023-02-05 18:22:48 CET
"Be careful what you  wish for, you might get it"

I just tried a network HTTP install, and got:

* retrieving   media.cfg
* '/usr/bin/wget' '--force-clobber' '-t' '3' '--retr-symlinks' '--timeout=60' '-P' '/mnt/var/cache/urpmi' 'http://ftgfiles1/mnt/cauldron/x86_64/media/media_info/media.cfg' |
* error: wget failed: exited with 4

* retrieval of [/mnt/var/cache/urpmi/media.cfg] failed

wget rc 4 indicates a network failure, but my in-house mirror is running fine and media.cfg is exactly where the installer is looking for it.

Note You need to log in before you can comment on or make changes to this bug.