I agree, I've always found that wget works better than aria2c and curl as the urpmi downloader.

or we use aria2 with metalink (needs mirrorbrain or co)

(In reply to Manuel Hiebel from comment #2)
> or we use aria2 with metalink (needs mirrorbrain or co)

Which we don't currently have set up. Switching the default from aria2c to
wget is a very simple change that works with what we have now.

Still valid with Mga7-beta3 (March 24th)

Adding mirrors for the different media always ends up with an error message saying the media couldn't be downloaded, which is misleading since they all seem absolutely usable right away (run # urpmi --auto-update in a terminal to see they are up-to-date).

Pushing the priority up.

Priority: Normal => High
CC: (none) => sebsweb
Keywords: (none) => 7beta3

How do I change drakconf to use wget instead of aria ?
With aria (likely due to my company paranoid proxy) the installation timeout and is impossible to install any new software from GUI.

CC: (none) => mtm_spm

Using rpmdrake, select Options, then select Media Manager.

That will run drakrpm-edit-media.

In drakrpm-edit-media (aka Media Manager), select Options, then Global Options.
Use the selection box for the "Download program to use" and select wget.
Note that wget must be installed for the option to use it to be shown.

If you're using a text only install, edit the global options in
/etc/urpmi/urpmi.cfg and add a line with ...
  downloader: wget

For example, in my system ...
# head -n 6 /etc/urpmi/urpmi.cfg
{
  downloader: wget
  resume: 1
  verify-rpm: 1
  xml-info: always
}

Hi,
This is High priority bug for a good reason.

Making Mageia even better than ever is best direction.
In order to do right thing, this bug should be examined and fixed as soon as possible.

Packagers, please make the status to Assigned when you are working on this.
Feel free to reassign the bug if bad-triaged. Also, if bug is old, please close it.

On October 1st 2020, we will drop priority to normal.

This really is night and day; repeated failures trying to update and install new packages, with default for a couple hours tries now and then!

( Additionally it fail with unspecified error in the popup, should rise a bug for that too; user is completely lost! )

Changed to wget and no failures.

Circumstances: live xfce 64 with persistence on Dell M4400 laptop, wifi, mirror umu.se

Keywords: 7beta3 => 8final
CC: (none) => fri

BTW, default is now not aria, but curl

(Mageia 8 final Live Plasma and Xfce checked)

Summary: Change default downloader from aria2c to wget => Change default package downloader from wget

Doh, thanks, it was late night, deleted wrong set of words... :)

Not being able to update or install new packages is severe and hard to fis for new users who dont know.  The error messages are not helpful at all, as they complain about the packages instead!

I believe the problem is triggered by a combination of wifi harware and router.
(and possibly versions of drivers, protocol...)
Basically always fail on my Dell M4400, never on Thinkpad T400, same router.

IMO it deserves a visual place, so in Errata now:
https://wiki.mageia.org/en/Mageia_8_Errata#Downloading_software

Keywords: (none) => IN_ERRATA8

Another reference: in Bug 28744 report.bug contain a few curl failures, but maybe not enough to point finger at curl.

On my affected laptop Dell Precision M4400, both wget and also aria2 works perfectly.

Without changing from curl it is impossible to update a mga8 install at all, it does not get through the initial batch of gcc, rpm etc packages.

FWIW same problem on my "new" laptop Thinkpad T510
So default setting is failing on 2 of 4 laptops here...

Two recent questions on this from users on our forum:
https://forums.mageia.org/en/viewtopic.php?t=14417

See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=28539
Source RPM: urpmi-8.114-2.mga7.src.rpm => urpmi
Whiteboard: MGA7TOO, MGA8TOO => MGA8TOO

Added error message example and urpmi --clean tip to errata.

I'm not against it but I think I remember that at some point we were defaulting to wget which didn't work for some people.

I tried to do a quick search in history.
Prior to 2009, drakx was using its own downloader:
https://gitweb.mageia.org/software/drakx/commit/?id=f1c5acfa0a593ef6ac6cac6fa8d7fbaf7e8600af

Urpmi always defaulted to curl if installed:
https://gitweb.mageia.org/software/rpm/urpmi/commit/?id=c9fdb6d4e5b72adba69f81692ee12c2e20366a85

CC: (none) => thierry.vignaud

As per the start of this bug report, it was aria2c that had the most problems,
then curl, with wget being much more reliable.

I'm not really partial to any of them, provided they work.  If you want to fix curl (or that's an MGA problem rather than a curl problem), that's fine with me.

If we're updating the download support, I did have a previous bug report saying that an install referencing a hostname for the repository did a DNS lookup for each reference rather than do it once and cache the IP address, which would be much more efficient.  Not to mention that it would seem to circumvent this problem, since curl is complaining midstream about no longer being able to resolve the repository hostname.

CC: (none) => ftg

I suspect the main problem with curl is that by default it doesn't retry on transient errors. Using the --retry option could help.

CC: (none) => mageia

Actually, the error reported by curl was that it couldn't resolve a hostname that it had used countless times before in the same install.

Coincidentally, in bug#28539, we see that the install stage1 can't resolve a hostname even though it has been passed a valid DNS server address and installed it in /etc/resolv.conf.  This could be some anomaly between whatever the stage 1 installer and perl in stage 2 uses for DNS as opposed to the standard resolver.

(In reply to Martin Whitaker from comment #20)
> I suspect the main problem with curl is that by default it doesn't retry on
> transient errors.

That default sounds really stupid...!

> Using the --retry option could help.

Is it possible to set the current drak tools to use curl with --retry?

(I could try using it so then for a while as a test)

You can use

  urpmi --downloader curl --retry 5 ...

(you only need the --downloader option if you have configured urpmi to use another method)

thanks, will try that.

Any way to command curl ---retry for the updater app or drakrpm?

According to TV in bug#28539 if you're using the legacy boot image (the one with a command line where you can enter parameters) saying "linux downloader=wget".

I am (also) chasing the problem users have when using mgaapplet and GUI tools for regular updating and program install.

'man urpmi.cfg' tells me you can add the --retry option in /etc/urpmi/urpmi.cfg. I've not tried it.

Tested drakrpm with curl attempting to making it retry by editing /etc/urpmi/urpmi.cfg, per 'man urpmi.cfg' :

Adding the line
    retry
After the line specifying the downloader does not work.
Checking by launching urpmi, urpmi spits out that it do not recognise that option.

Adopting to style of the other options, I changed it to
    retry: 3
Then it did not complain, but it also did not help
(using MCC -> update, like normal users - did not try using plain urpmi yet.)
Could also be that it drops connection so often that retries is futile:
When running, it takes just a second to fail all kernel related packages, not much at all got retrieved.
I guess curl use some protocol/dialect incompatible with something or a combination of things between internet and me. 

Both wget and aria2 works, every time I tried.  curl never.

I issued
# urpmi -clean
in between tests.

Interestingly, this machine worked mostly OK with default curl using our previous routers, but not the new ones, which apparently make this problem worse.  Could also be some change upstream lately.

__Equipment:

Laptop: Asus A717, wifi Atheros QCA6174, driver ath10k_pci (mga8 x86_64 Plasma)

Router(s): TP-link Deco S4, on latest fw (20210607), mesh of 3 units, laptop not using the one connected to fibre modem.  Administration app say all are well.

Fibre modem: hard to reach, can check if interest...

Fibre operator: IP-Only installed fibre, modem, their station

Internet supplier: Bredband2 (maybe too cheap, lost connection due to power failure (no UPS) almost all of monday due to storm, while others around using other suppliers had no problem.  No working fallback, nor excuse. so guess they generally don't aim for reliability...)

(In reply to Martin Whitaker from comment #23)
> You can use
> 
>   urpmi --downloader curl --retry 5 ...

Unfortunately retries do not help.

Small packages work either way, but large like kernel is impossible for me on all machines using curl, also with --retry


$ sudo urpmi --downloader curl kernel-desktop-5.10.16-1.mga8
-> "bad rpms"

$ sudo urpmi --clean
$ sudo urpmi --downloader curl --retry 5 kernel-desktop-5.10.16-1.mga8
-> "bad rpms"

$ sudo urpmi --clean
$ sudo urpmi --downloader wget kernel-desktop-5.10.16-1.mga8
-> success

This test was on my wired ethernet workhorse.
Before "upgrading" (duh!) the router, curl worked here.

Acer Aspire One laptop *do* work reliably per default settings.
Running updated 64-bit mga8, Xfce
wifi: Qualcomm Atheros AR9285, driver ath9k
CPU&GPU: Atom N450

Another one on the *failing* list is Thinkpad T43, with
Booted 32-bit mga8 Xfce live
wifi: Intel PRO/Wireless 2200BG (Calexico2), driver ipw2200
CPU Pentium M

Trying a netinstall now it of course fail for the same reason...
Is there a way today to trick mga8 netinstall to use wget or aria?

*** Bug 30075 has been marked as a duplicate of this bug. ***

CC: (none) => kmmos1

*** Bug 31049 has been marked as a duplicate of this bug. ***

CC: (none) => westel

This is still happening in the installer during fresh installs.  Again, it's a 6 RC from curl trying to download media.cfg using the same hostname that stage 1 used to load stage 2.

Elevating to release-blocker since not being able to run the install is sort of critical.

Priority: High => release_blocker

I had previously thought that the problem was that stage 2 was losing knowledge of the DNS server which DHCP passed to stage 1, but I've just tried an HTTP install from my in-house cauldron mirror using an IP address rather than a hostname and it fails the same way but with a curl error 7 (TCP connect failed) trying to get media.cfg.  So the problem is not necessarily anything in curl but instead that stage 2 is losing the network completely by the time it looks for media.cfg.

Just as a followup, since I couldn't get a netinstall to work, I downloaded the MGA9 Alpha and did a DVD install.  The wifi works perfectly.

Still valid mga9 alpha1 live plasma, curl 7.86.0-1

Changing the default to wget would only partially fix the original complaint, as urpmi will still use aria2c to download metadata. You have to explicitly set the downloader to override that. And it's aria2c that causes me the most problems with my nearest mirror (mirrorservice.org). That's not to say I'm against changing the default though.

However, as there's been no progress on this, I've modified the stage 2 installer to let the user select a specific downloader, both when adding additional media at the beginning, and when configuring update media at the end. I've also allowed the user to select a specific mirror when configuring update media, rather than always using the nearest mirror.

Great, thanks
I note to docteam ML for documenting

The downloader selection option seem not yet implemented?
netinstall dated 2022-12-24, 	Stage 2 mdkinst.sqfs 2022-12-21 

 -  Or what is the secret find the settings?

(In reply to Morgan Leijström from comment #40)
> The downloader selection option seem not yet implemented?
> netinstall dated 2022-12-24, 	Stage 2 mdkinst.sqfs 2022-12-21 
> 
>  -  Or what is the secret find the settings?

When using netinstall you aren't offered the option of adding additional media, so the choice of downloader will only appear when adding on-line media after installation is complete. Did you get that far?

If you want to change the downloader used by netinstall itself, that needs to be done in stage 1, not stage 2. That's supported by the "downloader=" boot command line option, as described in bug 28539.

Ah thanks.

No chance here to get netinstall installation that far as to adding additional media as there are too plenty errors with the default downloader...

Stage 2 asks what media to use (i .e tainted etc.)
That would be a good place to ask about downloader :)

Will try "downloader=" option  later.

If we by this bug decide to default to wget, the default of stage 1 IMO should also be wget.

 "downloader=wget" works for me

(In reply to Martin Whitaker from comment #38)
> Changing the default to wget would only partially fix the original
> complaint, as urpmi will still use aria2c to download metadata.
> You have to explicitly set the downloader to override that. 

Shouldn't we then raise separate a bug (depending on this bug) about using same default for metadata as for rpms?

Really confusing otherwise.

> And it's aria2c that causes
> me the most problems with my nearest mirror (mirrorservice.org). 

For me as default the metadata seem to never fail, so here aria2 (and wget) works, but not curl - it fails in the first rpm download/install transaction.
I have not tried different mirrors, using only umu.se.

If anyone have time, It could be interesting to try all mirrors with all three downloadrers and put the result in an table - i winder if it would be the same result for all users or it also depends on some bug in any box on the line or ISP traffic shaping methods.

What method is used to retrieve stage2, and can that be user altered if necessary?

As far as I understand this, we all seem to be barking up the wrong tree. The issue is not the installation procedure or drakconf, but urpmi, right???

CC: (none) => herman.viaene

In a default unaltered system, aria2c is used to download synthesis files, and curl is used to download rpm files.

That discrepancy itself is confusing, and should get fixed.

If I understand correctly, only if user explicitly set default downloader (i.e in GUI) that downloader is used, in all cases.

The above I think is for a separate bug.

----

urpmi can use different downloaders: aria2, curl, wget 

This bug is about what downloader it should use per default - for metadata & rpm files, for all installers, (and for netinstaller stage2 ?)

It seems wget is most reliable for most people.
The current default, curl, seem to be worst, and some have problems with aria2.

(In reply to Morgan Leijström from comment #43)
> For me as default the metadata seem to never fail, so here aria2 (and wget)
> works, but not curl - it fails in the first rpm download/install transaction.
> I have not tried different mirrors, using only umu.se.

Are you using HTTP or FTP? If HTTP, that's probably bug 31339.

http. Added comment in bug 31339.

As Thierry wrote in comment 17 that he wasn't against it, I have made the change. wget will now be preferred over curl if it is installed. aria2 will still be preferred for metadata unless a downloader is explicitly selected. As I don't know why metadata is treated as a special case, I am unwilling to change that aspect.

This can be tested when urpmi-8.129-1.mga9 and drakx-installer-stage2-18.54-2.mga9 reach the mirrors.

From https://ml.mageia.org/l/arc/discuss/2023-01/msg00091.html, it turns out there is a good reason to keep curl as the default. If the user makes changes to /etc/wgetrc, that can conflict with the wget options urpmi uses, and thus prevent urpmi from downloading any files.

curl has an option to skip reading its global configuration fila, and urpmi uses that option. wget does not have such an option.

So I think we'll need to revert the change I made.

(In reply to Martin Whitaker from comment #49)
> curl has an option to skip reading its global configuration fila, and urpmi
> uses that option. wget does not have such an option.

$ LC_ALL=C wget --help | grep config
       --config=FILE               specify config file to use
       --no-config                 do not read any config file

Grrr. I searched for mentions of wgetrc!

"Be careful what you  wish for, you might get it"

I just tried a network HTTP install, and got:

* retrieving   media.cfg
* '/usr/bin/wget' '--force-clobber' '-t' '3' '--retr-symlinks' '--timeout=60' '-P' '/mnt/var/cache/urpmi' 'http://ftgfiles1/mnt/cauldron/x86_64/media/media_info/media.cfg' |
* error: wget failed: exited with 4

* retrieval of [/mnt/var/cache/urpmi/media.cfg] failed

wget rc 4 indicates a network failure, but my in-house mirror is running fine and media.cfg is exactly where the installer is looking for it.

is this bug still valid ?

CC: (none) => mageia

fixed in git

(In reply to Nicolas Lécureuil from comment #54)
> fixed in git

I already changed it to prefer wget in commit dfe0f42d. Your change has no effect - a hash does not have a defined order.

But see comment 48 - aria2 is still preferred when downloading metadata. I don't know why that was done so didn't change it.

Running under strace shows it's using wget, including for the metadata,
but as per comment 49 and 50, it needs to ignore wgetrc if present.

Strace shows it's getting the metadata with ...
execve("/usr/bin/wget", ["/usr/bin/wget", "--force-clobber", "--no-timestamping", "-q", "--retr-symlinks", "--timeout=60", "-P", "/var/cache/urpmi/partial", "https://mirror.math.princeton.edu/pub/mageia/distrib/cauldron/x86_64/me
dia/core/release/media_info/MD5SUM"], 0x1a9ae70 /* 41 vars */) = 0

and later getting an update using wget.

This is with no downloader specified in /etc/urpmi/urpmi.cfg, which
is the case for installs where the user has not specified which downloader
to use.

The option --no-config still needs to be added.

On my systems I often need to change to wget even to set up repositories.
(I guess that is "downloading metadata")

(In reply to Dave Hodgins from comment #56)
> The option --no-config still needs to be added.

I added the --no-timestamping option to avoid the specific problem identified in comment 49. Possibly --no-config would be better, unless there are some options in wgetrc that we would want to be applied when using urpmi.

(In reply to Martin Whitaker from comment #55)
> (In reply to Nicolas Lécureuil from comment #54)
> > fixed in git
> 
> I already changed it to prefer wget in commit dfe0f42d. Your change has no
> effect - a hash does not have a defined order.
> 
> But see comment 48 - aria2 is still preferred when downloading metadata. I
> don't know why that was done so didn't change it.

according to the comment in urpm/download.pm i disagree:

    #- first downloader of @available is the default one
    my $preferred = $available[0];
and available is:

my @available = available_ftp_http_downloaders();

(In reply to Nicolas Lécureuil from comment #59)
> according to the comment in urpm/download.pm i disagree:
> 
>     #- first downloader of @available is the default one
>     my $preferred = $available[0];
> and available is:
> 
> my @available = available_ftp_http_downloaders();

The code for available_ftp_http_downloaders() is

  sub ftp_http_downloaders() { qw(wget curl prozilla aria2) }

  sub available_ftp_http_downloaders() {
      my %binaries = (
          wget => 'wget',
          curl => 'curl',
          prozilla => 'proz',
          aria2 => 'aria2c',
      );
      grep { -x "/usr/bin/$binaries{$_}" || -x "/bin/$binaries{$_}" } ftp_http_downloaders();
  }

ftp_http_downloaders() returns an ordered list of downloaders. That is what I changed in commit dfe0f42d (along with updating the documentation). The %binaries hash is a lookup table to map the downloader names to their binary executable names, to allow the grep statement to check whether each downloader is installed and remove it from the list if not. The grep statement doesn't change the list order. And Perl stores hash entries in a random order (different each time you run it). So changing the order of initialisation of %binaries has no effect.

ok thank you i understand better :-)

(In reply to Martin Whitaker from comment #58)
> (In reply to Dave Hodgins from comment #56)
> > The option --no-config still needs to be added.
> 
> I added the --no-timestamping option to avoid the specific problem
> identified in comment 49. Possibly --no-config would be better, unless there
> are some options in wgetrc that we would want to be applied when using urpmi.

Ok, Thanks. Looking through the other options, there are some that could
cause trouble such as --no-clobber. Other options that could be set such
as --bind-address may be needed for some users, so using --no-timestamping
is better then --no-config.

Regarding comment 52, netinstall doesn't use the testing repos.

Please ask for a freeze push and then close this bug finally. :-)

No need for a freeze push - the --no-timestamping option was added in urpmi-8.130 which was released back in January. The beta2 ISOs use wget by default, except when fetching the initial repository metadata, when aria2 is preferred unless you explicitly select wget, either by the "downloader=" boot command line option or in the installer GUI when adding additional media at the start or adding on-line media at the end.

sorry, but if i understand correctly this is fixed then no ? :-)

I confirm wget was visibly set as default in MCC on a beta 2 Live i tried.
(Changing that option is the first i ever do...)

Regarding using another downloader for initial repository data, my thought is:
Using two downloaders doubles the risk of any of them having trouble with a mirror/network any given user use...

aria2 is used to enable the use of metalink (https://en.wikipedia.org/wiki/Metalink). According to the comments in the code, aria2 is the only downloader that supports that, although from reading the Wikipedia article, that seems no longer true.

It's quite possible that the problems seen when using aria2 are due to the use of metalink, not to the use of aria2 in general.

My knowledge of metalink is limited to what I've just read on that Wikipedia page, so I can't say whether this is a useful feature or not.

Errata 9 updated.

Keeping this open to let anyone to dive into the metalink-aria2 need, comment 65, 66.

 - I think we should not do that now, but for Mageia 10

Any progress on this for mga10?

Whiteboard: MGA8TOO => MGA8TOO, MGA9TOO
Target Milestone: Mageia 9 => Mageia 10