Bug 31587 - curl and wget both started to fail downloading some given url ressource
Summary: curl and wget both started to fail downloading some given url ressource
Status: RESOLVED INVALID
Alias: None
Product: Mageia
Classification: Unclassified
Component: RPM Packages (show other bugs)
Version: Cauldron
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Mageia Bug Squad
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-02-22 22:11 CET by GG HH
Modified: 2023-03-01 21:45 CET (History)
1 user (show)

See Also:
Source RPM: curl-7.88.1-1.mga9.src.rpm
CVE:
Status comment:


Attachments

Description GG HH 2023-02-22 22:11:12 CET
Curl and wget both started to fail downloading some given url ressource on february, 15th.
At first, i thought the issue was on the website side but i now doubt it is.
This site is well known in my country and it's dubious that the issue last so long.
On top of that, it works perfectly fine with firefox.

It looks like there's an issue with the https certificate verification.
It would makes sense as, as long as i understand, firefox uses its own mechanism and do not rely on the OS setup.
Unfortunately downloading a new certificate from curl site does not fix the issue.
I also tried to export from firefox the cerfificate i think it uses but dit not get better results. Still I am not sure to have done it the right way.

Of course, a workaround is to disable the certificate verification with -k parameter.
It's acceptable in my specific context.

Some investigations i made.

$ rpm -q curl wget
curl-7.88.1-1.mga9
wget-1.21.3-2.mga9

$ curl "https://www.boursorama.com/"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$ echo $?
60
$ man curl | grep " 60  "
       60     Peer certificate cannot be authenticated with known CA certificates.

$ LC_MESSAGES=C wget "https://www.boursorama.com/" > /dev/null
--2023-02-22 20:32:12--  https://www.boursorama.com/
Resolving www.boursorama.com (www.boursorama.com)... 193.41.83.19
Connecting to www.boursorama.com (www.boursorama.com)|193.41.83.19|:443... connected.
ERROR: cannot verify www.boursorama.com's certificate, issued by ‘CN=DigiCert SHA2 Extended Validation Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US’:
  Unable to locally verify the issuer's authority.
To connect to www.boursorama.com insecurely, use `--no-check-certificate'.

$ strace -e file wget "https://www.boursorama.com/" 2>&1  > /dev/null |grep ^open.*/etc/ | uniq
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/wgetrc", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 4
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/pki/tls/cert.pem", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/pki/tls/certs", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4

$ strace -e file curl -s "https://www.boursorama.com/" 2>&1  > /dev/null |grep ^open.*/etc/ | uniq
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/gcrypt/hwf.deny", O_RDONLY) = -1 ENOENT (Aucun fichier ou dossier de ce type)
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 4
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 6
openat(AT_FDCWD, "/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 6

Files of interest shared by the two tools are:
/etc/pki/tls/openssl.cnf
/etc/crypto-policies/back-ends/opensslcnf.config
/etc/crypto-policies/back-ends/openssl.config
/etc/pki/tls/cert.pem

It is worth noting that cert.pem and tls-ca-bundle.pem are actually links to the same file
$ ls -al /etc/pki/tls/cert.pem /etc/pki/tls/certs/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-r--r--r-- 1 root root 218025 déc.   6 20:10 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root     57 déc.   6 18:00 /etc/pki/tls/cert.pem -> ../../../etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx 1 root root     60 déc.   6 18:00 /etc/pki/tls/certs/ca-bundle.crt -> ../../../../etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

$ rpm -q -f /etc/pki/tls/openssl.cnf /etc/crypto-policies/back-ends/opensslcnf.config /etc/crypto-policies/back-ends/openssl.config /etc/pki/tls/cert.pem
openssl-3.0.5-4.mga9
crypto-policies-20221110-2.mga9
crypto-policies-20221110-2.mga9
rootcerts-20221130.00-1.mga9
the history of the rpm package did not highlight obvious modifications related to my issue


Trying to make sure the issue is not an outdated cert file
$ LC_MESSAGES=C wget https://curl.se/ca/cacert.pem
--2023-02-22 21:55:06--  https://curl.se/ca/cacert.pem
Resolving curl.se (curl.se)... 2a04:4e42:a00::347, 2a04:4e42:600::347, 2a04:4e42::347, ...
Connecting to curl.se (curl.se)|2a04:4e42:a00::347|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 216583 (212K) [application/x-pem-file]
Saving to: ‘cacert.pem’

2023-02-22 21:55:06 (4,18 MB/s) - ‘cacert.pem’ saved [216583/216583]
$ curl --cacert cacert.pem "https://www.boursorama.com/"
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$ echo $?
60


https://curl.se/docs/sslcerts.html is beyond my skill.
i do think mageia curl is not built to use nss data
$ strace -e file curl -s "https://www.boursorama.com/" 2>&1 | grep ^open.*/etc/|sort
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 4
openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 6
openat(AT_FDCWD, "/etc/gcrypt/hwf.deny", O_RDONLY) = -1 ENOENT (Aucun fichier ou dossier de ce type)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 6
openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3


i am a bit stuck !


thanks
GG HH 2023-02-22 22:11:22 CET

CC: (none) => boulshet

Comment 1 sturmvogel 2023-02-23 06:50:26 CET
The error messages are the same with openSUSE tumbleweed and Fedora 37, if i use the same commands like you. So this is not Mageia specific and this bug seems rather invalid as it is a real website problem...
Comment 2 GG HH 2023-02-23 13:12:53 CET
thanks for your tests

i agree that the issue seems to not be on mageia side
on the other hand, we might also conclude that:
- it is not on the site side as it is working with firefox
- it is probably not on curl and wget side as it would be strange to have both tools start failing simultaneously... 
- it might be on a shared resource used by both which would not be uptodate (mageia or probably upstream)

Candidates could the files used by the two tools in (openssl-3.0.5-4.mga9, crypto-policies-20221110-2.mga9, rootcerts-20221130.00-1.mga9)

would you suggest to close  ?
Comment 3 sturmvogel 2023-02-23 13:27:07 CET
(In reply to GG HH from comment #2)
> - it is not on the site side as it is working with firefox
Firefox neither uses wget nor curl so this argument don't count.

(In reply to GG HH from comment #2)
> Candidates could the files used by the two tools in (openssl-3.0.5-4.mga9,
> crypto-policies-20221110-2.mga9, rootcerts-20221130.00-1.mga9)
Why? openSUSE Tumbleweed and Fedora 37 are using completely different versions and are also affected.

This looks like an issue on the wesbsite certificate...
Comment 4 sturmvogel 2023-02-23 13:40:41 CET
If you try to open the site from your comment 0 with firefox, you even get a warning that there is a security risk with this website as SEC_ERROR_UNKNOWN_ISSUER

So definitly a problem with the website certificate
Comment 5 sturmvogel 2023-02-23 13:59:55 CET
(In reply to sturmvogel from comment #4)
> If you try to open the site from your comment 0 with firefox, you even get a
> warning that there is a security risk with this website as
> SEC_ERROR_UNKNOWN_ISSUER
> 
> So definitly a problem with the website certificate

This warning occurs on openSUSE Tumbleweed MozillaFirefox-110.0-2.1 but not on Mageias ESR firefox-102.8.0-1.mga9
Comment 6 GG HH 2023-02-23 14:16:40 CET
thanks, closing

Ever confirmed: 1 => 0
Status: NEW => UNCONFIRMED

Comment 7 sturmvogel 2023-02-23 14:19:31 CET
inv

Status: UNCONFIRMED => RESOLVED
Resolution: (none) => INVALID

Comment 8 GG HH 2023-03-01 21:45:26 CET
fyi, the web server has been fixed

Note You need to log in before you can comment on or make changes to this bug.