If you run c_rehash on Mageia you do get hash symlinks in /etc/pki/tls/certs, so aside from the ca bundle being named differently, it does appear that both directories are in fact used the same way.  The compatibility symlink is probably still a good idea.

CC: (none) => luigiwalser, oe

More comments on your RedHat bug:

We already have /etc/pki/tls/certs which is a mix of having the bundle file and individual certs.  You can run c_rehash and it makes hash symlinks there (including one to ca-bundle.crt) and I don't see that that causes any problems.

I'm not sure in what cases running c_rehash is actually necessary.  It certainly doesn't seem to be in the default case.  *Maybe* it is if you add your own certs there in some cases, but maybe it isn't.

There are things that use individual certs in that directory, like the ldap libs.  If you're using LDAP authentication with your LDAP server using your own internal CA signed cert and use tls_reqcert demand in /etc/openldap/ldap.conf, you can put your internal CA cert in /etc/pki/tls/certs and that makes that work.

We (Mageia) would like to use Fedora's ca-certificates instead of our rootcerts package to ease future maintenance (Bug 11398), but it's unclear exactly what are all of the steps that would be needed to make that change.  Making a crazy hackjob out of the nss package like Fedora has doesn't look desirable, and then there's all of the Fedora-specific p11* stuff that would have to be sorted out as to what needs to be imported and how it needs to be used.

So, you're right that c_rehash will operate on OpenSSL's 'default certificate directory' if no directory is specified - nice catch, somehow I hadn't noticed that. But it truly means 'operate on': it reads cert files from that directory, and writes hash symlinks to it...it doesn't find the certs from elsewhere or from the bundle file and write hashes for those. So it's only useful if you actually have some individual certificate files in OPENSSLDIR/certs - which didn't *seem* to be the case on Mageia to me, but of course you'd know better than me.

In Fedora we don't ship individual trusted CA certificate files in /etc/pki/tls/certs (or have any script or anything that puts them there); maybe there's something in Mageia I don't know about / forgot about which does so?

IIUC, a hash symlink to ca-bundle.crt is not really of much practical use. The hash itself identifies the certificate that should be in the file, and when it's operating on a hash directory, OpenSSL will only use the hash file as a source for the specific certificate which matches the hash. I *think* all it will actually achieve is to trust the very first or very last (not sure which) certificate listed in the bundle file.

"There are things that use individual certs in that directory, like the ldap libs.  If you're using LDAP authentication with your LDAP server using your own internal CA signed cert and use tls_reqcert demand in /etc/openldap/ldap.conf, you can put your internal CA cert in /etc/pki/tls/certs and that makes that work."

So, that's another point of confusion: those are really two completely different uses of the directory. Fedora has the same confusion - we similarly have packages which generate self-signed server certs in that directory, or have their default configuration file expect their server cert to be in that directory.

That's a *completely different use case*, though - we're completely outside the realm of the system default trust store at that point. In a way it'd be much clearer if we'd all decided to use two different directories for these two different purposes.

As far as I'm aware, you'd never need to use c_rehash on a server cert for any reason.

"I'm not sure in what cases running c_rehash is actually necessary.  It certainly doesn't seem to be in the default case.  *Maybe* it is if you add your own certs there in some cases, but maybe it isn't."

For the purpose of trusting certificates (as opposed to using them as server certificates), I believe it would in fact work to place individual files in /etc/pki/tls/certs on Fedora or Mageia and run c_rehash manually, yeah. This is because the 'OPENSSLDIR' is /etc/pki/tls , so /etc/pki/tls/certs is the 'default CA trust path', and OpenSSL will try to read *both* the 'default CA trust file' *and* the 'default CA trust path' when you tell it to use the system default trust store. So yeah, I think if you stick a cert in there and run c_rehash, things that use the OpenSSL default trust store will trust it, along with all the things in the bundle file.

The point of the /etc/ssl/certs directory is to be compatible with stuff that (incorrectly) assumes that's some kind of 'standard' path and just uses it - it doesn't look for a bundle file in the Fedora/Mageia location, it doesn't ask OpenSSL to use its own 'default' locations, it just hardcodes /etc/ssl/certs .

Right now, on both Fedora and Mageia, such software won't work well because it won't get the system default trusts. If someone were to manually populate /etc/pki/tls/certs with trusted certificates and run c_rehash then that would be respected, but the default system trusted certs won't be used.

"but it's unclear exactly what are all of the steps that would be needed to make that change"

It's somewhat...in-depth, yeah. :) I'm not sure the 'crazy hackjob' comment is entirely justified, but it is a fairly big change indeed. OpenSUSE has adopted p11-kit since last summer with their own little script for writing out bundles and hash dirs instead of Fedora's 'update-ca-trust' - SUSE's is https://github.com/openSUSE/ca-certificates - and Arch has adopted the entire Fedora system: https://bugs.archlinux.org/task/41909 . The best person to ask for advice on switching to Fedora's system would be Stef Walter, I think. So far as fresh installs go, I believe Mageia could literally just dump the entire Fedora system into place and it would work; I think the main questions would be about transition, whether Mageia has some existing system for 'put certificates here and do X and they will be trusted' that is documented and/or commonly used, and how to ensure a smooth transition from that to the new system.

Yeah, so for a start c_rehash only runs on .pem files, so it won't run on ca-bundle.crt by default. If I copy it to ca-bundle.pem and run c_rehash I get a hash:

[adamw@adam PHPMailer-adamwill (cert-validate *%)]$ ls -l /etc/ssl/certs/
total 1756
-r--r--r--. 1 root root 257045 Jan 13 15:27 ca-bundle.pem
lrwxrwxrwx. 1 root root     13 Jan 13 15:27 eacdeb40.0 -> ca-bundle.pem

but as I thought, it doesn't really 'work', as in, if you use /etc/ssl/certs as a CApath with the bundle file and symlink present, the certificates listed in the bundle aren't trusted, certificate verifications fail. CApath validation really only works with hashed individual certificate files.

tested on Mageia 5 Beta 1, seems you've patched your c_rehash to run on .crt files, but I think it still won't actually make using /etc/ssl/certs (or /etc/pki/tls/certs) as a CApath trust the system trust bundle, that's just not how the hashing thing works.

I get symlinks for .pem or .crt files with c_rehash.  I guess for the bundle file the symlink it makes might not be useful.  Ultimately whether this works or not is up to how the application is looking for the certs.  Is it looking for the bundle as a hardcoded path, is it looking for an individual cert with a hardcoded name in a config somewhere, or is it looking for a hash.

Mageia has scripts that generate self-signed certs for services that use TLS when you install those packages (like apache-mod_ssl, postfix, squid) and it stores their public certificates in /etc/pki/tls/certs, just as Fedora does.  Otherwise, packages don't generally ship individual certs in that directory (rsyslog-gnutls does, not sure what that's about).  So yes, for this sense, it has nothing to do with the trust, it's the certs that are being served to remote clients.  However, for my example use case for LDAP, this is not what I was referring to.  For that case, you can put a CA certificate that the remote LDAP server that you're using for authentication used to sign its certificate, and its presence in the certs directory will add it to the trust such that tls_reqcert can verify that the remote LDAP SSL cert is valid.

As for the crazy hackjob comment, I think it's fair.  Those changes have done the exact opposite of ease the maintenance burden for the nss package, and have also made it a lot more complicated.

We do have a p11-kit package, but AFAIK that's a fairly standard thing that everyone has.  I was referring to additional stuff Fedora has for managing the trust stores.  I'm not sure what it's all called.

Is c_rehash not generating symlinks for all of the certs in a bundle maybe something that's a bug that should be fixed upstream in OpenSSL?  Or should software not be relying on these symlinks?

"Is c_rehash not generating symlinks for all of the certs in a bundle maybe something that's a bug that should be fixed upstream in OpenSSL?"

Well, no, not really, because it's just...not designed to work that way. How it operates is clearly documented: a CApath contains individual certificate files, not bundles. That's the design of the software. If you want to use a bundle file, you just tell OpenSSL that's what you want to use (CAfile).

The whole point of the hashing mechanism is to speed up lookups of individual certificate files. From 'man x509':

       -subject_hash
           outputs the "hash" of the certificate subject name. This is used in OpenSSL to form an index to allow
           certificates in a directory to be looked up by subject name.

so it's just meaningless to hash a bundle file, it wouldn't achieve anything.

Debian briefly patched c_rehash to operate on bundle files, but they botched it and it broke stuff (they only changed c_rehash, they didn't actually check that openssl could find the certificate from the hash symlink to the bundle file when operating in CApath mode, and it can't: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/854927 ). Debian backed that patch back out shortly after.

In case I'm not being clear, recap: you can load two different types of trusted CA stores into OpenSSL, a "CApath" or a "CAfile", as documented at the top of 'man verify'. When used as a library, the function SSL_CTX_load_verify_locations() takes CApath or CAfile arguments (or, in fact, both, in which case it'll load from both). When asked to use its 'default locations' it will try to load OPENSSLDIR/certs as a 'CApath' and OPENSSLDIR/cert.pem as a 'CAfile'.

Fedora and Mageia provide OPENSSLDIR/cert.pem (for them, OPENSSLDIR is /etc/pki/tls, and they have a /etc/pki/tls/cert.pem symlink); they have an OPENSSLDIR/certs directory, but they don't populate it in the expected format, so OOTB OpenSSL will not load anything from it.

Debian provides OPENSSLDIR/certs; it does not provide an OPENSSLDIR/cert.pem. For Debian OPENSSLDIR is /etc/ssl, and it has a hashed /etc/ssl/certs. It does not ship a /etc/ssl/cert.pem .

Software which knows about the different locations in Debian and Fedora/Mageia and explicitly tries to load each one will work fine in both - but does not need /etc/ssl/certs on Fedora/Mageia.

Software which just asks OpenSSL for its 'default' locations will work fine on both - but does not need /etc/ssl/certs on Fedora/Mageia.

Software which doesn't do things properly, but just expects /etc/ssl/certs to exist and be a CApath-style hashed directory full of individual trusted certificate files with hash symlinks (because it is on Debian), will *not* work on Fedora/Mageia because we don't provide /etc/ssl/certs in the right format.

Software which doesn't do things properly, but just expects /etc/ssl/certs/ca-certificates.crt to exist (because it does on Debian), will *not* work on Fedora/Mageia because we don't provide a /etc/ssl/certs/ca-certificates.crt file.

Ergo, our current /etc/ssl/certs is either 99.9% or 100% useless (depending on whether having a /etc/ssl/certs/ca-bundle.crt somehow once helped Flash, and whether it still does, which doesn't seem to be the case).

If we just removed our /etc/ssl/certs symlinks entirely, nothing catastrophic would happen, nothing would *break*. But if we actually want to be compatible with bad software that just expects either /etc/ssl/certs to be a CApath or /etc/ssl/certs/ca-certificates.crt to be a CAfile - as is the case on Debian - then we should change our /etc/ssl/certs to actually work that way.

"We do have a p11-kit package, but AFAIK that's a fairly standard thing that everyone has.  I was referring to additional stuff Fedora has for managing the trust stores.  I'm not sure what it's all called."

The only thing Fedora really needs on top of p11-kit is a spectacularly simple script that calls it a few times:

http://pkgs.fedoraproject.org/cgit/ca-certificates.git/tree/update-ca-trust

All the heavy lifting bits are actually in p11-kit. But of course just having a p11-kit *package* doesn't do much if you don't hook it into your rootcerts package and whatever, it'll just lie there.

I think you could use the p11-kit stuff to generate trust bundles without adopting all the NSS stuff, if you wanted, but again stefw would be a better person to ask.

Thanks for the explanation.

So, there's nothing coming from packages that are in OPENSSLDIR/certs that need to have hash symlinks for them, because all we have there is a bundle file for which the hash symlink isn't helpful and public certificates for services which are not part of the trust store so nothing is looking for their hash symlinks.

Saying that Fedora/Mageia don't populate OPENSSLDIR/certs in the expected format isn't really fair, as they aren't putting anything in there that needs to have hash symlinks in the first place.  If any sysadmins want to put their own certs in that directory, it's their responsibility to run c_rehash.

Software which expects the entire trust store to be a hashed directory is wrong.

We have patched many things to explicitly load our bundle file, but perhaps it'd be better to have them use openssl, that way they could use the bundle and the directory as the trust, which would make it easier to add things to it.  However, this would be more complicated and would generally take adding additional code to such packages rather than just patching a file path for a bundle they're already programmed to use.

Perhaps we could ship a symlink of /etc/pki/tls/certs/ca-certificates.crt > ca-bundle.crt, which would help the software using the hardcoded Debian bundle file (as long as we kept the /etc/ssl/certs -> /etc/pki/tls/certs link in place).  Of course, software doing this is just as wrong as software expecting just a hash store.

So you say Debian has this bundle file (ca-certificates.crt) as well as a individual cert store with symlinks, so do they duplicate all of the individual certs from the bundle into the directory?  Is that what you mean by "change our /etc/ssl/certs to actually work that way" ?

(In reply to Adam Williamson from comment #9)
> I think you could use the p11-kit stuff to generate trust bundles without
> adopting all the NSS stuff, if you wanted, but again stefw would be a better
> person to ask.

I guess the purpose of the NSS stuff is that under normal circumstances, nss compiles the certs from the bundle into itself when it builds, so you can't dynamically update the trust store after the fact, without patching nss.

"Saying that Fedora/Mageia don't populate OPENSSLDIR/certs in the expected format isn't really fair"

Sure, I just meant "Debian's expected format". This whole thing is 'extra credit' stuff, to try and be helpful to bad software. None of it's critical / required. Debian doesn't ship a /etc/pki/tls/certs either. :P I just felt that it's odd for both Fedora and Mageia to go to the trouble of shipping a /etc/ssl/certs, but for it not actually to be any real *use*. To be super duper clear, I'm not advocating we make /etc/ssl/certs Debian compatible *and then go around dropping all our package patches to use our canonical bundle location and say 'the compatibility symlink is the Way Forward!'*. Nope, definitely not. It's just a small last-resort safety net.

"Software which expects the entire trust store to be a hashed directory is wrong."

Basically, yeah. Any software which expects one particular type of trust store and/or any particular location is wrong, or at least sub-optimal. As you say...

"We have patched many things to explicitly load our bundle file, but perhaps it'd be better to have them use openssl"

the *optimal* design is for the software to just let OpenSSL handle it. In an ideal world, we would have every damn app in the world just use its SSL library's defaults. It's not an ideal world, but that's the goal. And yes, if you can't manage that, at least patch them to use the distro's 'canonical' trust store location. I tend to consider that to be /etc/pki/tls/ca-bundle.crt for Fedora; I'm guessing Mageia would say the same.

The main advantage of patching them to use the library default rather than a particular path is just that it's way more robust in general. It helps out all distributions / platforms, and it also in theory allows you to change OpenSSL's default in future without having to go back and change all the patches. It's just clearly the sensible thing to do. Usually, though, you want to write code that tries to use the library default then falls back on location guessing and/or configuration, because apparently there *are* cases out there where you somehow get an OpenSSL library that doesn't have a valid default store, some bundled copy or badly compiled Windows one or something.

"However, this would be more complicated and would generally take adding additional code to such packages rather than just patching a file path for a bundle they're already programmed to use."

Sometimes, yeah. The worst case is if the upstream code doesn't know about CAfiles; I've come across a few which are written such that they can only use CApaths. In that case you're kind of stuck writing a slightly more substantial patch to at least teach them to use CAfiles too, and in that case you may as well go the extra mile and make it try to use the default trust store when possible too, I've written a few patches along those lines - e.g. https://projects.duckcorp.org/issues/350 , https://github.com/PHPMailer/PHPMailer/pull/339 .

"Perhaps we could ship a symlink of /etc/pki/tls/certs/ca-certificates.crt > ca-bundle.crt, which would help the software using the hardcoded Debian bundle file (as long as we kept the /etc/ssl/certs -> /etc/pki/tls/certs link in place).  Of course, software doing this is just as wrong as software expecting just a hash store."

Yep - that's the easy part of 'being more Debian compatible', it's just one extra symlink, and it at least helps anything which just blindly looks for that file (I have come across a few which did this at least at some point). Making /etc/ssl/certs also be a hashed directory is a bit more work.

"So you say Debian has this bundle file (ca-certificates.crt) as well as a individual cert store with symlinks, so do they duplicate all of the individual certs from the bundle into the directory?  Is that what you mean by "change our /etc/ssl/certs to actually work that way" ?"

More or less. Debian has a system for generating trust stores from source files that's somewhat like p11-kit's (though a bit less capable): see http://man.he.net/man8/update-ca-certificates . Basically they have certificate files in /usr/share/ca-certificates and /usr/local/share/ca-certificates , and a config file - /etc/ca-certificates.conf - which expresses trust in the certificates (so it can say things like 'certificate foo is trusted, certificate bar is blacklisted'). The update-ca-certificates command populates /etc/ssl/certs based on that source information: whenever it's run it generates the /etc/ssl/certs/ca-certificates.crt bundle, sticks individual certificate files into /etc/ssl/certs, and runs c_rehash.

p11-kit broadly achieves the same thing, along with lots of other things - we don't just ship ca-bundle.crt as a file directly in our ca-certificates package any more. ca-certificates contains a source trust bundle; that 'update-ca-trust' file I mentioned earlier takes care of generating the final trust bundles (in multiple formats) from it and any site configuration found in p11-kit's various source locations. /etc/pki/tls/certs/ca-bundle.crt is now a symlink to one of the files output by 'update-ca-trust'.

If Mageia didn't yet implement some kind of similar mechanism for generating different format trust bundles from source files, of course it'd be harder to provide a hashed /etc/ssl/certs - but as you say, you could at least do half the job quite easily, and provide a /etc/ssl/certs/ca-certificates.crt symlink.

I forgot to say earlier that I also concluded from your previous post that we should remove our patch to have c_reshash generate symlinks for .crt files.

So for Mageia, the way forward would be to either go closer to the Debian model (by having the ca-certificates.crt symlink and having something break out the individual certs and hash them) or the Fedora model, which is more capable and flexible and secure and is in Fedora, which is much easier for us to sync with in general (and we do this a lot anyway).

If we don't go full Debian model, just putting the ca-certificates.crt symlink might help some software, but certainly anything packaged should be patched to use ca-bundle.crt, and anything third party looking for ca-certificates.crt just needs to be fixed, so it's debatable the value of supporting that.  If we don't do that or go full Debian model (maybe using SuSE's script), then maybe the /etc/ssl/certs symlink is pointless after all.  That would apply to Fedora as well.

I suppose the only remaining possible uses for the /etc/ssl/certs symlink would be that it's a little easier to find than /etc/pki/tls/certs if you do need to deal with it, whether you're poking around in /etc blindly, or Googling for this.

"and anything third party looking for ca-certificates.crt just needs to be fixed, so it's debatable the value of supporting that"

I kinda feel like it's worth doing just because it's so easy and, you know, crap happens - there's so much software out there, it's just damn hard to fix it all. if we can maybe make things work a bit better almost for free, why not.

the other thing I forgot to mention is there really ought to be a README somewhere explaining that /etc/ssl/certs is a compatibility location and nothing in the distro should use it; anything that does ought to be patched to use the canonical location.

Assigning to David Walser, who planned to fix this, according to 
https://bugs.mageia.org/show_bug.cgi?id=11398#c8

CC: (none) => marja11
See Also: (none) => https://bugs.mageia.org/show_bug.cgi?id=11398
Assignee: bugsquad => luigiwalser

Fedora still hasn't fixed this, and in fact has recently added more "cross-distro compatibility symlinks":
https://src.fedoraproject.org/rpms/ca-certificates/c/9bd23da27ffac43abfb42ae5c982d2320d6cd8fd?branch=rawhide

we have largely re-synced our rootcerts package with Fedora's ca-certificates a year ago, so we're using all their p11-kit stuff too.  I guess we should sync in some of their more recent changes and close this.