Let's Encrypt DST Root CA X3 Expiration

[raf]

On Sat, Oct 02, 2021 at 04:14:05AM -0500, Ryan Schmidt <ryandesign at macports.org> wrote:

> macports.org and other secure web sites that use Let's Encrypt may
> no longer be accessible to you if you use older versions of macOS
> or older browsers or user agents. For example, the libcurl in macOS
> 10.14 can't talk to many Let's Encrypt web sites now, including
> distfiles.macports.org and packages.macports.org, and MacPorts uses
> macOS libcurl to download things. Safari and many browsers don't use
> libcurl so they may be affected differently.
> 
> Let's Encrypt is a free certificate provider used by macports.org
> and many other web sites to provide https encryption. Certificates
> they issue depend on their "ISRG Root X1" certificate which was
> cross-signed by the "DST Root CA X3" certificate, because DST Root
> CA X3 was more widely accepted by browsers when Let's Encrypt got
> started years ago. Both of these root certificates are included in the
> certificate chain served by web sites that use Let's Encrypt.
> 
> ISRG Root X1 itself has been trusted by browsers for some time
> now and DST Root CA X3 expired a couple days ago on September 30,
> 2021. Apparently in order to provide the widest compatibility,
> certificate chains continue to list the old expired root certificate
> after the new one. The idea as I understand it is that browsers should
> see the ISRG Root X1 certificate, realize that it itself is already
> trusted by the OS or browser, and not even look at the next expired
> DST Root CA X3 certificate in the chain.
> 
> They advertised this root certificate expiration as being a very minor
> situation, but unfortunately it seems that a large portion of Apple
> devices cannot deal with this change. On many Macs, it seems that the
> entire certificate chain is being validated, and the expired extra
> root certificate is causing the connection to be disallowed. What
> alerted me to the problem in the first place was seeing many failed
> builds on our Buildbot system due to fetch failures.
> 
> I'm not certain what to do to address this. On the web servers
> we control, we can apparently remove the expired DST Root CA X3
> certificate from the chain that we send. That will help those
> systems that already trust ISRG Root X1. I'm not sure how far back
> that is. For older systems, we can modify master_sites.tcl and
> archive_sites.tcl to change which OS versions try to access our mirror
> servers via https. None of this necessarily helps our build server be
> able to mirror distfiles in the first place. If you have ideas, let me
> know.
> 
> https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

There is a discussion on the LetsEncrypt community site
with instructions for installing the ISRG Root X1
certificate on older versions of macOS:

  https://community.letsencrypt.org/t/connection-errors-on-apple-devices/161107/16

Here are instructions for 10.10 and 10.11:

  https://community.letsencrypt.org/t/connection-errors-on-apple-devices/161107/25

Here's another approach that worked on 10.9.5 and
10.11.6 (i.e., override the expiry by always trusting
DST Root CA X3:

  https://community.letsencrypt.org/t/connection-errors-on-apple-devices/161107/28

Here's my approach for 10.6.8 because the above didn't
work (i.e., export root certificates from a later macOS
and import them):

  https://community.letsencrypt.org/t/connection-errors-on-apple-devices/161107/36

But these are all client-side fixes. The only
server-side fix seems to be to change to a different
certificate authority. I didn't see any mention of
removing the DST Root CA X3 certificate from the chain.
That would probably only work from macOS 10.12 onwards.
ISRG Root X1 is only trusted by macOS since 10.12.
Earlier than that, it needs to be added.

The rest is a bit rambly. It might be best to just
consult the LetsEncrypt community forum above.

I should mention that I didn't notice any problems with
macports on 10.6.8. curl has its own root certificates
and they are fine. And I was able to do an upgrade. But
I might have already installed the ISRG Root X1
certificate at least into my "local" keychain before
trying it (or maybe into the "System" and "System
Roots" keychains).

However, I still don't think that it's entirely OK.
Firefox on 10.6.8 can access macports.org with no
problem (but it certainly wasn't accessing my
LetsEncrypt-certified sites beforehand), but Sarafi
tells me that it can't verify macports.org's identity,
but it still lets me access it. If I quit and restart
Safari, and visit macports.org, it does the same thing.
Firefox uses its own root certificates. Safari uses the
system ones. But I definitely have ISRG Root X1 in both
the "System" and "System Roots" keychains. So I don't
know why Safari has a problem. In Keychain Access, it's
marked as "Always Trust" but it also says "trusted for
macports.org" (I don't know why it says that). That's
confusing. But in Safari, when viewing the certificate,
there was a checkbox labelled "Always trust". After
checking it, quitting and restarting Safari, it visited
macports.org without complaint.

I should also mention that I can't find DST Root CA X3
in my 10.6.8 keychains. So that's wierd. Otherwise, I
probably should have set it to always trust (not that
having ISRG Root X1 set to always trust convinced
Safari to trust it immediately - I had to tell Safari
itself to trust it as well).

cheers,
raf