This gives an overview of how good CAs are complying with the various standards like the CA/Browser forum baseline requirements and RFC 5280. It gives an overview of how things change over time.
The data contains about 73 million certificates, but it contains a lot of self-signed certificates. I'm only using those that have a chain to one of the certificates in the Mozilla CA certificate store, which leaves about 8.3 million certificates.
The charts contain 4 lines:
This is a requirement from the CA/B forum baseline requirement in section 9.2.1.
In May 2014 we still see 2.7% of the certificates with this error. We saw improvements in the generation of new certificates over time but then in February and April we saw a significant increase again. The rate for new certificates was around 1% but increased to about 2.5%. So far it looks like May we see a rate of around 0.2% of the new certificates.
This is a requirement from the CA/B forum baseline requirement in section 9.2.5.
In May 2014 we still see 3.2% of the certificates with this error. Since February 2013 we saw a very significant drop in the number of generated certificates with that error. Since September 2013 we have an average of around 0.01%.
This is a requirement from the CA/B forum baseline requirement in section 9.2.4c and 9.2.4d.
In May 2014 we still see 5.6% of the certificates with this error. Since February 2013 we have an average of 0.6% of the generated certificates with this error.
This is a requirement from the CA/B forum baseline requirement in section 9.3.4.
In May 2014 we still see 3.5% of the certificates with this error. Since July 2012 we have an average of around 1.3% of the generate certificates with this error, but it doesn't really seem to be stable.
This is a requirement from the CA/B forum baseline requirement in section 9.1.4.
In May 2014 we still see 3.4% of the certificates with this error, and it seems to be getting worse.
This is a requirement from the CA/B forum baseline requirement in section 9.1.3.
In May 2014 we still see 0.1% of the certificates with this error, and there seems to be progress.
Trying to decode the distinguished name resulted in an error.
In May 2014 we still see 0.22% of the certificates with this error and we seem to be making progress.
The distinguished name contains non-printable control characters.
In May 2014 we see 79 certificates with this error, and we also seem to be making progress with this.
This is a requirement from the CA/B forum baseline requirement in section 9.2.4c or 9.2.4d.
In May 2014 we see 20 the certificates with this error. The error rate seems to be stable.
During the parsing of the certificate there was some error.
The graph currently doesn't show any errors because it currently prevents me from adding those to the database. But I have over 250 certificates that pass the signature check but fail to parse.
This graph used to contain more errors, but those were caused by a bug in the library returning a parsing errror.
This is a requirement from the CA/B forum baseline requirement in section 9.2.4b.
This error hasn't been seen yet.
Somewhere in the distinguished name is a null character.
I have only seen 1 of those where the signature check still passes. It's in the organization's name and shouldn't cause any real issue.
The certificate is not a version 3.
There seems to be 2 such end user certificate.
According to CA/B forum baseline requirement in section 9.4.1 only in special cases can it be longer than 39 months.
About 7% of the new certificates are still valid longer than 39 months. In May 2014 we still see about 10% of the used certificates longer than 39 months. Since we look 5 year (60 month) in the future, all certificates that are currently known that expire then are valid for longer than 39 months.
This is a requirement of CA/B forum baseline requirement in section 9.4.1.
We seem to making progress on this, going from around 4% of the new certificates being longer than 60 months to about 1.5% of those generated in April 2014. We also see the number of used certificates with such a long validity period go from 8% to 5% in May 2014.
CA/B forum baseline requirement in appendix A requires that for certificates with RSA key it should have at least have 2048 bit.
Since January 2009 no certificate was issued using MD5. Since May 2014 we actually stopped seeing MD5 based signatures, but there 3 that are still valid and not revoked with the last one expiring in July 2016.
Some of CA/Browser forum baseline requirements seems to be getting adopted good, but there are still some certificates generated that don't follow the requirements. Other requirements don't seem to get adopted. Those that don't get adopted seem to have to do with things about the CA itself and not with subject of the certificates.
Some of the other requirements have mixed results and some are even getting worse.
There is a surprising amount of long lived certificates. This results in it taking a long time to get those requirements adopted.
There seem to be many certificates that were used at some point, stopped being used but were never revoked.
This is based on the data from scans done by the university of Michigan between June 2012 and January 2014, and on the the Rapid7's sonar data starting from October 2013. It's also using the certificate transparency data provided by Google.
I see a lot of certificates that are valid until 2019. CA/B forum will still allow validity period of up to 39 months, but they may still issue for 60 months (5 year) under some conditions. The graphs have data until where there are enough certificates that expire that month to have the statistics make sense, which is 5 years after the last scan (April 2019). The stats start 5.5 year before the 1st scan (November 2006), since from there they seem to make some sense. There are certificates that are valid longer than that but the numbers start to get so low that it's not useful to get statistics from them.
This are only stats for subscriber certificates and it does not take any errors for other certificates in the chain into account. It also ignores the validity period to find the chain resulting in slightly more certificates that appear to chain to the Mozilla certificate store than that there really are. But the main goal is to find certificates that are generated wrongly, so a few false positives isn't really a problem.
I didn't import all the data yet for the "seen" graphs, at some point there will probably more points on those graphs.
Currently identity and domain validated is only checked using the CA/B forum OID while CAs can use their own OID for it.
The seen graphs are only for those certificates that were valid at the date of the scan. We also see a lot of certificates that aren't valid (anymore).
The CRLs cover more than 99.9% of the valid certificates. There about 2500 different working CRLs URLs.
I'll add more test to this later. Ideas for tests are always welcome.
It's inspired by Microsoft's paper.
11 May 2014: I've been contacting CAs about the missing subject alternative name extension, since I think that's currently the biggest problem. Hopefully we'll see things improve over time.
22 May 2014: Added charts about validity period
25 May 2014: Started filing bugs in the Mozilla bugtracker instead.
28 May 2014: Added chart about certificates with RSA keys shorter than 2048 bit.
29 May 2014: Added the "Valid" lines to the charts.
30 May 2014: Imported the certificate transparency data from Google.
5 June 2014: Imported all CRLs mentioned in the certificates.
14 June 2014: Added charts with signature algorithm.
6 May 2016: I recently made x509lint available, and started adding more checks and so more charts.
---
Contact: Kurt Roeckx <kurt@roeckx.be>