Item — HN Reader

I've added "Could you prevent or monitor this kind of attack?" section.

There are several indications which could be used to discover the attack from day 1:

* All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names

* Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods

* Monitor SSL/TLS certificate changes on all your services using external service

* Monitor MAC address of default gateway for changes

5 comments

Nice addition. Me personally being the paranoid type, I don't trust the transparency log for monitoring but rather for writing up the root cause analysis. Reason being is that if someone can legally compel Hetzner and/or Linode and/or LetsEncrypt to do or not do something then the same entity can compel the certificate transparency site to ignore something. But you covered what I would do and that is to have multiple nodes doing active monitoring of TLS changes using an external service. That service being openssl s_client in my case.

In the case of Jabber it might be interesting to add some monitoring in the application that uses a cryptographic signed payload to list all the valid fingerprints and send an alert message to the server if something odd is happening like public key pinning but without a hard-fail. That list could be pre-loaded with new certificate fingerprints prior to being deployed. If the oddness is confirmed then perhaps add some way to tell the clients that certificate is likely forged. That way both server operator and client are aware that something evil this way comes.

[Edit] - Looks like based on the comment from MattJ100 that some Jabber servers and some clients can already do something like this.

> if someone can legally compel Hetzner and/or Linode and/or LetsEncrypt to do or not do something then the same entity can compel the certificate transparency site to ignore something

the major certificate transparency logs are operated by several independent global companies, Apple and Google for LE. it's unlikely that they will agree to forge their global CT logs for a single government. more importantly, SCTs allow cryptographic proof of anyone lying, making such an action very dangerous for their continued participation in the WebPKI ecosystem.

This sounds like the methodology for blockchain and multiple ledgers. Ultimately however crt.sh is hosted somewhere and while there may be multiple controllers that have access to logs on the front-end, someone hosting that site could be compelled or blackmailed into tinkering with the levers behind the scene to exclude activity on a domain. I'm not suggesting that is what is happening, just that it could and Apple, Google and others would have plausible deniability. On the other hand having active probes distributed around the world on multiple ISP's looking for fingerprint changes would be much harder to hide though more expensive to operate along similar lines to archive.is or using distributed Nagios NPRE agents or using ThousandEyes probes.

> Ultimately however crt.sh is hosted somewhere

So is it possible to run my own copy of crt.sh? How demanding it is (e.g. data size)?

The crt.sh code is all open on GitHub, so can be hosted yourself. Last I checked a few months ago, the main ‘certificates’ table and indexes etc was close to 20TB, and there’s more than just that. It’s big, but has everything. A slimmed down database of just some lightweight info on the certs and issuer (notBefore, not after, SANs, serials etc) runs maybe a TB in bigquery.

A log monitor? Or specifically the crt.sh codebase?

Yes, the logs are public and I ran equivalent monitoring for a previous employer before the pandemic. You will need to consume records at the rate they're created to keep up, Let's Encrypt have stats you can look at, if you can cope with twice their typical daily throughput you'll be fine on average, but peaks will swamp you temporarily so design for that.

You can choose whether to store everything, or just stuff you consider interesting, and you can choose whether to care forever or only until expiry (so 398 days)

If you want everything (full certificates), indefinitely, that's a lot of data. Um, several terabytes per year maybe?

this is a very very silly take. Apple and Google have armies of lawyers who spend all day telling cops with subpoenas from all around the world to fuck off.

The problem is that Let's Encrypt issued a fake certificate without proper validation.

UPD: I was initially wrong. It looks like many CAs will issue a valid certificate to attacker capable of doing MitM.

why do you keep making this incorrect claim?

it was a DV cert, and it was considered validated because someone was MITMing the traffic. that would have worked against any CA, nothing at all to do with Let's Encrypt.

Corrected the comment to reflect that SSL cert infrastructure is ridiculously easy to hack.

I mentioned this in the operators channel just now, but also worth noting is that channel binding (e.g. SCRAM-*-PLUS in combination with RFC 9266) also mitigates this attack. Essentially the connecting client is able to detect the mismatch between the certificate or handshake the server thinks it is presenting, and the one the client actually sees.

ejabberd and Prosody support it, and a number of clients too.

> I mentioned this in the operators channel just now, but also worth noting is that channel binding (e.g. SCRAM-*-PLUS in combination with RFC 9266) also mitigates this attack.

Doesn't that require the server to persistently store user passwords in plaintext?

Not at all. If you mean SCRAM, it was actually pretty novel when it was introduced because it allows the server to store a hash, the client to store a hash, and only a hash is exchanged over the wire.

SCRAM also allows the client to verify that the server possesses the password (or a hash of it), so a MITM that just says "yep, your credentials are correct - go ahead!" can be detected.

Channel binding is an addition to SCRAM (it's usable outside of SCRAM too) that allows securely checking the TLS stream as well. Specifically it allows verification that the TLS stream is terminated by the same entity that has (a hash of) your password.

no, scram is all about not exchanging and not storing plain text passwords. the plus variants add tls channel binding.

DANE (with DNSSEC) may be an option to look at too, with public keys in DNS. though probably client support isn't universal (i have no (recent) experience using jabber). the CA was one weak point here.

checking for new public keys in certificates when keeping private keys between renewals would also help.

i suppose any storage and possibly ram should also be considered compromised? e.g. private keys stored on disk without encryption, but also by being read into memory. especially on the vm.

DNSSEC operators can be strong-armed the same way. You will also lose out on transparency logs.

In the end law enforcement can also walk up to the machine/hypervisor and steal/monitor interesting things from that as well. This is funnily the exact evil maid threat scenario so many (especially Linux people) find unrealistic.

you could run your own dnssec servers (not too hard). and monitor the DS records for your domain at the TLD. have there been any reported cases of law enforcement changing DNSSEC (e.g. adding a DS record)?

i agree there's probably no way to keep a machine hosted at a company secure from law enforcement. also why i suggested storage and anything in ram on the machine can be considered compromised. this attack (swapping out network connection for a while to get a certificate through let's encrypt) was probably easiest/least intrusive. if it wasn't an option, the next easiest option would be taken. perhaps the options that are harder to execute are more likely to be detected, or less likely to be worthwhile.

Law enforcement routinely manipulates the DNS; famously so.

> have there been any reported cases of law enforcement changing DNSSEC (e.g. adding a DS record)?

It's not like anybody is watching or using DNSSEC like that. Also at best you might be able to detect a change but it won't prevent the attack and neither will it leave a long-term mark like CT would.

>keep a machine hosted at a company secure from law enforcement

Your own hardware + continuous video monitoring is probably good enough. The idea is not to keep it secure, but to know when a breach has happened.

> DNSSEC operators can be strong-armed the same way. You will also lose out on transparency logs.

Kepping DNS registry, CA, and hosting in different jurisdictions could be a noticeable improvement...

You don’t need DANE to thwart this particular attack: DNSSEC and CAA could have prevented the MITM certificates from being issued in the way they were. If the xmpp.ru and jabber.ru domains used DNSSEC and CAA, it would not have been enough for the MITM to strong-arm Linode and Hetzner: they would also have to strong-arm the domains’ DNS providers. I don’t think the attackers could have done it stealthily (unnoticed for 3 months) without the co-operation of the .ru registry and a MITM attack on all of the Let’s Encrypt validation vantage points.

I am one of the co-authors of the DANE SRV RFC https://www.rfc-editor.org/rfc/rfc7673 which is what XMPP would use. I don’t follow XMPP development so I don’t know if it has been deployed. I would like it if DANE were more widely used, but it’s not pertinent to this attack.

I think it's a great example why DNSSEC would be bad: at least here, we had transparency logs and there was a simple method to get an attack notification. With DNSSEC, there would be no such thing.

I disagree. Forging DNSSEC requires LE to strong arm the domain registrar to modify the DS keys per domain which would also require them pointing to an adversary DNS server to answer those requests with a valid signature. That action would definitely be noticed. Without DNSSEC you are back to square 1 of unsigned DNS that can be modified in-flight, a much worse situation. I personally believe we should be advocating for more DNSSEC, not less.

What I think you're advocating for is that DNSSEC should have it's own transparency log that shows when domains update or cycle their DNSSEC KSK/ZSKs which is a great idea, we would just need to get all the domain registrars on board as well which as we can already see with the transparency project, you can't get buy in from everyone.

Well, with transparency project we _did_ get buy in from everyone because browsers (like Mozilla and Google) forced it. And there are severe penalties for not obeying transparency log, and well as technical measures to enforce it (for example certs with no CT log would not work in Chrome).

This of course only works because there is competition. If TrustCor fails to behave, then this CA is dropped. Customers complain, but not very strongly - there are plenty of CAs around, it's annoying to have to switch to another one but still eminently doable. And TrustCor is out of business completely, all their investment is permanently gone.

With DNSSEC.. what do you do if you domain registrar does not play by the rules, say is caught double-issuing DS keys?

Answer is: likely nothing. Most orgs are not going to move .com to .io because of that -- there links, SEO, even things like business cards. People would complain, _maybe_ registrar would get a fine for 0.0001% of their annual income (but I would not count on this). And that's it, the system will stay insecure and they will continue to double-issue certificates.

The separation of CA from registrar is a _feature_, and a very helpful one. Competition is good, and we should not make our existing natural monopolies even more useful.

> LE to strong arm the domain registrar to modify the DS keys per domain

It is important to note that when a law enforcement with this capability would just legally take ownership of the domain instead. If the domain is under control getting domain validated certificates is trivial. There are many legal precedents with domain takedowns.

It is equally important to note that only TLDs under LE jurisdiction is ever an issue. It is not clear whether this specific attack under .ru had been ever remotely possible with any type of authenticated domains.

good point. had me looking for a transparency log for dnssec, arriving at https://www.huque.com/2014/07/30/dnssec-key-trans.html. it seems there hasn't been activity on that topic in years.

an attack would involve a tld operator being compelled to cooperate, or their signing keys being compromised (and then probably adding a DS record for your domain). if that were found out, it would be bad news for trust in dnssec. have there been any known case of this happening? it would be hard (impossible?) to fully detect without transparency log (like for ca certificates).

There's also a DNS CAA record that sets up a whitelist of authorities that can issue certificates for the domain.

At least for Let’s Encrypt, you can also use CAA to require a specific account as well or validation method. This could prevent your hosting provider from issuing on your behalf if they don’t control DNS.

https://letsencrypt.org/docs/caa/#the-accounturi-parameter

I use those as well. I am curious if there has ever been a case where LetsEncrypt were legally compelled to create exemptions for domains as in ignoring CAA and not logging to the transparency log or to just outright issue certs to an agency for specific names. CAA account and method restrictions would be negated at that point.

The more I think about it I would wager that Linode and Hetzner were just law enforcement having good taste in VPS providers. It's more likely to me that LE was the compelled target.

Let’s Encrypt publishes legal transparency reports on what law enforcement asks us.

https://letsencrypt.org/documents/ISRG-Legal-Transparency-Re...

We’ve never been ordered to issue a certificate.

While I am a Let’s Encrypt employee, this message is not an official communication of Let’s Encrypt and shouldn’t be interpreted as such.

I appreciate that. I assume that would not cover and NSL which is also a gag order but nice that you have something for all other above board legal requests. I see you have a column for NSL but have no idea how that could ever increment without violating the NSL.

For some background I've spent a good deal of time with lawyers and C-Levels playing devils advocate trying to find a way to indirectly notify customers but it's just not legally possible in the United States of America and there is nobody that would risk violating one. People here on HN often bring up canaries but they are not compatible with a NSL.

Other entities that publish transparency reports and that have received NSLs have reported them in bucketed ranges. I forgot the exact granularity.

It does not appear that NSLs compel arbitrary actions or require the recipient to actively lie about having received one.

they could go the extra mile there and put a canary page about not having accepted any gag orders (which they could remove when they do)

It's a scary world we're living in...

Let's encrypt (and other CAs) should not allow validation using unsecure schemes like HTTP. Especially for sites that have valid certificate from other CA.