Before you ship a "security mitigation" ...

Hey everybody,

During my years doing vulnerability research and my time in Project Zero, I frequently encountered proposals for new security mitigations. Some of these were great, some of these - were not so great.

The reality is that most mitigations or "hardening" features will impose a tax on someone, somewhere, and quite possibly a heavy one. Many security folks do not have a lot of experience running reliable infrastructure, and some of their solutions can break things in production in rather cumbersome ways.

To make things worse, many mitigations are proposed and implemented with very handwavy and non-scientific justifications - "it makes attacks harder", it "raises the bar", etc., making it difficult or impossible for third parties to understand the design and trade-offs considered during the design.

Over the years, I have complained about this repeatedly, not least in this Twitter thread:

https://twitter.com/halvarflake/status/1156815950873804800

This blog post is really just a recapitulation of the Twitter thread:

Below are rules I wrote for a good mitigation a while ago: “Before you ship a mitigation...

1. Have a design doc for a mitigation with clear claims of what it intends to achieve. This should ideally be something like "make it impossible to achieve reliable exploitation of bugs like CVE1, CVE2, CVE3", or similar; claims like "make it harder" are difficult to quantify. If you can't avoid such statements, quantify them: "Make sure that development of an exploit for bugs like CVE4, CVE5 takes more than N months".
   
2. Pick a few historical bugs (ideally from the design doc) and involve someone with solid vuln-dev experience; give him a 4-8 full engineering weeks to try to bypass the mitigation when exploiting historical bugs. See if the mitigation holds, and to what extent. Most of the time, the result will be that the mitigation did not live up to the promise. This is good news: You have avoided imposing a tax on everybody (in complexity, usability, performance) that provides no or insufficient benefit.
3. When writing the code for the mitigation, *especially* when it touches kernel components, have a very stringent code review with review history. The reviewer should question any unwarranted complexity and ask for clarification.
   Follow good coding principles - avoid functions with hidden side effects that are not visible from the name etc. - the stringency of the code review should at least match the stringency of a picky C++ readability reviewer, if not exceed it.”

There are also these three slides to remember:

In short: Make it easy for people to understand the design rationale behind the mitigation. Make sure this design rationale is both accessible, and easily debated / discussed. Be precise and quantifiable in your claims so that people can challenge the mitigation on it's professed merits.