Saturday, August 17, 2019

Rashomon of disclosure

In a world of changing technology, there are few constants - but if there is one constant in security, it is the rhythmic flare-up of discussions about disclosure on the social-media-du-jour (mailing lists in the past, now mostly Twitter and Facebook).

Many people in the industry have wrestled with, and contributed to, the discussions, norms, and modes of operation - I would particularly like to highlight contributions by Katie Moussouris and Art Manion, but there are many that would deserve mentioning whose true impact is unknown outside a small circle. In all discussions of disclosure, it is important to keep in mind that many smart people have struggled with the problem. There may not be easy answers.

In this blog post, I would like to highlight a few aspects of the discussion that are important to me personally - aspects which influenced my thinking, and which are underappreciated in my view.

I have been on many (but not most) sides of the table during my career:
  • On the side of the independent bug-finder who reports to a vendor and who is subsequently threatened.
  • On the side of the independent bug-finder that decided reporting is not worth my time.
  • On the side of building and selling a security appliance that handles malicious input and that needs to be built in a way that we do not add net exposure to our clients.
  • On the side of building and selling a software that clients install.
  • On the side of Google Project Zero, which tries to influence the industry to improve its practices and rectify some of the bad incentives.
The sides of the table that are notably missing here are the role of the middle- or senior-level manager that makes his living shipping software on a tight deadline and who is in competition for features, and the role of the security researcher directly selling bugs to governments. I will return to this in the last section.

I expect almost every reader will find something to vehemently disagree with. This is expected, and to some extent, the point of this blog post.

The simplistic view of reporting vulnerabilities to vendors

I will quickly describe the simplistic view of vulnerability reporting / patching. It is commonly brought up in discussions, especially by folks that have not wrestled with the topic for long. The gist of the argument is:
  1. Prior to publishing a vulnerability, the vulnerability is unknown except to the finder and the software vendor.
  2. Very few, if any, people are at risk while we are in this state.
  3. Publishing about the information prior to the vendor publishing a patch puts many people at risk (because they can now be hacked). This should hence not happen.
Variants of this argument are used to claim that no researcher should publish vulnerability information before patches are available, or that no researcher should publish information until patches are applied, or that no researcher should publish information that is helpful for building exploits.

This argument, at first glance, is simple, plausible, and wrong. In the following, I will explain the various ways in which this view is flawed.

The Zardoz experience

For those that joined Cybersecurity in recent years: Zardoz was a mailing list on which "whitehats" discussed and shared security vulnerabilities with each other so they could be fixed without the "public" knowing about them.

The result of this activity was: Every hacker and active intelligence shop at the time wanted to have access to this mailing list (because it would regularly contain important new attacks). They generally succeeded. Quote from the Wikipedia entry on Zardoz:
On the other hand, the circulation of Zardoz postings among computer hackers was an open secret, mocked openly in a famous Phrack parody of an IRC channel populated by notable security experts.[3]
History shows, again and again, that small groups of people that share vulnerability information ahead of time always have at least one member compromised; there are always attackers that read the communication.

It is reasonably safe to assume that the same holds for the email addresses to which security vulnerabilities are reported. These are high-value targets, and getting access to them (even if it means physical tampering or HUMINT) is so useful that well-funded persistent adversaries need to  be assumed to have access to them. It is their job, after all.

(Zardoz isn't unique. Other examples are unfortunately less-well documented. Mail spools of internal mailing lists of various CERTs were circulated in hobbyist hacker circles in the early 2000s, and it is safe to assume that any dedicated intelligence agency today can reproduce that level of access.)

The fallacy of uniform risk

Risk is not uniformly distributed throughout society. Some people are more at-risk than others: Dissidents in oppressive countries, holders of large quantities of cryptocurrency, people who think their work is journalism when the US government thinks their work is espionage, political stakeholders and negotiators. Some of them face quite severe consequences from getting hacked, ranging from mild discomfort to death.

The majority of users in the world are much less at risk: The worst-case scenario for them, in terms of getting hacked, is inconvenience and a moderate amount of financial loss.

This means that the naive "counting" of victims in the original argument makes a false assumption:  Everybody has the same "things to lose" by getting hacked. This is not the case: Some people have their life and liberty at risk, but most people don't. For those that do not, it may actually be rational behavior to not update their devices immediately, or to generally not care much about security - why take precautions against an event that you think is either unlikely or largely not damaging to you?

For those at risk, though, it is often rational to be paranoid - to avoid using technology entirely for a while, to keep things patched, and to invest time and resources into keeping their things secure.

Any discussion of the pros and cons of disclosure should take into account that risk profiles vary drastically. Taking this argument to the extreme, the question arises: "Is it OK to put 100m people at risk of inconvenience if I can reduce the risk of death for 5 people?"

I do not have an answer for this sort of calculation, and given the uncertainty of all probabilities and data points in this, I am unsure whether one exists.

Forgetting about patch diffing

One of the lessons that our industry sometimes (and to my surprise) forgets is: Public availability of a patch is, from the attacker perspective, not much different than a detailed analysis of the vulnerability including a vulnerability trigger.

There used to be a cottage industry of folks that analyze patches and write reports on what the fixed bugs are, whether they were fixed correctly, and how to trigger them. They usually operated away from the spotlight, but that does not mean they do not exist - many were our customers.

People in the offensive business can build infrastructure that helps them rapidly analyze patches and get the information they need out of them. Defenders, mostly due to organizational and not technical reasons, can not do this. This means that in the absence of a full discussion of the vulnerability, defenders will be at a significant information disadvantage compared to attackers.

Without understanding the details of the vulnerability, networks and hosts cannot be monitored for its exploitation, and mitigations-other-than-patching cannot be applied.

Professional attackers, on the other hand, will have all the information about a vulnerability not long after they obtain a patch (if they did not have it beforehand already).

The fallacy of "do not publish triggers"

When publishing about a vulnerability, should "triggers", small pieces of data that hit the vulnerability and crash the program, be published?

Yes, building the first trigger is often time-consuming for an attacker. Why would we save them the time?

Well, because without a public trigger for a vulnerability, at least, it is extremely hard for defensive staff to determine whether a particular product in use may contain the bug in question. A prime example of this CVE-2012-6706: Everybody assumed that the vulnerability is only present on Sophos; no public PoC was provided. So nobody realized that the bug lived in upstream Unrar, and it wasn't until 2017 that it got re-discovered and fixed. 5 years of extra time for a bug because no trigger was published.

If you have an Antivirus Gateway running somewhere, or any piece of legacy software, you need at least a trigger to check whether the product includes the vulnerable software. If you are attempting at building any form of custom detection for an attack, you also need the trigger.

The fallacy of "do not publish exploits"

Now, should exploits be published? Clearly the answer should be no?

In my experience, even large organizations with mature security teams and programs often struggle to understand the changing nature of attacks. Many people that are now in management positions cut their teeth on (from today's perspective) relatively simple bugs, and have not fully understood or appreciated how exploitation has changed.

In general, defenders are almost always at an information disadvantage: Attackers will not tell them what they do, and gleefully applaud and encourage when the defender gets a wrong idea in his head about what to do. Read the declassified cryptolog_126.pdf Eurocrypt trip report to get a good impression of how this works.
Three of the last four sessions were of no value whatever, and indeed there was almost nothing at Eurocrypt to interest us (this is good news!). The scholarship was actually extremely good; it's just that the directions which external cryptologic researchers have taken are remarkably far from our own lines of interest. 
Defense has many resources, but many of them are misapplied: Mitigations performed that do not hold up to an attacker slightly changing strategies, products bought that do not change an attacker calculus or the exploit economics, etc.

A nontrivial part of this misapplication is information scarcity about real exploits. My personal view is that Project Zero's exploit write-ups, and the many great write-ups by the Pwn2Own competitors and other security research teams (Pangu and other Chinese teams come to mind) about the actual internal mechanisms of their exploits is invaluable to transmit understanding of actual attacks to defenders, and are necessary to help the industry stay on course.

Real exploits can be studied, understood, and potentially used by skilled defenders for both mitigation and detection, and to test other defensive measures.

The reality of software shipping and prioritization

Companies that sell software make their money by shipping new features. Managers in these organizations get promoted for shipping said features and reaching more customers. If they succeed in doing so, their career prospects are bright, and by the time the security flaws in the newly-shipped features become evident, they are four steps in the career ladder and two companies away from the risk they created.

The true cost of attack surface is not properly accounted for in modern software development (even if you have an SDLC); largely because this cost is shouldered by the customers that run the software - and even then, only by a select few that have unusual risk profiles.

A sober look at current incentive structures in software development shows that there is next to zero incentive for a team that ships a product to invest in security on a 4-5 year horizon. Everybody perceives themselves to be in breakneck competition, and velocity is prioritized. This includes bug reports: The entire reason for the 90-day deadline that Project Zero enforced was the fact that without a hard deadline, software vendors would routinely not prioritize fixing an obvious defect, because ... why would you distract yourself with doing it if you could be shipping features instead?

The only disincentive to adding new attack surface these days is getting heckled on a blog post or in a Blackhat talk. Has any manager in the software industry ever had their career damaged by shipping particularly broken software and incurring risks for their users? I know of precisely zero examples. If you know of one, please reach out, I would be extremely interested to learn more.

The tech industry as risk-taker on behalf of others

(I will use Microsoft as an example in the following paragraphs, but you can replace it with Apple or Google/Android with only minor changes. The tech giants are quite similar in this.)

Microsoft has made 248bn$+ in profits since 2005. In no year did they make less than 1bn$ in profits per month. Profits in the decade leading up to 2005 were lower, and I could not find numbers, but even in 2000 MS was raking in more than a billion in profits a quarter. And part of these profits were made by incurring risks on behalf of their customers - by making decisions to not properly sandbox the SMB components, by under-staffing security, by not deprecating and migrating customers away from insecure protocols.
The software product industry (including mobile phone makers) has reaped excess profits for decades by selling risky products and offloading the risk onto their clients and society. My analogy is that they constructed financial products that yield a certain amount of excess return but blow up disastrously under certain geopolitical events, then sold some of the excess return and *all* of the risk to a third party that is not informed of the risk.

Any industry that can make profits while offloading the risks will incur excess risks, and regulation is required to make sure that those that make the profits also carry the risks. Due to historical accidents (the fact that software falls under copyright) and unwillingness to regulate the golden goose, we have allowed 30 years of societal-risk-buildup, largely driven by excess profits in the software and tech industry.

Now that MS (and the rest of the tech industry) has sold the rest of society a bunch of toxic paper that blows up in case of some geopolitical tail events (like the resurgence of great-power competition), they really do not wish to take the blame for it - after all, there may be regulation in the future, and they may have to actually shoulder some of the risks they are incurring.

What is the right solution to such a conundrum? Lobbying, and a concerted PR effort to deflect the blame. Security researchers, 0-day vendors, and people that happen to sell tools that could be useful to 0-day vendors are much more convenient targets than admitting: All this risk that is surfaced by security research and 0-day vendors is originally created for excess profit by the tech industry.

FWIW, it is rational for them to do so, but I disagree that we should let them do it :-). 

A right to know

My personal view on disclosure is influenced by the view that consumers have a right to get all available information about the known risks of the products they use. If an internal Tobacco industry study showed that smoking may cause cancer, that should have been public from day 1, including all data.

Likewise, consumers of software products should have access to all known information about the security of their product, all the time. My personal view is that the 90 days deadlines that are accepted these days are an attempt at balancing competing interests (availability of patches vs. telling users about the insecurity of their device).

Delaying much further or withholding data from the customer is - in my personal opinion - a form of deceit; my personal opinion is that the tech industry should be much more aggressive in warning users that under current engineering practices, their personal data is never fully safe in any consumer-level device. Individual bug chains may cost a million dollars now, but that million dollars is amortized over a large number of targets, so the cost-per-individual compromise is reasonably low.

I admit that my view (giving users all the information so that they can (at least in theory) make good decisions using all available information) is a philosophical one: I believe that withholding available information that may alter someone's decision is a form of deceit, and that consent (even in business relationships) requires transparency with each other. Other people may have different philosophies.

Rashomon, or how opinions are driven by career incentives

The movie Rashomon that gave this blog post the title is a beautiful black-and-white movie from 1950, directed by the famous Akira Kurosawa. From the Wikipedia page:
The film is known for a plot device that involves various characters providing subjective, alternative, self-serving, and contradictory versions of the same incident.
If you haven't seen it, I greatly recommend watching it.

The reason why I gave this blog post the title "Rashomon of disclosure" is to emphasize the complexity of the situation. There are many facets, and my views are strongly influenced by the sides of the table I have been on - and those I haven't been on.

Everybody participating in the discussion has some underlying interests and/or philosophical views that influence their argument.

Software vendors do not want to face up to generating excess profits by offloading risk to society. 0-day vendors do not want to face up to the fact that a fraction of their clients kills people (sometimes entirely innocent ones), or at least break laws in some jurisdiction. Security researchers want to have the right to publish their research, even if they fail to significantly impact the broken economics of security.

Everybody wants to be the hero of their own story, and in their own account of the state of the world, they are.

None of the questions surrounding vulnerability disclosure, vulnerability discovery, and the trade-offs involved in it are easy. People that claim there is an easy and obvious path to go about security vulnerability disclosure have either not thought about it very hard, or have sufficiently strong incentives to self-delude that there is one true way.

After 20+ years of seeing this debate go to and fro, my request to everybody is: When you explain to the world why you are the hero of your story, take a moment to reflect on alternative narratives, and make an effort to recognize that the story is probably not that simple.

Tuesday, October 02, 2018

Turing completeness, weird machines, Twitter, and muddled terminology

First off, an apology to the reader: I normally spend a bit of effort to make my blog posts readable / polished, but I am under quite a few time constraints at the moment, so the following will be held to lesser standards of writing than usual.

A discussion arose on Twitter after I tweeted that the use of the term "Turing-complete" in academic exploit papers is wrong. During that discussion, it emerged that there are more misunderstandings of terms that play into this. Correcting these things on Twitter will not work (how I long for the days of useful mailing lists), so I ended up writing a short text. Pastebin is not great for archiving posts either, so for lack of a better place to put it, here it comes:

Our misuse of "Turing completeness" and "weird machine" is harmful and confusing

1. "Turing completeness"

TC refers to computability (in terms of simulating other computers), and is well-defined there. It means "can simulate any Turing machine". There are a few things to keep in mind about TC:

  • None of the machines we use day-to-day are TC in the absence of I/O. TC requires infinite memory.
  • TC arises really quickly; all you need is one instruction that subtracts and performs a conditional branch if not zero instruction. It also arises in quite minimal recurrence equations randomly (see the fact that game of life is Turing-complete).

For the exploitation case, the desirable outcome is "we got integer arithmetic and arbitrary read and write". Turing-completeness says nothing about exploitability as exploitability has little to do with computation and much more to do with "what sort of states are reachable given the possible interactions with my target". These are two very distinct questions.

Font renderers are Turing-complete, Javascript is, and considering that if I/O is present, part of the computation may actually be performed on the attacker side, IMAP is as well.

It is simply a complete mis-use of terms. Even the paper that first used it just used it to say "we eyeballed the instructions we got, and they look like we can probably do everything we'd want to do":
The set of gadgets we describe is Turing complete by inspection, so return-oriented programs can do anything possible with x86 code.
Let's not mis-use a precisely defined term for something else just because saying "TC" makes us feel like we are doing real computer science.

Exploitation is about reachable states (and transitions) much more than about computability.

2. "Weird machines"


Weird machines are one of the most tragically misunderstood abstractions (which, if people understand it properly, helps greatly in reasoning about exploitation).

The point of weird machine research is *not* about showing that everything is Turing complete. The point of weird machine research is that when any finite state automaton is simulated, and when that simulation gets corrupted, a new machine emerges, with it's own instruction set. It is this instruction set that gets programmed in attacks. Constraining the state transitions (and hence the reachable states) of a weird machine is what makes exploitation impossible. The computational power (in the TC sense) is secondary.

The key insight that would have saved us all from the dreary experience of reading 25 tit-for-tat-ROP-and-bad-mitigation papers is that if you do not constrain what the weird machines that emerge on a memory corruption can do, your mitigation is probably not going to help. Most mitigations blacklist a particular weird machine program, without constraining the machine's capabilities.

Weird machines are the proper framework in which to reason about pretty much all exploits that are not side-channel based or missing-auth-based.

Now, unfortunately, the abstraction was well-understood intuitively by a few people that had done a lot of exploitation, but not made precise or put in writing. As a consequence, other researchers heard the term, and "filled it with their imagination": They then used the term wherever they found a surprising method of computation in random things by using them as designed. This made the term even harder to understand - in the same way that nobody will be able to understand "Turing complete" by just reading ROP papers.

I am particularly passionate about this part because I spent a few months putting the informally-defined weird machine onto sound footing and into precise definitions to writing to prevent the term from getting even more muddled :-)

To summarize:


  • The use of "Turing completeness" in ROP papers is an abuse; the use of that term does not correspond to the use of the term in theoretical CS. If we are going to use rigorous terms, we should make sure we use them correctly. I see students and researchers be confused about the actual meaning all the time now.
  • The ROP-tit-for-tat-mitigation paper deluge (and the subsequent unhealthy focus on CFI) could have been avoided if people had reasoned about ROP things at the right abstraction levels (weird machines). They would have had the chance, because there were quite a few conversations between the authors of the tit-for-tat-papers and people that understood things properly :-), but my suspicion is that there was no incentive to reason on an abstraction level that messes with your ability to salami-slice more papers.
  • It was made harder to understand the concept of weird machines because people that had not properly understood weird machines started calling everything that looked vaguely interesting "weird machines". Nobody was sitting on any review boards to stop them :-), so now that term (which also has a precise definition) is used in a wrong manner, too.

My larger point is: We have so much handwavy and muddled terminology in the papers in this field, it is harmful to young researchers, other fields, and PhD students. The terminology is confusing, often ill-defined (what is a gadget?), terms that have a precise meaning in general CS get used to mean something completely else without anybody explaining it (Turing complete).

This creates unnecessary and harmful confusion, and it should be fixed.

Saturday, March 31, 2018

A bank statement for app activity (and thus personal data)

During my long sabbatical in 2015-2016 I had plenty of time to think about random things and come up with strange ideas. Most of these ideas are more funny than practical - their primary use is boring people that are reckless enough to have drinks with me.

This blog post describes one of these ideas. With the recent renewed interest in privacy and overreach of smart phone apps, it seems like a topic that is - at least temporarily - less boring than usual.

ML, software behavior, and the boundary between 'malicious' and 'non-malicious'

I have seen a lot of human brain power (and a vast amount of computational power) thrown at the problem of automatically deciding whether a given piece of software is good or bad. 

This is usually done as follows:
  1. Collect a lot of information about the behavior of software (normally by running the software in some simulated environment)
  2. Extract features from this information
  3. Apply some more-or-less sophisticated machine learning model to decide between "good" or "bad"
The underlying idea behind this is that there is "bad" behavior, and "good" behavior, and if we could somehow build a machine learning model that is sufficiently powerful, we could automatically decide whether a given piece of software is good or bad.

In practice, this rarely works without significant false-positive problems, or significant false-negative-problems, or all sorts of complicated corner-cases where the system fails.

In 2015, I had to deal with the fallout of the badly-phrased Wassenaar wording: Export-control legislation which tried to define "bad behavior" for software. During this, it became clear to me that the idea that behavior alone determines good/bad is flawed.

The behavior of a piece of software does not determine whether it is malicious or not. The true defining line between malicious and non-malicious software is whether software does what the user expects it to do

Users run software because they have an expectation for what this software does. They grant permissions for software because they have an expectation for the software to do something for them ("I want to make phone calls, so clearly the app should use the microphone"). This permission is given conditionally, with context -- the user does not want to give the app permission to switch on the microphone when the user does not intend to make a phone call.

The question of malicious / non-malicious software is a question of alignment between user expectations and software behavior.

This means, in practice, that efforts in applying machine learning to separate malicious from non-malicious software are doomed to fail, because they fail to measure the one dimension through which the boundary between good and bad runs.

Intuitively, this can be illustrated with the two pictures below. They show the same set of red and green points in 3d-space from two different perspectives -- once with their z-axis projected away, and once in a 3-d plot where the z-axis is still visible:

Cloud of points from the side, with the "important" dimension projected away. It is near-impossible to draw a sane boundary between red and green points, and whatever boundary you draw won't generalize well.

Same cloud of points, with the "important" dimension going from left to right. It is much clearer how to separate green from red points now.
The question that arises naturally, then, is:

How can one measure the missing dimension (user intent)?

User intent is a difficult thing to measure. The software industry has the practice of forcing the user to agree to some ridiculously wide-reaching terms-of-services or EULA that few users read, even fewer understand, and which are often near-equivalent to giving the person you hire to clean your flat a power of attorney over all your documents, and allowing them to throw parties in your flat while you are not looking.

It is commonly argued that - because the user clicked "agree" to an extremely broad agreement - the user consented to everything the software can possibly do.

But consent to software actions is context-dependent and conditioned on particular, specific actions. It is fine for my messenger to request access to my camera, microphone and files - I may need to send a picture, I may need to make a phone call, and I may need to send an attachment. It is not OK for my messenger to use my microphone to see if a particular ultrasonic tracker sound is received, it is not OK for my messenger to randomly search through files etc.

Users do not get to tell the software vendor their intent and the context for which they are providing consent.

Now, given that user intent is difficult to measure up-front - how about we simply ask the user whether something that an app / software did was what he expected it to do?

Information and attention is a currency - but one with bad accounting

The modern ad economy runs on attention and private data. The big advertising platforms make their money by selling the combination of user attention and the ability to micro-target advertisements given contextual data about a user. The user "pays" for goods and services by providing attention and private data.

People often fear that big platforms will "sell their data". This is, at least for the smarter / more profitable platforms, an unnecessary fear: These platforms make their money by having data that others do not have, and which allows better micro-targeting. They do not make their money "selling data", they make money "monetizing the data they have".

The way to think about the relationship between the user and the platform is more of a clicheed "musician-agent" relationship: The musician produces something, but does not know how to monetize it. His Agent knows how to monetize it, and strikes a deal with the musician: You give me exclusive use of your product, and I will monetize it for you - and take a cut from the proceeds.

The profits accumulated by the big platforms are the difference between what the combination of attention & private data obtained from users is worth and the cost of obtaining this attention and data.

For payments in "normal" currency, users usually have pretty good accounting: They know what is in their wallet, and (to the extent that they use electronic means for payments) they get pretty detailed transaction statements. It is not difficult for a normal household to reconstruct from their bank statements relatively precisely how much they paid for what goods in a given month.

This transparency creates trust: We do not hesitate much to give our credit card numbers to online service providers, because we know that we can intervene if they charge our credit cards without reason and in excess of what we agreed to.

Private information, on the other hand, is not accounted for. Users have no way to see how much private data they provide, and whether they are actually OK with that.

A bank statement for app/software activity

How could one empower users to account for their private data, while at the same time helping platform providers identify malicious software better?

By providing users with the equivalent of a bank statement for app/software activity. The way I imagine it would be roughly as follows:

A separate component of my mobile phone (or computer) OS keeps detailed track of app activity: What peripherals are accessed at what times, what files are accessed, etc.

Users are given the option of checking the activity on their device through a UI that makes these details understandable and accessible:
  • App XYZ accessed your microphone in the last week at the following times, showing you the following screen:
    • Timestamp 1, screenshot 1
    • Timestamp 2, screenshot 2
  • Does this match your expectations of what the app should do? YES / NO
  • App ABC accessed the following files during the last week at the following times, showing you the following screen:
    • Timestamp 3, screenshot 3
      • Filename
      • Filename
      • filename
  • Does this match your expectations of what the app should do? YES / NO
At least on modern mobile platforms, most of the above data is already available - modern permissions systems can keep relatively detailed logs of "when what was accessed". Adding the ability to save screenshots alongside is easy.

Yes, a lot of work has to go into a thoughtful UI, but it seems worth the trouble: Even if most users will randomly click on YES / NO, the few thousand users that actually care will provide platform providers valuable information about whether an app is overreaching or not. At the same time, more paranoid users (like me) would feel less fearful about installing useful apps: If I see the app doing something in excess of what I would like it to do, I could remove it.

Right now, users have extremely limited transparency into what apps are actually doing. While the situation is improving slowly (most platforms allow me to check which app last used my GPS), it is still way too opaque for comfort, and overreach / abuse is likely pervasive.

Changing this does not seem hard, if any of the big platform providers could muster the will.

It seems like a win / win situation, so I can hope. I can also promise that I will buy the first phone to offer this in a credible way :-).

PS: There are many more side-benefits to the above model - for example making it more difficult to hack a trusted app developer to then silently exfiltrate data from users that trust said developer - but I won't bore you with those details now.

Wednesday, February 21, 2018

Two small notes on the "malicious use of AI" report

After a long hiatus on this blog, a new post! Well, not really - but a whitepaper was published today titled "The Malicious Use of Artificial Intelligence", and I decided I should cut/paste/publish two notes that apply to the paper from an email I wrote a while ago.

Perhaps they are useful to someone:
1) On the ill-definedness of AI: AI is a diffuse and ill-defined term. Pretty much *anything* where a parameter is inferred from data is called "AI" today. Yes, clothing sizes are determined by "AI", because mean measurements are inferred from real data.

To test whether one has fallen into the trap as viewing AI as something structurally different from other mathematics or computer science (it is not!), one should try to battle-test documents about AI policy, and check them for proportionality, by doing the following:

Take the existing test and search/replace every occurrence of the word "AI" or "artificial intelligence" with "Mathematics", and every occurrence of the word "machine learning" with "statistics". Re-read the text and see whether you would still agree.

2) "All science is always dual-use":

Everybody that works at the intersection of science & policy should read Hardy's "A mathematicians apology". https://www.math.ualberta.ca/mss/misc/A%20Mathematician%27s%20Apology.pdf

I am not sure how many of the contributors have done so, but it is a fascinating read - he contemplates among other things the effect that mathematics had on warfare, and to what extent science can be conducted if one has to assume it will be used for nefarious purposes.

My favorite section is the following:
We have still one more question to consider. We have concluded that the trivial mathematics is, on the whole, useful, and that the real mathematics, on the whole, is not; that the trivial mathematics does, and the real mathematics does not, ‘do good’ in a certain sense; but we have still to ask whether either sort of mathematics does harm. It would be paradoxical to suggest that mathematics of any sort does much harm in time of peace, so that we are driven to the consideration of the effects of mathematics on war. It is every difficult to argue such questions at all dispassionately now, and I should have preferred to avoid them; but some sort of discussion seems inevitable. Fortunately, it need not be a long one.
There is one comforting conclusions which is easy for a real mathematician. Real mathematics has no effects on war.
No one has yet discovered any warlike purpose to be served by the theory of numbers or relativity, and it seems very unlikely that anyone will do so for many years. It is true that there are branches of applied mathematics, such as ballistics and aerodynamics, which have been developed deliberately for war and demand a quite elaborate technique: it is perhaps hard to call them ‘trivial’, but none of them has any claim to rank as ‘real’. They are indeed repulsively ugly and intolerably dull; even Littlewood could not make ballistics respectable, and if he could not who can? So a real mathematician has his conscience clear; there is nothing to be set against any value his work may have; mathematics is, as I said at Oxford, a ‘harmless and innocent’ occupation. The trivial mathematics, on the other hand, has many applications in war.
The gunnery experts and aeroplane designers, for example, could not do their work without it. And the general effect of these applications is plain: mathematics facilitates (if not so obviously as physics or chemistry) modern, scientific, ‘total’ war.

The most fascinating bit about the above is how fantastically presciently wrong Hardy was when speaking about the lack of war-like applications for number theory or relativity - RSA and nuclear weapons respectively. In a similar vein - I was in a relationship in the past with a woman who was a social anthropologist, and who often mocked my field of expertise for being close to the military funding agencies (this was in the early 2000s). The first thing that SecDef Gates did when he took his position was hire a bunch of social anthropologists to help DoD unravel the tribal structure in Iraq.

The point of this disgression is: It is impossible for any scientist to imagine future uses and abuses of his scientific work. You cannot choose to work on "safe" or "unsafe" science - the only choice you have is between relevant and irrelevant, and the militaries of this world *will* use whatever is relevant and use it to maximize their warfare capabilities.

Tuesday, August 22, 2017

A quick post on Wikipedia-scrubbing and a historical document on binary diffing

I am a huge fan of Wikipedia -- I sometimes browse Wikipedia like other people watch TV, skipping from topic to topic and - on average - being impressed by the quality of the articles.

One thing I have noticed in recent years, though, is that the base-democratic principles of Wikipedia open it up to manipulation and whitewashing - Wikipedia's guidelines are strict, and a person can get a lot of negative information removed just by cleverly using the guidelines to challenge entries. This is no fault of Wikipedia -- in fact, I think the guidelines are good and useful -- but it is often instructive to read the history of a particular page.

I recently stumbled over a particularly amusing example of this, and feel compelled to write about it.

More than a twelve years ago, when BinDiff was brand-new and wingraph32.exe was still the graph visualization tool of choice, there was a controversy surrounding a product called "CherryOS" - which purported to be an Apple emulator. A student had raised the allegation that "CherryOS" had misappropriated source code from an open-source project called "PearPC" on his website, and the founder of the company selling CherryOS (somebody by the name of Arben Kryeziu) had threatened the student legally over this claim.

In order to help a good cause, we did a quick analysis of the code similarities between CherryOS and PearPC, and found that approximately half of the code in CherryOS was verbatim copy & paste from PearPC. We wrote a small report, provided it to the lawyer of the student under allegation, and the entire kerfuffle died down quickly. Wikipedia used to have a page that detailed some of the drama for a few years thereafter.

I recently stumbled over the Wikipedia page of CherryOS, and was impressed: The page had been cleaned of any information that supported the code-theft claims, and offered a narrative where there had never been conclusive consensus that CherryOS was full of misappropriated code. This is not a reflection of what happened back then at all.

Anyhow, in a twist of fate, I also found an old USB stick which still contained a draft of the 2005 note we wrote. For the sake of history, here it is :-)

I had forgotten how painful it was to look at disassembly CFGs in wingraph32. Sometimes, when I am frustrated at the speed at which RE tools improved during my professional life, it is useful to be reminded what the dark ages looked like.


Saturday, October 01, 2016

"Why do you work in security instead of something more lasting ?"

This post grew out of a friend on Facebook asking (I paraphrase) "why do you spend your time on security instead of using your brainpower for something more lasting ?". I tried to answer, and ended up writing a very long reply. Another friend then encouraged me to re-post my reply to a wider audience. The below is a slightly edited and expanded version. It is much less polished than my usual blog posts, more personal, and somewhat stream-of-conscious-y. Apologies for that.

Why do I work in security instead of on something more lasting?

Predictions about what is "lasting" are very difficult to make :-). I think outside of the exploit-of-the-day, there's lasting work to be done in understanding of exploitation (because machines and automata aren't going away, and neither are programming mistakes), and I sincerely hope I'll have opportunity to do that work.

I tried my hand in cryptography / academia, and found it more prone to political trends/fads and less blindly results-oriented than security - to my great disappointment. When all attacks are of theoretical complexity 2^96, verifying and replicating results becomes difficult, and objective truth suffers (see below).

In the following, I will state a few things that I really like about the computer security community. I did not realize this immediately - instead, I learnt this over many years and engagement in other communities.
  1. Original thinkers. I used to joke that there are less than 2 dozen reasons why security as a field doesn't suck, and I know many of them personally. Now, the 2 dozen is bullshit, but what is true that in all the noise & hype, I have met a number of very fun, unconventional, and deeply insightful thinkers of very different backgrounds. They are few and far between, but I wouldn't have met them without security, and I am grateful for having met them. Many exploits require considerable inventiveness, and non-obvious / creative ways of solving problems; they are sometimes like a good joke / magic trick: With an unexpected twist that makes you laugh in disbelief.
  2. Tolerance of non-conformism and diverse educational backgrounds. There are few other industries where people who did not finish high school mix with people with postgraduate degrees, and debate on even terms. With all it's problems and biases, the part of the community I grew up with did not care about gender, skin color, or parental income - everybody was green writing on a black screen.
  3. Intellectual honesty. When discussing attacks, there is "objective truth" - you can establish whether an attack works or does not work, and checking reproducibility is easy. This is not true in many other disciplines, and "truth" becomes a matter of social consensus - even in pure math, where proof should be absolute. Having objective truth is extremely helpful to prevent a discipline to devolve into scholasticism.
Many other fields which may be more "lasting" do not have the luxury of these three points. Also be aware that my visibility into the security community is very skewed:

My skewed view of the security community

It is common to hear negative things about the community - that it is elitist, full of posturing, or of people that are mean / demeaning to others with less experience. This is not the community I experience - and this discrepancy has been puzzling me for a while.

For one thing, everybody is always nice to me. I am not sure why this is the case, but the only non-niceties I encountered in this industry were in leaked email spools. This makes it difficult for me to notice people being mean to newcomers and elitist - and it saddens me to hear that people are being shit to each other.

People weren't always nice to me - like any group of teenagers, 1990's IRC was very often not a friendly place, and #cracking would kickban you for asking a question. I found a home of sorts in a channel called #cracking4newbies - a very welcoming environment dedicated to joint learning. It was great for me: I could ask questions, and either got answers or links to documentation. A few members of #cracking were no longer active, and held status in the channel for historical reasons, #cracking4newbies on the other hand was full of eager & active youngsters.

I somehow managed to avoid being around the posturing and status games much, and in some bizarre stroke of luck, have managed to do so up to this day. The people in the security community I spend time with are genuinely interested in the technical challenges, genuinely curious, and usually do not care about the posturing part. The posturing may happen at industry conferences, but I tend to not notice - the technically interesting talks tend to adhere to substance-over-style, and the rest is as relevant to me as big advertisements for broken content inspection appliances.

All I want to say with this section is: I do not know how I managed to avoid experiencing the bad sides of the security community much. Some of it was luck, some of it was instinct. There are plenty of things I find annoying about the security community (but that is for another post :-), but in my day-to-day life, I don't experience much of it. If you are in security, and feel that the community is elitist or demeaning to people learning, I hope you succeed in seeking out the (many) people I encountered that were happy to share, explain, and just jointly nerd out on something. Feel free to reach out any time.

On building vs. breaking

I quite often hear the phrase "I quit security and I am much happier building instead of breaking things". This is a normal sentiment - but for me, security was never about "just" breaking things. Tooling was always inadeqate, workflows horribly labour-intensive, and problems were always tackled on the lowest level of abstraction, missing the forest for the trees.

In my reverse engineering classes, I always encourage people to be tool builders. Most of security work today is akin to digging trenches with chopsticks. Invest in designing and building shovels. Perhaps we will even get a bulldozer in my lifetime. Slowly but surely, the industry is changing in that direction: Microsoft is commercializing SAGE, no code auditor is more productive (even though more in-depth) than a farm of computers running AFL - but the discrepancy between the quality and quantity of tools that developers have available vs. the tools that security review has available is still vast.

I like my work most when I can cycle through building / breaking phases: Try to break something, notice how insanely badly the tooling is, cycle through an iteration of tool development, return to the breaking etc.

I realize this isn't the path for everybody, but I don't think that security is "always just about breaking". The most persistent person gets bored of chopstick-trench-digging. Invest in tooling. Being a better developer makes you a better hacker. And perhaps you like building more than breaking, and I can't fault you for that.

My friend Sören happens to be one of the best C++ developers I know. When we first met in undergraduate math class, I described what I do for a living to him (reading code for subtle mistakes), and he said "that sounds like one of the worst imaginable jobs ever". He is a builder, and I have nothing but admiration and respect for him - and from the builder's perspective, his assessment is right.

I still like finding subtle bugs. To paraphrase another person who I respect a lot: "People still search for new stuff in Shakespeare hundreds of years later".

Using security as an excuse for broad learning

I once read that "cryptography gathers many very different areas of mathematics like a focal lens". The same is very true of security and computer science. Security happens at the boundaries between layers, and I have used working in security as an excuse to learn about as many layers as possible: Low-level assembly, high-level stuff on formal verification, and even electrical engineering problems and their implications on security.
People talk about "full stack engineers" a lot; security allows me to roam the full stack of abstractions in computer science without guilt. All layers are relevant for security, all layers are interesting in their own right, and each layer has it's own funny quirks.

Summary

Given the length of this blog post, it is evident that I have asked myself the question "why do I do this" many times. And I have thought about devoting attention to other things often enough. Who knows, I am 35, so I have about 30 years of professional activity ahead of me - which may be enough to fail in one or two other fields before returning to give grandfather-security keynotes. :-)

But right now, I am actually enjoying having my hands dirty and thinking about heap layout for the first time in years.

Saturday, September 03, 2016

Essays about management in large(r) organisations (1): Process and flexibility

Even though I often profess that my primary interests are technical, by this point in my life I have been exposed to a variety of different organisations and management styles: From the self-organizing chaos of the 1996-2002 cracking/hacking groups, through the small engineering-centric startup zynamics, via the various organisations (both governmental and industry) I consulted for at some point, to the large (but nonetheless engineering-centric) culture at Google.


I enjoy thinking about organisations - their structure, how information flows, their strengths and dysfunctions. Part of it may be the influence of my father (who wrote extensively on matrix organisations, but also on organisations that fail); the other part is certainly the recognition that both company culture and organisational culture matter. In any organisation, setting the culture and organisational structure - and keeping it healthy - is paramount, and probably the key element that will allow long-term success. Ignore culture and organisation structure (both explicit and implicit) at your peril.

I had a lot of time to think in the last year, so in the coming months I will write a few posts / essays about company culture and management.

The first post is about organisational processes - why they are important, but also how they can take on a life of their own and strangle flexibility.

A technical anecdote to start with

In early 2004, the first prototype of BinDiff started to work properly - just when Microsoft released MS04-001: A series of amusing little memory corruptions inside the H.323 parsing component of Microsoft ISA server (a now-discontinued firewall product). Using BinDiff on the patch, it was evident that the problems were inside the ASN.1 PER parsing routines in a central library - but instead of fixing the library, the patch fixed the issue inside ISA server.
The patch fixed only one exploit path, but the actual vulnerability was still there. This meant that any other program using the same library remained vulnerable, and the patch had now effectively disclosed the security issue. I started searching for other applications that used this library. The first program I found which was also affected by this vulnerability was Netmeeting - Microsoft had inadvertently given a remote code execution bug in Netmeeting to everybody. It wasn't until MS04-011, at some point in April, that this vulnerability got fixed in the correct place -- the library.

The technical details of the bug are not terribly interesting - what is interesting is what the mistake revealed about flaws in Microsoft’s organisational structure, and how they reacted to the bug report.

What could we deduce/learn/extrapolate from this event?

  • Bug reports were likely routed to the product teams - e.g. if a bug is reported in your product, the bug report is routed to you.
  • Responsibility for fixing a bug appears to lie with the product teams (see above), and teams are incentivized (either directly or indirectly through feature deadlines etc.) to get bug reports “off their desk” quickly.
  • Patching shared central code is harder than patching code you own (for various reasons - perhaps compatibility concerns, other priorities from other teams, or perhaps even a heavyweight process to ask for changes in critical code).

What likely happened is that the ISA team decided that dealing with the issue on their side is enough - either because they did not realize that the same issue will affect others, or because dealing with the other team / the library is a pain, or for some other unknown reason. Microsoft’s bug fixing process incentivized “shallow” fixes, so for attackers, finding the ultimate root cause of a vulnerability could expose other vulnerable programs.

This is a classical example of making a locally convenient decision that adversely affects the larger organisation.

From what I heard, Microsoft learned from this event and made organisational changes to prevent similar mistakes in the future. They introduced a process where all patches are reviewed centrally before they go out to ensure that they don't inadvertently fix a bug in the wrong spot, or disclose a vulnerability elsewhere.

Processes as organisational learning


In what an MBA would call ‘organisational learning’, a process was created out of the experience with a previous failure in order to prevent the mistake from happening again. A process is somewhat similar to organisational scar tissue - the organisation hurt itself, and to prevent such injury in the future, the process is established.

Surprisingly, most organisations establish processes without documenting explicitly what sort of failure and what sort of incident caused the process to be established. This knowledge usually only lives in the heads of individuals that were there, or in the folklore of those that talked to those that were there. After a half a decade or so, nobody remembers the original incident - although the process will be alive and kicking.

A process can prevent an organisation from doing something stupid repeatedly - but all too often, the process takes on a life of its own: People start applying the process blindly, and in the hands of an overly-literally-minded person, the process becomes an obstacle to productivity or efficiency. The person in charge of applying and enforcing the process may themselves not know why it is there - just that it is "the process", and that bad things can happen when one doesn't follow it.

My grandfather used to say (I will paraphrase) : "a job with responsibility is a job where you don’t simply apply the rules, but need to make judgements about how and where to make exceptions". This quote carries an important truth:

People at all places in an organisation need to be ...
  1. Empowered to make exceptions: After demonstrating sound judgement, people need to feel empowered to make exceptions when the letter of a process gets in the way of the greater good and changing the process would be excessive (for example, in a one-off situation).
  1. Empowered to challenge processes: The reasoning behind a process must to be accessible to organisation members, and there needs to be a (relatively pain-free) method to propose changing the process. Since powerlessness is one of the main drivers of occupational burnout, this will help keep individuals and the organisational structure healthy.

Some organisations get the “exception” part right - most big organisations only function because people are regularly willing to bend / twist / ignore processes. Very, very few organisations get the “challenge” part right-- making sure that every employee knows and understands that processes are in the service of the company, and that improvements to processes are welcome.

I think that the failure to achieve the challenge-process frequently arises due  to "lack of institutional memory". When organisations fail to keep track of why a process was created, all sorts of harmful side-effects arise:
  1. Nobody can meaningfully judge the spirit of the process - what was it designed to prevent?
  2. Making an exception to the process is riskier - if you do not know what it was designed to prevent, how can you know that in this particular case that risk does not apply?
  3. Amending the process becomes riskier. (Same reason as above.)
  4. Challenging the process cannot happen in a decentralized / bottom-up fashion: It is often the most junior employees who may have the freshest eyes for obstructive processes - but since they do not know the history of why the processes exists, they often can't effectively propose a change since they don’t know the organisation well enough to rule out unwanted side-effects. This directly sabotages decentralised, bottom-up improvements of workflows.

What is a healthy way to deal with processes?
  1. Realize that they are a form of “organisational memory”: They are often formed as reaction to some unpleasant event - with the intent of preventing this event from repeating. It is also important to realize that unchecked and unchallenged processes can become organisational “scar tissue” - more hindrance than help.
  2. Keep track of the exact motivation for creating each process -- the “why”. This will involve writing half a page or more, and checking with others involved in the creation of the process that the description is accurate and understandable.
  3. The motivations behind the process should be accessible to everybody affected by it.
  4. Everybody should know that company processes are supposed to support, not hinder, getting work done. Everybody should feel empowered to suggest changes in a process - ideally while addressing why these changes will not lead to a repeat of the problem the process was designed to prevent.
  5. People should be empowered to deviate from the process or ignore it - but frequent or even infrequent-but-recurring exceptions are a red flag that the process needs to be improved. Don't accumulate "legacy process" and "organisational debt" through the mechanism of exception-granting.
  6. Everybody should be aware that keeping processes functional and lean is crucial to keeping the organisation healthy. Even if a process is unreasonable and obstructive, most people instinctively try to accept it - but the first instinct should ideally be to change it for the better. Constructively challenging a broken process is a service to the organisation, not an attack.
  7. It may be sensible to treat processes a bit like code - complete with ownership of the relevant process, and version control, and handover of process ownership when people change jobs. Amendments to processes can then be submitted as text, reviewed by the process owner, discussed, and eventually approved - much like a patch or removal of dead code.

Keeping an organisation healthy is hard. The most crucial ingredient to keeping it healthy, though, is that the members of the organisation care to keep it healthy. Therefore it is absolutely critical to encourage fixing the organisation when something is broken - and to not discourage people into "blindly following the process".