Someone is wrong on the internet (AGI Doom edition)

The last few years have seen a wave of hysteria about LLMs becoming conscious and then suddenly attempting to kill humanity. This hysteria, often expressed in scientific-sounding pseudo-bayesian language typical of the „lesswrong“ forums, has seeped into the media and from there into politics, where it has influenced legislation.

This hysteria arises from the claim that there is an existential risk to humanity posed by the sudden emergence of an AGI that then proceeds to wipe out humanity through a rapid series of steps that cannot be prevented.

Much of it is entirely wrong, and I will try to collect my views on the topic in this article - focusing on the „fast takeoff scenario“.

I had encountered strange forms of seemingly irrational views about AI progress before, and I made some critical tweets about the messianic tech-pseudo-religion I dubbed "Kurzweilianism" in 2014, 2016 and 2017 - my objection at the time was that believing in an exponential speed-up of all forms of technological progress looked too much like a traditional messianic religion, e.g. "the end days are coming, if we are good and sacrifice the right things, God will bring us to paradise, if not He will destroy us", dressed in techno-garb. I could never quite understand why people chose to believe Kurzweil, who, in my view, has largely had an abysmal track record predicting the future.

Apparently, the Kurzweilian ideas have mutated over time, and seem to have taken root in a group of folks associated with a forum called "LessWrong", a more high-brow version of 4chan where mostly young men try to impress each other by their command of mathematical vocabulary (not of actual math). One of the founders of this forum, Eliezer Yudkowsky, has become one of the most outspoken proponents of the hypothesis that "the end is nigh".

I have heard a lot of of secondary reporting about the claims that are advocated, and none of them ever made any sense to me - but I am also a proponent of reading original sources to form an opinion. This blog post is like a blog-post-version of a (nonexistent) YouTube reaction video of me reading original sources and commenting on them.

I will begin with the interview published at https://intelligence.org/2023/03/14/yudkowsky-on-agi-risk-on-the-bankless-podcast/. 

The proposed sequence of events that would lead to humanity being killed by an AGI is approximately the following:

1. Assume that humanity manages to build an AGI, which is a computational system that for any decision "outperforms" the best decision of humans. The examples used are all zero-sum games with fixed rule sets (chess etc.).
   
2. After managing this, humanity sets this AGI to work on improving itself, e.g. writing a better AGI.
   
3. This is somehow successful and the AGI obtains an "immense technological advantage".
4. The AGI also decides that it is in conflict with humanity.
5. The AGI then coaxes a bunch of humans to carry out physical actions that enable it to then build something that kills all of humanity, in case of this interview via a "diamondoid bacteria that replicates using carbon, hydrogen, oxygen, nitrogen, and sunlight", that then kills all of humanity.

This is a fun work of fiction, but it is not even science fiction. In the following, a few thoughts:

Incorrectness and incompleteness of human writing

Human writing is full of lies that are difficult to disprove theoretically

As a mathematician with an applied bent, I once got drunk with another mathematician, a stack of coins, and a pair of pliers and some tape. The goal of the session was „how can we deform an existing coin as to create a coin with a bias significant enough to measure“. Biased coins are a staple of probability theory exercises, and exist in writing in large quantities (much more than loaded dice).

It turns out that it is very complicated and very difficult to modify an existing coin to exhibit even a reliable 0.52:0.48 bias. Modifying the shape needs to be done so aggressively that the resulting object no longer resembles a coin, and gluing two discs of uneven weight together so that they achieve nontrivial bias creates an object that has a very hard time balancing on its edge.

An AI model trained on human text will never be able to understand the difficulties in making a biased coin. It needs to be equipped with actual sensing, and it will need to perform actual real experiments. For an AI, a thought experiment and a real experiment are indistinguishable.

As a result, any world model that is learnt through the analysis of text is going to be a very poor approximation of reality. 

Practical world-knowledge is rarely put in writing

Pretty much all economies and organisations that are any good at producing something tangible have an (explicit or implicit) system of apprenticeship. The majority of important practical tasks cannot be learnt from a written description. There has never been a chef that became a good chef by reading sufficiently many cookbooks, or a woodworker that became a good woodworker by reading a lot about woodworking.

Any skill that affects the real world has a significant amount of real-world trial-and-error involved. And almost all skills that affect the real world involve large quantities of knowledge that has never been written down, but which is nonetheless essential to performing the task.

The inaccuracy and incompleteness of written language to describe the world leads to the next point:

No progress without experiments

No superintelligence can reason itself to progress without doing basic science

One of the most bizarre assumptions in the fast takeoff scenarios is that somehow once a super-intelligence has been achieved, it will be able to create all sorts of novel inventions with fantastic capabilities, simply by reasoning about them abstractly, and without performing any basic science (e.g. real-world experiments that validate hypotheses or check consistency of a theory or simulation with reality).

Perhaps this is unsurprising, as few people involved in the LessWrong forums and X-Risk discussions seem to have any experience in manufacturing or actual materials science or even basic woodworking.

The reality, though, is that while we have made great strides in areas such as computational fluid dynamics (CFD), crash test simulation etc. in recent decades, obviating the need for many physical experiments in certain areas, reality does not seem to support the thesis that technological innovations are feasible „on paper“ without extensive and painstaking experimental science.

Concrete examples:

1. To this day, CFD simulations of the air resistance that a train is exposed to when hit by wind at an angle need to be experimentally validated - simulations have the tendency to get important details wrong.
2. It is safe to assume that the state-supported hackers of the PRCs intelligence services have stolen every last document that was ever put into a computer at all the major chipmakers. Having all this knowledge, and the ability to direct a lot of manpower at analyzing these documents, have not yielded the knowledge necessary to make cutting-edge chips. What is missing is process knowledge, e.g. the details of how to actually make the chips.
3. Producing ballpoint pen tips is hard. There are few nations that can reliably produce cheap, high-quality ballpoint pen tips. China famously celebrated in 2017 that they reached that level of manufacturing excellence.

Producing anything real requires a painstaking process of theory/hypothesis formation, experiment design, experiment execution, and slow iterative improvement. Many physical and chemical processes cannot be accelerated artificially. There is a reason why it takes 5-8 weeks or longer to make a wafer of chips.

The success of of systems such as AlphaGo depend on the fact that all the rules of the game of Go are fixed in time, and known, and the fact that evaluating the quality of a position is cheap and many different future games can be simulated cheaply and efficiently.

None of this is true for reality: 

1. Simulating reality accurately and cheaply is not a thing. We cannot simulate even simple parts of reality to a high degree of accuracy (think of a water faucet with turbulent flow splashing into a sink). 
2. The rules for reality are not known in advance. Humanity has created some good approximations of many rules, but both humanity and a superintelligence still need to create new approximations of the rules by careful experimentation and step-wise refinement.
3. The rules for adversarial and competitive games (such as a conflict with humanity) are not stable in time.
4. Evaluating any experiment in reality has significant cost, particularly to an AI.

A thought experiment I often use for this is: 

Let us assume that scaling is all you need for greater intelligence. If that is the case, Orcas or Sperm Whales are already much more intelligent than the most intelligent human, so perhaps an Orca or a Sperm Whale is already a superintelligence. Now imagine an Orca or Sperm Whale equipped with all written knowledge of humanity and a keyboard with which to email people. How quickly could this Orca or Sperm Whale devise and execute a plot to kill all of humanity?

People that focus on fast takeoff scenarios seem to think that humanity has achieved the place it has by virtue of intelligence alone. Personally, I think there are at least three things that came together: Bipedalism with opposable thumbs, an environment where you can have fire, and intelligence.

If we lacked any of the three, we would not have built any of our tech. Orcas and Sperm Whales lack thumbs and fire, and you can’t think yourself to world domination.

Superintelligence will also be bound by fundamental information-theoretic limits

The assumption that superintelligences can somehow simulate reality to arbitrary degrees of precision runs counter to what we know about thermodynamics, computational irreducibility, and information theory.

A lot of the narratives seem to assume that a superintelligence will somehow free itself from constraints like „cost of compute“, „cost of storing information“, „cost of acquiring information“ etc. - but if I assume that I assume an omniscient being with infinite calculation powers and deterministically computational physics, I can build a hardcore version of Maxwells Demon that incinerates half of the earth by playing extremely clever billards with all atoms in the atmosphere. No diamandoid bacteria (whatever that was supposed to mean) necessary.

The reason we cannot build Maxwells Demon, and no perpetuum mobile, is that there is a relationship between information theory and thermodynamics, and nobody, including no superintelligence, will be able to break it.

Irrespective of whether you are a believer or an atheist, you cannot accidentally create capital-G God, even if you can build a program that beats all primates on earth at chess. Cue reference to the Landauer principle here.

Conflicts (such as an attempt to kill humanity) have no zero-risk moves

Traditional wargaming makes extensive use of random numbers - units have a kill probability (usually determined empirically), and using random numbers to model random events is part and parcel for real-world wargaming. This means that a move “not working”, something going horrendously wrong is the norm in any conflict. There are usually no gainful zero-risk moves; e.g. every move you make does open an opportunity for the opponent.

I find it somewhat baffling that in all the X-risk scenarios, the superintelligence somehow finds a sequence of zero-risk or near-zero risk moves that somehow yield the desired outcome, without humanity finding even a shred of evidence before it happens.

A more realistic scenario (if we take the far-fetched and unrealistic idea of an actual synthetic superintelligence that decides on causing humans harm for granted) involves that AI making moves that incur risk to the AI based on highly uncertain data. A conflict would therefore not be brief, and have multiple interaction points between humanity and the superintelligence.

Next-token prediction cannot handle Kuhnian paradigm shifts

Some folks have argued that next-token prediction will lead to superintelligence. I do not buy it, largely because it is unclear to me how predicting the next token would deal with Kuhnian paradigm shifts. Science proceeds in fits and bursts; and usually you stay within a creaky paradigm until there is a „scientific revolution“ of sorts. The scientific revolution necessarily changes the way that language is produced — e.g. a corpus of all of human writing prior to a scientific revolution is not a good representation of the language used after a scientific revolution - but the LLM will be trained to mimic the distribution of the training corpus. People point to in-context learning and argue that LLMs can incorporate new knowledge, but I am not convinced of that yet - the fact that all current models fail at generating a sequence of words that - when cut into 2-tuples - occur rarely or never in the training corpus shows that ICL is extremely limited in the way that it can adjust the distribution of LLM outputs.

Enough for today. Touch some grass, build some stuff

In theory, theory equals practice. In practice it doesn't. Stepping out of the theoretical realm of software (where generations of EE and chip engineers sacrificed their lives to give software engineers an environment where theory is close to practice most of the time) into real-world things that involve dust, sun, radiation, and equipment chatter is a sobering experience that we should all do more often. It's easy to devolve into scholasticism if you're not building anything.

Some experiments to help me understand Neural Nets better, post 1 of N

While I have been a sceptic of using ML and AI in adversarial (security) scenarios forever, I also quite like the fact that AI/ML has become important, if only to make me feel like my Math MSc (and abortive Math PhD) were not a waste of time.

I am a big proponent of "bottom-up" mathematics: Playing with a large number of examples to inform conjectures to be dealt with later. I tend to run through many experiments to build intuition; partly because I have crippling weaknesses when operating purely formally, partly because most of my mathematics is somewhat "geometric intuition" based -- e.g. I rely a lot on my geometric intuition for understanding problems and statements.

For a couple years I've wanted to build myself a better intuition about what deep neural networks actually "do". There are folks in the community that say "we cannot understand them", and folks that say "we believe in mechanistic interpretability, and we have found the neuron to recognize dogs"; I never found either statement to be particularly convincing.

As a result, earlier this year, I finally found time to take a pen, pencil, and wastebasket and began thinking a bit about what happens when you send data through a neural network consisting of ReLU units. Why only ReLUs? Well, my conjecture is that ReLUs are as good as anything, and they are both reasonably easy to understand and actually used in practical ML applications. They are also among the "simplest examples" to work with, and I am a big fan of trying the simple examples first.

This blog post shares some of my experiments and insights; I called it the "paper plane or origami perspective to deep learning". I subsequently found out that there are a few people that have written about these concepts under the name "the polytope lens", although this seems to be a fringe notion in the wider interpretability community (which I find strange, because - unsurprisingly - I am pretty convinced this is the right way to think about NNs).

Let's get started. In order to build intuition, we're going to work with a NN that is supposed to learn a function from R^2 to R - essentially learning a grayscale image. This has several advantages:

1. We can intuitively understand what the NN is learning.
2. We can simulate training error and generalisation errors by taking very high-resolution images and training on low-resolution samples.
3. We stay within the realm of low-dimensional geometry for now, which is something most of us have an intuitive understanding of. High dimensions will create all sorts of complications soon enough.

Let's begin by understanding a 2-dimensional ReLU neuron - essentially the function f(x, y) = max( ax + by + c, 0) for various values of a, b, and c.

This will look a bit like a sheet of paper with a crease in it:

How does this function change if we vary the parameters a, b, or c? Let's begin by varying a:

Now let's have a look at varying b:

And finally let's have a look at varying c:

So the parameters a, b, c really just decide "in which way" the plane should be folded / creased, and the steepness and orientation of the non-flat part. It divides the plane into halfspaces; the resulting function is 0 on one half-plane and linear (respectively affine) on the other.

As a next step, let's imagine a single-layer ReLU network that takes the (x,y) coordinates of the plane, and then feeds it into 10 different ReLU neurons, and then combines the result by summing them using individual weights.

The resulting network will have 3 parameters to learn for each neuron: a, b, and c. Each "neuron" will represent a separate copy of the plane that will then be combined (linearly, additively, with a weight) into the output function. The training process will move the "creases" in the paper around until the result approximates the desired output well.

Let's draw that process when trying to learn the picture of a circle: The original is here:

This shows us how the network tries to incrementally move the creases around so that on each of the convex areas that are created by the creases, it can choose a different affine function (with the condition that on the "creases" the functions will take on the same value).

Let's do another movie, this time with a higher number of first-layer neurons - 500. And let's see how well we will end up approximating the circle.

Aside from being mesmerizing to watch, this is also kinda intriguing and raises a bunch of questions:

1. I don't understand enough about Adam as an optimizer to understand where the very visible "pulse" in the optimization process is coming from. What's going on here?
2. I am pretty surprised by the fact that so many creases end up being extremely similar -- what would cause them to bundle up into groups in the way they do? The circle is completely rotation invariant, but visually the creases seem to bunch into groups much more than random distribution would suggest. Why?
3. It's somewhat surprising how difficult it appears to be to learn a "sharp" edge, the edge between white and black in the above diagram is surprisingly soft. I had expected it to be easier to learn to have a narrow polytope with very large a/b constants to create a sharp edge, somehow this is difficult? Is this regularization preventing the emergence of sharp edges (by keeping weights bounded)?
   

Clearly, there's work to do. For now, some entertainment: Training the same 500-neuron single-layer network to learn to reproduce a picture of me with a face full of zinc sunscreen:

It's interesting (perhaps unsurprising) that the reproduced image feels visually like folded paper.

Anyhow, this was the first installment. I'll write more about this stuff as I play and understand more.
Steps I'll explain in the near future:

1. What happens as you deepen your network structure?
2. What happens if you train a network on categorical data and cross-entropy instead of a continuous output with MSE?
3. What can we learn about generalization, overfitting, and overparametrization from these experiments?

See you soon.

The end of my Elastic/optimyze journey ...

Hey all,

== tl;dr ==

Today is my last day at Elastic. I'll take an extended break and focus on rest, family, health, writing, a bit of startup mentoring/investing, and some research - at least for a while.

I'm thankful for my great colleagues and my leadership at Elastic - y'all are stellar, even if I was often grumbly about some technical or architectural issues. I'll also miss the ex-optimyze team a lot; you were the best team anyone doing technically sophisticated work could wish for - great individuals, but in sum greater than the parts. I think the future for the tech we built is bright, particularly in light of the recent Otel events :)

========

Extended Version:

Today is my last day at Elastic, and with that, the last day of my journey with optimyze. I am leaving with a heavy heart, and complicated emotions. The 5 years of optimyze (3 years optimyze, 2 years optimyze-integration-into-Elastic) were intense - moderately intense on the work front, but extremely intense on the life front. Fate somehow managed to cram a lot of the ups and downs of midlife into a very small number of years.

A timeline:

1. I left Google on the 31st of December 2018, and started optimyze.cloud in February 2019. I was highly motivated by the idea of building a company that aligns my ecological, economic, and technical interests. I visited the RSA conference in SF in spring 2019 to network and get people interested in our "cut-of-savings" consulting approach. I met Corey Quinn for coffee, and to this day much appreciate all the sage advice he had (even if I had to ignore some and learn the hard lesson myself).
2. In May 2019, I was elated to (finally!) become a father for the first time.
3. During 2019, my co-founder Sean and me mostly spent our time trying to get our "cut-of-savings" consulting business of the ground, only to be thwarted by the unfortunate combination that (a) companies nimble enough to do it were too small to make it worth it, and (b) companies big enough to make it worth it couldn't figure out how to make the contract work from a legal and accounting perspective.
   We did a few small gigs with friendly startups, and realized in late summer that a zero-instrumentation, multi-runtime, fleet-wide profiler was sorely missing as a product. We also realized that with BOLT making progress, there'd be real value in being a SaaS that sits on profiling data from different verticals. Hence the vision for optimyze.cloud as a product company was born.
4. By late 2019, we had a prototype for unwinding C/C++ stacks using .eh_frame, and Python code, both from eBPF. We knew we could be really zero-friction in deployment, which made us very happy and excited.
5. We decided to raise funding, and did so over the winter months - with the funding wire transfer finally hitting our (Silicon Valley Bank) account some time in early 2020. We started building, and hiring what would turn out the best team I've ever worked on.
6. We had a working UI and product by late fall 2020, and the first in-prod deployments around the same time. One particular part of the stack was too slow (a particular query that we knew we'd need to move to a distributed K/V store, but hadn't done yet), and we spent the next few months rebuilding that part of the stack to use Scylla.
7. We made some very bad calls on the investor relations front, I foolishly stumbled into a premature, fumbled, and retrospectively idiotic fundraise, into the middle of which my second child was born and the first acquisition offers came in.
8. We launched Prodfiler in August 2021, to great acclaim and success. People loved the product, they loved the frictionless deployment, they loved the fact that all their stack traces were symbolized out of the box etc. - the product experience was great.
9. In mid-October, we were acquired by Elastic with the closing date November 1st. My mother had a hip surgery from which complications arose, which led to her being transferred into an ICU.
   The day the deal closed, my mother fell into a coma, and she would never wake up again. I spent the next weeks shuttling back and forth between Zurich (where my wife and my two kids were) and Essen, Germany, to spend time bedside in the ICU.
   My mother died in the morning hours of Jan 1st 2022, a few hours after the fireworks.
10. My elderly father needed a lot of help dealing with the aftermath; at the same time the transition into the Elastic tech stack was technically challenging to pull off.
11. In Summer 2022, my father stumbled after a small leg surgery, fell, and hit his head; after some complications in the German medical system, it became clear that the injury had induced dementia. We transferred him to a specialist hospital in Berlin and ultimately to a care home close to my brother's family. Since then, I've been shuttling back and forth to see him often.
12. After two years of hard work at Elastic, we finally managed to launch our product again in fall 2023.

So the entire thing was 5 years, in which I had two children, started a company, hired the best team I've known, launched a product I was (and am) immensely proud of, then lost my mother, most of my father ... and "reluctantly let go" of the company and product.

The sheer forces at play when you cram so much momentum into such a short time-frame will strain everybody; and they will strain everybody's support system. I'm extremely grateful for my entire support system, in particular my brother. I don't know how I would've fared without him, but I hope my kids will have as good a relationship with each other as I do with my brother.

I'm also grateful to the folks at Elastic and the optimyze team, who were extremely supportive and understanding as I was dealing with complications outside of work.

I'm proud that we managed to build, I am also proud that we managed to port it to the Elastic stack and re-launch it. Even after more than 2 years focused on porting the back-end, our profiler remains ahead of the competition. I'm optimistic about what Elastic and the team can build on top of our technology, in particular with OTel profiling moving toward reality.

At the same time, I am pretty spent. My productivity is nowhere near where I expect it to be (it never is - I have difficulty accepting that I am a finite human - but the gap is bigger than usual), and this leads to me having difficulty switching off: When I feel like I am not getting the things I want to get done done, my brain wants to compensate by working more - which is rarely the right step.

So, with a heavy heart, I decided that I will take an extended break. It's been intense, and emotional, and I need some time to rest and recover, and accompany my father on his last few steps into the darkness (or light?). 2019 and 2020 were among the happiest years of my life, the last chunk of 2021 and most of 2022 the most difficult parts of my life. 2023 was trending up, and I expect things to continue trending up for the foreseeable future.

I have planned to do a bit of writing (I think having done two companies, one bootstrapped and one with VC money, gives me a few things I'd like to pass on), perhaps a bit of angel investing or VC scouting, perhaps a bit of consulting where things of particular interest arise - but mostly, I intend to stretch, breathe, be there for my kids, and get a clear view of the horizon.

A list of factors that act(ed) as drag on the European Tech/Startup scene

This post is an adaption of a Twitter thread where I listed the various factors that in my experience led to a divergence of the trajectories of the US tech industry around Silicon Valley (SV) and the tech industry in Europe. Not all of these factors are current (some of the cultural ones are less pronounced today than they used to be), and some of them could be relatively easily fixable.

I'll add a separate post on policy suggestions at a later point.

I should also note that there's many great things about Europe -- I still live here, I'd build my next company here, and I don't think I'd ever want to migrate to SV. I'll also write about the advantages in the future.

Now, on to the list, which was spawned by a thread with @martin_casado and @bgurley on the website previously known as Twitter.

1. Cultural factors: When I was growing up in the 90s, there was significant uncertainty in the labor market, and one way to achieve economic security was seeking a government job. In many European countries, running a limited liability construct into insolvency effectively bans you from running another one in the foreseeable future. The mentality of "start a company in your 20s, and if you fail, you can either try again or get a job" wasn't a thing. So we are operating from a risk-averse base, due to a labor market with then-sluggish job creation and strong incumbent effects. (Bert Hubert has written a more extensive article on the cultural factors here).
2. A terrifyingly fragmented market, along legal, linguistic, and cultural lines. Imagine every US state had its own language, defense budget, legal system, tax system, culture, employment law etc. - in the US, you build a product and you tap into a market of 340m people. The biggest market in Europe is Germany at 80m, not even a quarter of the size. Then France (65m), Italy (59m), Spain (47m), and then things fragment into a long tail. By the time you hit 340m customers, you're operating in 9-10 countries, 7+ languages and legal systems etc.
3. Equally fragmented capital markets that are individually much smaller. Take the US stock market and cut it into 10+ pieces. This has knock-on effects for IPOs: IPOs, when they happen, tend to be much smaller. Raising large amounts of capital is more difficult, while big wins are smaller. This has terrible knock-on effects all the way down to seedstage VCs: If the power law home run you're angling for is 1/10th the size of the home run in the US, early stage investors need to be way more risk averse. You can see this even today where most European VC funds will offer less money at worse terms than their US counterparts. It was much worse in 2006-2007, when the Samwers were almost the only game in town for VC in the EU.
   Smaller IPOs also mean that it is comparatively much more attractive to sell to an existing (US-based) giant.
4. The absence of a DARPA to shoulder fundamental research risks in technology. Different stages of R&D require different investors. The government is in the strange situation that they can indirectly benefit from investments without having an ownership stake because it gets to tax GDP. That means at the extremely high risk end of R&D, fundamental research, it can afford to just finance many many long shots blindly and (comparatively) simply, as it doesn't need to track ownership. So how do you fund fundamental R&D without it devolving into scholasticism? Interestingly, the most basic test ("can I use this to cause some damage") is already helpful. Europe's defense sector has never since WW2 grasped it's role in advancing technology, and it's terribly fragmented, underfunded, and can't do much research. DARPA has financed the early-stage development of many enabling technologies. Having a guaranteed customer (DoD) for high risk research has enabled better and higher risk-taking, and had large downstream effects.
5. Terrible legislation with regards to employee stock options. People talk about how many big companies in Europe are family-owned as if that is something good. It's also a symptom of legal systems that make (or made) it terribly difficult to give lots of equity to early employees. This is slowly changes through concerted lobbying, but it is still difficult in most jurisdictions, and not unified at all.
6. The way the EU is constructed where the EU gives a directive and each country implements it's own flavor is worst-case for legal complexity. Imagine if every state got to re-implement its own flavor of each federal law.
7. Founder Brain Drain. Why would an ambitious founder not go to where the markets are bigger, capital is easier to raise on better terms, and incentivizing early employees is easier?
8. Ecosystem effects permit risk-taking by employees in SV. SV has such strong demand for talent that an employee can "take risks" on early stage startups because the next job is easy to get. If you live in a place with just 1-2 big employers, leaving with intent to return is riskier.
9. Network effects and path dependence. The fragmentation of the market led to smaller players in search and ads that then sold to larger US-based players. Without the deep revenue streams, no European player had the capital or expertise to go into cloud. As a result, there is no European player with enough compute, or datasets, or capital to effectively compete in cloud or AI. China has homegrown players, even Russia has to some extent, Europe's closest equivalent are OVH and Hetzner, which sell on price, not on higher-level services.
10. GDPR after effects: EUparl saw that in situations where US states are fragmented they can act as a standards body, and there's a weird effect of "if we cannot be relevant through tech, we can still be relevant through shaping the legal landscape", and that's what leads to this terrible idea of "Europe as regulatory superpower", where it is more important for members of EUparl to have done "something" than having done "something right" - a mentality that seems to prefer bad regulation over no regulation, when good regulation would be needed. GDPR led to higher market concentration in Ads, which arguably undermines privacy in a different way, and it's imposed huge compliance and convenience cost on everybody. But in EUparl it's celebrated as success, because hey, for once Europe was relevant (even if net effects are negative).
11. Pervasive shortsightedness among EU national legislators, undermining the single market and passing poor laws with negative side effects for startup and capital formation. The best example is Germans "exit tax": Imagine you are an Angel Investor in the US but if you move out of state it triggers immediate cap gains on all your illiquid holdings/Angel Investments at the valuation of the last round. It essentially means you can't angel invest if you don't know if you'll have to move in the next 8-10 years because you don't know if you can afford the tax bill. It's hair-raisingly insane, and likely illegal under EU rules, but who wants to fight the German IRS in European court?

I think these are the most important factors that come to mind. I'll add more if I remember more of them.

Also, given that this post has a strong resonance with extreme "anti government" and "libertarian" types, please be aware that I am very much on a different area of the political spectrum (centre-left, somewhere where the social democrats used to reside historically in Germany). I am strongly in favor of good and competent regulation to ensure markets function, competition works, and customers are protected.

Book Review: "This Is How They Tell Me the World Ends"

This blog post is a review of the book "This Is How They Tell Me the World Ends" by Nicole Perlroth. The book tries to shed light on the "zero day market" and how the US government interacts in this market, as well on various aspects of nation-to-nation "cyberwarfare".

I was excited to see this book come out given that there are relatively few hats in this field I have not worn. I have worked in information security since the late 1990's; I was part of a youth culture that playfully pioneered most of the software exploitation techniques that are now used by all major military powers. I have run a business that sold technology to both defenders and offensive actors. I have written a good number of exploits and a paper clarifying the theoretical foundations for understanding them. I have trained governments and members of civil society on both the construction and the analysis of exploits, and on the analysis of backdoors and implants. I have spent several months of my life reading the disassembled code of Stuxnet, Duqu, and the Russian Uroburos. I spent half a decade at Google on supporting Google's defense against government attackers; I spent a few additional years in Project Zero trying to nudge the software industry toward better practices. Nowadays, I spend my time on efficiency instead of security.

I have always been close to, but never part of, the zero-day market. My background and current occupation give me a deep understanding of the subject, while not tying me economically to any particular perspective. I therefore feel qualified like few others to review the book.

"This Is How They Tell Me the World Ends" tackles an important question: What causes the vulnerability of our modern world to "cyberattacks"? Some chapters cover various real-world cyberattacks, some chapters try to shed light on the "market for exploits", and the epilogue of the book discusses ideas for a policy response.

The author managed to get access to a fantastic set of sources. Many things were captured on the record that were previously only discussed on background. Several chapters recount interviews with former practitioners in the exploit market, and these chapters provide a glimpse into the many fascinating and improbable personalities that make up the field. This is definitely a strong asset of the book.

Given the exciting and impactful nature of the "cyberwar" subject, the many improbable characters populating it, and the many difficult and nuanced policy questions in the field, the level of access and raw material the author gathered could have been enough for a fantastic book (or even two).

Unfortunately, "This Is How They Tell Me the World Ends" is not a fantastic book. The potential of the source material is diluted by a large number of inaccuracies or even falsehoods, a surprising amount of ethnocentricity and US-American exceptionalism (that, while being a European, I perceived to border on xenophobia), a hyperbolic narration style, and the impression of facts bent to support a preconceived narrative that has little to do with reality.

For the layperson (presumably the target audience of this book) the many half-truths and falsehoods make the book an untrustworthy guide to an important and difficult topic. For the expert, the book may be an entertaining, if jarring read, provided one has the ability to dig through a fair bit of mud to find some gold. I am confident that the raw material must be great, and where it shines through, the book is good.

Inaccuracies and Falsehoods

The topic is complex, and technical details can be difficult to get right and transmit clearly. A book without any errors cannot and should not be expected, and small technical errors should not concern the reader. That said, the book is full of severe and significant errors - key misunderstandings and false statements that are used as evidence and to support conclusions - and those do raise concerns.

I will highlight a few examples of falsehoods or misleading claims. I only found those falsehoods that overlapped with expertise of mine; extrapolating from this, I am afraid that there may be many more in the book.

The following examples are from the first third of the book; and they are illustrative of the sort of mistakes throughout: Facts are either twisted or exaggerated to the point of becoming demonstrably false; and these twists and exaggerations seem to always happen in support of a narrative that places an unhealthy focus on zero-days.

First, one of the more egregious falsehoods is the claim that NSA hacked into Google servers to steal data:

... the agency hacked its way into the internal servers at companies like Google and Yahoo to grab data before it was encrypted.

This simply did not happen. As far as anyone in the industry knows, in the case of Google, unencrypted network connections between datacenters were tapped. This may sound inconsequential, but undermines the central "zero days are how hacking happens" theme of the book.

Second, the entire description of zero-days is full of false claims and hyperbole:

Chinese spies used a single Microsoft zero-day to steal some of Silicon Valley's most closely held source code.

This alludes to the Aurora attacks on Google; but anyone that knows Google's internal culture knows that source code is not most closely held by design. Google always had a culture where every engineer could roam through almost all the code to help fix issues.

...Once hackers have figured out the commands or written the code to exploit it, they can scamper through the world's computer networks undetected until the day the underlying flaw is discovered

This is simply not true. While a zero-day exploit will provide access to a given machine or resource, it is not a magic invisibility cloak. The Chinese attackers were detected, and many other attackers are routinely detected in spite of having zero-day exploits.

...Only a select few multinationals are deemed secure enough to issue the digital certificates that vouch (...) that Windows operating system could trust the driver  (...) Companies keep the private keys needed to abuse their certificates in the digital equivalent of Fort Knox.

This section is at best misleading: The driver in question was signed with a stolen JMicron "end-entity" certificate. There are thousands of those, all with the authority to sign device drivers to be trusted, and the due diligence to issue one used to be limited to providing a fax of an ID and a credit card number.

The "select few multinationals" Perlroth writes about here are the certificate authorities that issue such "end-entity" certificates. It is true that a CA is required to keep their keys on a hardware security module (a very high-security setup), and that the number of CAs that can issue driver-signing certificates is limited (and falling).

The text makes it appear as if a certificate from a certificate authority (and hence from a hardware security module) had been stolen. This is simply false. End-entity certificates are issued to hardware vendors routinely, and many hardware vendors play fast and loose with them.

(It is widely rumored - but difficult to corroborate - that there used to be a thriving black market where stolen end-entity certificates were traded a few years ago; the going rate was between $30k and $50k if I remember correctly.

Ethnocentricity and US exceptionalism

As a non-US person, the strangest part of the book was the rather extreme ethnocentricity of the book: The US is equated with "respecting human rights", everything outside of the US is treated as both exotic and vaguely threatening, and the book obsesses about a "capability gap" where non-US countries somehow caught up with superior US technology.

This ranges from the benign-but-silly (Canberra becomes the "outback", and Glenn Greenwald lives "in the jungles of Brazil" - evoking FARC-style guerillas, when - as far as I am informed - he lives in a heavily forested suburb of Rio) to seriously impacting and distorting the narrative.

The author seems to find it unimaginable that exploitation techniques and the use of exploits are not a US invention. The text seems to insinuate that exploit technologies and "tradecraft" were invented at NSA and then "proliferated" outward to potentially human-rights-violating "foreign-born" actors via government contractors that ran training classes.

This is false, ridiculous, and insulting on multiple levels.

First off, it is insulting to all non-US security researchers that spent good parts of their lives pioneering exploit techniques.

The reality is that the net flow of software exploitation expertise out of NSA is negative: Half a generation of non-US exploit developers migrated to the US over the years and acquired US passports eventually. The US exploit supply chain has always been heavily dependent on "foreign-born" people. NSA will enthusiastically adopt outside techniques; I have yet to learn about any exploitation technique of the last 25 years that "leaked" out of NSA vs. being invented outside.

The book's prologue, when covering NotPetya, seems to imply that Russia had needed the Shadowbrokers leaks - ("American weapons at its disposal") - to cause severe damage. Anybody with any realistic visibility into both the history of heap exploitation and the vulnerability development community knows this to be absolutely wrong.

Secondly, it seems to willfully ignore recent US history with regards to human rights. Somehow implying that the French police or the Norwegian government have a worse human rights track record than the US government - which unilaterally kills people abroad without fair trial via the drone-strike program, relatively recently stopped torturing people, and keeps prisoners in Guantanamo for 15+ years by having constructed a legal grey zone outside of the Geneva Conventions - is a bit rich.

In the chapter on Argentina, Ivan Arce calls the author out on her worldview (which was one of my favorite moments in the book), but this seems to have not caused any introspection or change of perspective. This chapter also reveals an odd relationship to gender: The narrative focuses on men wreaking havoc, and women seem to exist to rein in the out-of-control hackers. Given that there are (admittedly few, but extremely capable) women and non-binary folks active in the zero-day world, I find this narrative puzzling.

There is also an undercurrent that everything bad is caused by nefarious foreign intervention: The author expresses severe doubts that the 2016 US election would have had the same outcome without "Russian meddling", and in the Epilogue writes "it is now easier for a rogue actor to (...) sabotage (...) the Boeing 737 Max", somehow managing to link a very US-American management failure to vague evil forces.

In its centricity on the US and belief in US exceptionalism, its noticeable grief about the 2016 US election, and the vague suspicion that everything bad must have a foreign cause, the reader learns more about the mindset of a certain subset of the US population than about cybersecurity or cyberwarfare.

Hyperbolic language

The book is also made more difficult to read by constant use of hyperbolic language. Exploits are capable of "crashing Spacecraft into earth", "detonated to steal data", and things always need to be "the most secure", "the most secret", and so forth. The book would have benefitted from the editor-equivalent of an equalizer to balance out the wording.

The good parts

There are several things to like about the book: The chapters that are based on interviews with former practitioners are fun and engaging to read. The history of software exploits is full of interesting and unorthodox characters, and these chapters provide a glimpse into their world and mindsets.

The book also improves as it goes on: The frequency of glaring falsehoods seems to decrease - which lets the fact that it is generally engaging come through.

Depending on what one perceives the thesis of the book to be, one can also argue that the book advances an important point. The general subject - "how should US government policy balance offensive and defensive considerations" - is a deep and interesting one, and there is a deep, important, and nuanced discussion to be had about this. If the underlying premise of the book is "this discussion needs to be had", then that is good. The book seems to go much beyond this (reasonable) premise, and seems to mistakenly identify the zero-day market as the root cause of pervasive insecurity.

As a result, the book contributes little of utility to a defensive policy debate. The main drivers of the cyber insecurity are hardly discussed until the Epilogue: The economic misincentives that cause the tech industry to earn hundreds of billions of dollars from creating the vulnerabilities in the first place (for every million earned through the sale of exploits, an order of magnitude or two more is earned through the sale of the software that creates the security flaw), and the organisational misincentives that keep effective regulation from arising (NSA - rightly - has neither mission or authority to regulate the tech industry into better software, so accusing them of not doing so is a bit odd). By placing too much emphasis on governments knowing about vulnerabilities, the book distracts from the economic forces that create a virtually infinite supply of them.

The Epilogue (while containing plenty to disagree with) was one of the stronger parts of the book. The shortness makes it a bit shallow, but it touches on many points that warrant a serious discussion. (Unfortunately, it again insinuates that "ex-NSA hackers tutor Turkish Generals in their tradecraft"). If anything, the Epilogue can be used as a good (albeit incomplete) list of topics to discuss in any cybersecurity policy class.

Concluding thoughts

I wish the book realized more of the potential that the material provided. The debate about the policy trade-offs for both offense and defense needs to be had (although there is less of a trade-off than most people think: Other countries have SIGINT agencies that can do offense, and defensive agencies focused on improving the overall security level of society; fixing individual bugs will not fix systemic misincentives), and a good book about that topic would be very welcome.

Likewise, a book that gives a layperson a good understanding of the zero-day trade and the practitioners in the trade would be both useful and fascinating.

The present book had the potential to become either of the above good books - the first one by cutting large parts of the book and expanding the Epilogue, the second one by rigorous editing and sticking to the truth.

So I regret having to write that the present book is mostly one of unfulfilled potential, and that the layperson needs to consult experts before taking any mentioned "fact" in the book at face value.

The missing OS

Preface:

When I joined Google in 2011, I quoted a quip of a friend of mine:

"There are roughly one and a half computers in the world, and Google has one of them."

The world has changed quite a bit since 2011, and there may possibly be half a dozen computers in the world now. That said, for the following text to make sense, when I say "the computer", I mean a very large assembly of individual machines that have been connected to make them act like one computer.

Actual blog post:

The tech landscape of modern microservice deployments can be confusing - it is fast-changing, with a proliferation of superficially similar projects claiming to do similar things. Even to me as someone fairly deeply into technology, it isn't always clear what precise purpose the different projects serve.

I've quipped repeatedly about "Datacenter OS" (at least here and here), and mused about it since I first left Google for my sabbatical in 2015. I recently had the chance to chat with a bunch of performance engineers (who sit very much at the crossing between Dev and Ops), and they reminded me to write up my thoughts. This is a first post, but there may be more coming (particularly on the security models for it).

Warning: This post is pure, unadulterated opinion. It is full of unsubstantiated unscientific claims. I am often wrong.

I claim the following:

When we first built computers, it took a few decades until we had the first real "operating systems". Before a 'real' OS emerged, there were a number of proto-OS -- collections of tools that had to be managed separately and cobbled together. There were few computers overall in the world, and if you wanted to work on one, you had to work at a large research institution or organization. These machines ran cobbled-together OSs that were unique to that computer.

Since approximately 2007, we're living through a second such period: The "single computer" model is replaced with "warehouse-sized computers". Initially, few organizations had the financial heft to have one of them, but cloud computing is making "lots of individual small computers" accessible to many companies that don't have a billion of cash for a full datacenter.

The hyperscalers (GOOG, FB, but also Tencent etc.) are building approximations to a "proto-datacenter-OS" internally; Amazon is externalizing some of theirs, and a large zoo of individual components for a Datacenter-OS exist as open-source projects.

What does not exist yet is an actual complete DatacenterOS that "regular" companies can just install.

There is a "missing OS" - a piece of software that you install on a large assembly of computers, and that transform this assembly of computers into "one computer".

What would a "Datacenter OS" consist of? If you look at modern tech stacks, you find that there is a surprising convergence - not in the actual software people are running, but in the "roles" that need to be filled. For each role, there are often many different available implementations.

The things you see in every large-scale distributed infrastructure are:

1. Some form of cluster-wide file system. Think GFS/Colossus if you are inside Google, GlusterFS or something like it if you are outside. Many companies end up using S3 because the available offerings aren't great.
   
2. A horizontally scalable key-value store. Think BigTable if you are inside Google, or Cassandra, or Scylla, or (if you squint enough) even ElasticSearch.
3. A distributed consistent key-value store. Think Chubby if you are inside Google, or etcd if you are outside. This is not directly used by most applications and mostly exists to manage the cluster.
4. Some sort of pub/sub message queuing system. Think PubSub, or in some sense Kafka, or SQS on AWS, or perhaps RabbitMQ.
5. A job scheduler / container orchestrator. A system that takes the available resources, and all the jobs that ought to be running, and a bunch of constraints, and then solves a constrained bin-packing optimization problem to make sure resources are used properly. Think Borg, or to some extent Kubernetes. This may or may not be integrated with some sort of MapReduce-style batch workload infrastructure to make use of off-peak CPU cycles.

I find it very worthwhile to think about "what other pieces do I have on a single-laptop-OS that I really ought to have on the DatacenterOS?".

People are building approximations of a process explorer via Prometheus and a variety of other data collection agents.

One can argue that distributed tracing (which everybody realizes they need) is really the Datacenter-OS-strace (and yes, it is crucially important). The question "what is my Datacenter-OS-syslog" is similarly interesting. 

A lot of the engineering that goes into observability is porting the sort of introspection capabilities we are used to having on a single machine to "the computer".

Is this "service mesh" that people are talking about just the DatacenterOS version of the portmapper?

There are other things for which we really have no idea how to build the equivalent. What does a "debugger" for "the computer" look like? Clearly, single-stepping on a single host isn't the right way to fix problems in modern distributed systems - your service may be interacting with dozens of other hosts that may be crashing at the same time (or grinding to a halt or whatever), and re-starting and single-stepping is extremely difficult.

Aside from the many monitoring, development, and debugging tools that need to be rebuilt for "the computer", there are many other - even more fundamental - questions that really have no satisfactory answer. Security is a particularly uncharted territory:

What is a "privileged process" for this computer? What are the privilege and trust boundaries? How does user management work? How does cross-service authentication and credential delegation work? How do we avoid re-introducing literally every logical single-machine privilege escalation that James Forshaw describes in his slides into our new OS and the various services running there? Is there any way that a single Linux Kernel bug in /mm does not spell doom for our entire security model?

To keep the post short:

In my opinion, the emerging DatacenterOS is the most exciting thing that has happened in computer science in decades. I sometimes wish I was better at convincing billionaires to give me a few hundred million dollars to invest in interesting problems -- because if there is a problem that I think I'd love to work on, it'd be a FOSS DatacenterOS - "install this on N machines, and you have 'a computer'".

A lot of the technological landscape is easier to understand if one asks the question: What function in "the computer" does this particular piece of the puzzle solve? What is the single-machine equivalent of this project?

This post will likely have follow-up posts, because there are many more ill-thought-out ideas I have on the topic:

• Security models for a DatacenterOS
• Kubernetes: Do you want to be the scheduler, or do you want to be the OS? Pick one.
• How do we get the power of bash scripting, but for a cluster of 20k machines?
  
  

 

 

 

My Twitter-Discussion-Deescalation Policy

Twitter is great, and Twitter is terrible. While it enables getting in contact and starting loose discussions with a great number of people, and while it has certainly helped me broaden my perspectives and understanding of many topics, it also has a lot of downsides.

Most importantly, Twitter discussions, due to their immediacy of feedback and the fact that everybody is busy, often end up in shouting matches where "learning from each other while discussing a topic" (the actual purpose of a discussion) is forgotten.

Most importantly: Twitter can be very repetitive, and it can be very difficult to convey the context for complex topics - and nobody has time to repeat all the context in each Twitter discussion.

Today, I am recovering from a migraine attack that coincided with my kid having a cranky night, and as a result, I cut a few Twitter discussions short. The people on the receiving end of this "short-cutting" may rightly feel slighted, so I am writing this blog post in preparation for future similar situations.

There are some topics (often related to security or economics) about which I have thought for a reasonably long time. Particularly for security, we're talking about a few decades of hands-on experience with a fairly obsessive work on the topic, both on the theoretical and on the practical side. Rooted in this experience, I sometimes make statements on Twitter. These statements may be in conflict with what other people (you?) may think, and we may engage in a discussion. It is possible, though, that we will reach a point in the discussion where my feeling is "oh, in order to convey my point, I'd now need to spend 25 minutes conveying the context necessary for my point, and I only have a few hours in my day after I deduct sleep and other obligations".

At this point, I need to make a judgement call: Do I invest that time? I also need to make the call without having the most important context: Does the other side care about understanding me at all?

So if we end up in a Twitter discussion, and I reply to you with a link to this blog post at some point, please understand: I have run out of time to spend on this Twitter thread, and I need to cut the discussion short because conveying the necessary context is too time consuming without knowing that this is actually desired, and that our discussion is a mutual learning exercise.

If you very much care about the topic, and about understanding the perspective I have, I will happily schedule a 25-minute video call to discuss in person, and will obviously make an effort to understand your perspective, too. My DM's are open, ping me and I will send you a calendly link.

My self-help guide to making sense of a confusing world

It has become painfully evident over the last decade or so that social media has a somewhat corrosive effect on "truth" and "discussion". There are a variety of reasons for this - many unidentified - but a few factors are:

1. For every opinion, no matter how bizarre, it has become easy to find a community with similar beliefs.
2. The discoverability of almost all information coupled with the shortening of attention spans allows people with strange beliefs to search for information at - at least if only glanced at for 15 seconds - may be interpreted to confirm their strange belief.
3. Algorithms that maximize engagement also maximize enragement -- if the algorithms show me content that draws me into a time-sink discussions with no benefit, they are "winning" (in terms of the metrics against which they are evaluated).
4. The social media platforms favor "immediacy of feedback" vs. taking time to think things through. Social media discussions often devolve into name-calling or rapid-fire quoting of dubious studies without any qualitative understanding - people quote papers and sources they never critically evaluated.

Aside from that, creating false but engagement-producing content has become a veritable industry. Attention can be monetized, so one can earn good money by making up lies that appear credible to some subset of the population. The quality of mainstream reporting has been caught up in this downward spiral.

The result of this is "fake news" and the "death of a consensus on reality"; strange conspiracy theories; and generally many many hours wasted. The problem cuts across class and educational layers; it is simply not true that "only the uneducated" fall prey to internet falsehoods.

Personally, I am terrified of believing things that are not true. I am not quite sure why; but to assuage my fears of misleading myself, I have adapted a number of habits to function as checks on my own beliefs.

By and large, I have found them very useful.

In this blog post, I intend to share my thoughts on how people mislead themselves, and the checks I have decided to apply to my own beliefs. I am not always successful, but this is the bar I try to measure myself against. My hope is that this is helpful for others; perhaps it can help reject false beliefs somewhere.

So let's begin with an underlying assumption I make:

People tend to believe in things that help them.

As a young man I believed that people try to understand a situation, and then form a belief based on that. This is not what I observed in my life. My observation is that people choose beliefs and systems of belief to fulfill a function for them.

My father is born in the 30s in Germany, and as a pre-teen and early teen, he got a front-row seat to watch all adults perform an ideological 180-degree turn in front of him. The question of "how do people adjust their beliefs" has always been important in my discussions with him.

My conclusion is that people are very good at identifying what they want, and what is beneficial to them. They also like to feel good about themselves, and about what they do. Given these constraints, people tend to pick beliefs and systems of belief that ...

• ... allow them to do what they want to do.
• ... allow them to reap benefits.
• ... allow them to feel good about themselves at the same time.

I alluded to this with the sentence "Everybody wants to be the hero of their own story" in my disclosure Rashomon post.

It is crucially important to be aware that belief systems have a functional role for those that believe them. This is why it can be so hard to "convince" anyone of the incorrectness of their belief system: You are asking the person to give up more than a false belief - often, you are asking the person to adjust their view of themselves as being less benign than they like to believe, or you are asking the person to adjust their view in a manner that would cast doubt on their ability to obtain some other benefit.

When I write "people" above, this includes you and me.

Being aware of the functional role of beliefs is hence important when you investigate your own beliefs (more on that later). Trying to believe what makes us feel good is the most fundamental cognitive bias.

So what am I trying to do to counter that bias? Here's my list of 7 habits:

1. Clarify your beliefs
2. Ask about the benefits of your beliefs
3. Seek out original sources
4. Examine evidence for your beliefs and competing hypotheses
5. What new information would change your beliefs?
6. Provide betting odds
7. Discuss for enlightenment, not "winning"

Habit 1: Clarify beliefs

It may sound odd, but it takes conscious effort to turn everyday diffuse "belief" into an actual clearly articulated statement. For me, nothing quite clarifies thoughts like writing them down - often things that appear clear and convincing in my head turn out to be quite muddled and unstructured when I try to put them to paper.

Asking oneself the question "what are my beliefs on a given topic", and trying to write them down coherently, is surprisingly powerful. It forces a deliberate effort to determine what one actually believes, and committing to that belief in writing (at least to oneself).

Habit 2: Ask about the benefits of your beliefs - "am I the baddie?"

Awareness of the functional role of beliefs is important when examining one's own beliefs. Feynman famously said about science that "the first principle is that you must not fool yourself and you are the easiest person to fool".

When examining my own beliefs, it try to ask myself: What benefits does this belief bestow on me? What does this belief permit me to do? How does this belief make me feel good about myself?

It is quite often helpful to actively try to examine alternative narratives in which one casts oneself in a bad light. Trying to maintain our positive image of ourselves is a strong bias; making a conscious effort at examining alternate, unflattering narratives can be helpful.

My wife and me sometimes jokingly play "mock custody battle" - a game where we jokingly try to portray each other as some sort of terrible person and reinterpret each others entire biography as that of a villain - and it is quite enlightening.

Habit 3: Seek out original sources, distrust secondary reporting

Both for things like presidential debates and for scientific papers, the secondary reporting is often quite incorrect. In political debates, it is often much less relevant how the debate went and what happened -- only a small fraction of the population will have witnessed the event. What really counts is the narrative that gets spun around what happened.

You can observe this during election season in the US, where as soon as the debate is over, all sides will try to flood all the talkshows and newscasts with their "narrative" (which has often been pre-determined prior to the debate happening - "He is a flip-flopper. He flipflops." or something along those lines).

Likewise, scientific papers often get grossly misrepresented in popular reporting, but also by people that only superficially read the paper. Reporting is often grossly inaccurate, and if you are an expert on a topic, you will notice that on your topic of expertise, reporting is often wrong; at the same time, we somehow forget about this and assume that it is more accurate on topics where we are not experts (the famous "Gell-Mann amnesia").

A friend of mine (scientist himself) recently forwarded me a list of papers that estimated COVID-19 IFR; one of them reported an IFR of 0%. Closer examination of the contents of the paper revealed that they examined blood from blood donors for antibodies; nobody that died of COVID-19 went to donate blood 2 weeks later, so clearly there were no fatalities in their cohort.

Nonetheless, the paper was cited as "evidence that the IFR is lower than people think".

A different friend sent me a "scientific paper" that purported to show evidence that tetanus vaccines had been laced with a medication to cause infertility. Examining the paper, it was little than assembling a bunch of hearsay; no experimental setup, no controlling, no alternative hypothesis etc. Examining the homepages of the authors revealed they were all strange cranks, peddling various strange beliefs. It was "published" in a "we accept everything" journal.

Acquiring the habit to read original sources is fascinating - there are bookshelves full of books that are quoted widely, mostly by people who have never read them (Sun Tzu, Clausewitz, and the Bible are probably the most common). It is also useful: Getting into the habit of reading original papers helps cut out the middle-man and start judging the results directly; it is also a good motivator to learn a good bit of statistics.

 

Habit 4: Examine evidence for your beliefs, analyze competing hypotheses

Once one's own beliefs are clarified in writing, and one has looked at the primary sources, one can gather evidence for one's belief.

While one does so, one should also make a deliberate effort to gather and entertain competing hypotheses: What other explanations for the phenomenon under discussion exist? What are hypotheses advanced by others?

Given one's own beliefs, and alternate hypotheses, one can look at the evidence supporting each of them carefully.

Two principles help me at this stage to discount less-credible hypotheses (including my own):

• Occam's Razor: Often, the simpler explanation is the more likely explanation
• Structural malice: Malicious individuals are pretty rare, but if an incentive structure exists where an individual can benefit from malice while explaining it away, the tendency is for that to happen.
• Incompetence is much more common than competent malice. The Peter principle and Parkinsons law apply to human organisations.

After this step, I end up forming an opinion - looking at the various competing hypotheses, I re-examine which I find most credible. Often, but not always, it is the beliefs that I declared initially, but with reasonable frequency I have to adjust my beliefs after this step.

Habit 5: What new information would change my opinion?

John Maynard Keynes is often quoted with "When the facts change, I change my mind; what do you do, Sir?". It is worth examining what would be necessary to change one's beliefs ahead of time.

Given my belief on a subject right now, what new information would need to be disclosed to me for me to change my mind?

This is very helpful to separate "quasi-religious" beliefs from changeable beliefs.

Habit 6: Provide betting odds

This is perhaps the strangest, but ultimately one of my more useful points. Over the last years, I have read a bit about the philosophy of probability; particularly De Finetti's "Philosophical Lectures on Probability".

When we speak about "probability", we actually mean two different things: The fraction for a given outcome if we can repeat an experiment many times (coin-flip etc.), and the strength of belief that a given thing is true or will happen in situations where the experiment cannot be repeated.

These are very different things - the former has an objective truth, the second one is fundamentally subjective.

At the same time, if my subjective view of reality is accurate, I will assign good probability values (in the 2nd sense) to different events. ("Good" here means that if a proper scoring rule was applied, I would do well).

Beliefs carry little cost, and little accountability. Betting provides cost for being wrong, and accountability about being wrong.

This means that if I truly believe that my beliefs are correct, I should be willing to bet on them; and through the betting odds I provide, I can quantify the strength of my belief.

For me, going through the exercise of forcing myself to provide betting odds has been extremely enlightening: It forced me to answer the question "how strongly do I actually believe this?".

In a concrete example: Most currently available data about COVID-19 hints at an IFR of between 0.49% and 1% (source) with a p-value of < 0.001. My personal belief is that the IFR is almost certainly >= 0.5%. I am willing to provide betting odds of 3:1 (meaning of you bet against me, you get 3x the payout) for the statement "By the end of 2022, when the dust around COVID-19 has settled, the empirical IFR for COVID-19 will have been greater than 0.5%".

This expresses strong belief in the statement (much better than even odds), but some uncertainty around the estimate in the paper (the p-value would justify much more aggressive betting odds).

(Be aware that these betting odds are only valid for the next 4 days, as my opinion may change).

To sum up: Providing betting odds is a great way of forcing oneself to confront one's own strength of belief. If I believe something, but am unwilling to bet on it, why would that be the case? If I believe something, and am unwilling to provide strong odds in favor of that belief, why is that the case? Do I really believe these things if I am unwilling to bet?

Habit 7: Discuss for enlightenment, not "winning"

When I was young, my father taught me that the purpose of a discussion is never to win, or even to convince. The purpose of a discussion is to understand - the topic under discussion, or the position of the other side, or a combination thereof. This gets lost a lot in modern social media "debates".
 
Social media encourages participating in discussions and arguing a side without ever carefully thinking about one's view on a topic. The memoryless and repetitive nature of the medium allows one to spend countless hours re-hashing the same arguments over and over, without making any advance, and ignoring any thought-out arguments that may have been put in writing.

After few weeks after the "Rashomon of Disclosure"-post, a Twitter discussion about disclosure erupted; and I upset a few participants by more or less saying: "Y'know, I spent the time writing down my thoughts and arguments around the topic, and I am very willing to engage with anybody that is willing to spend the time writing down their thoughts and arguments, but I am not willing to engage in Twitter yelling matches where we ignore all the nuance and just whip up tribal sentiment."

This was not universally well-received, but refusal to engage in social media yelling matches and the dopamine kick that arises from the immediacy of the experience is an important step if we want to understand either the topic or the other participants in the debate.

Summary

This was a very long-winded post. Thank you for having read this far. I hope this post will be helpful - perhaps some of the habits can be useful to others. If not, this blog post can at least explain why I will sometimes withdraw from social media discussions, and insist on long-form write ups as a means of communication. It should also be helpful to reduce bewilderment if I offer betting odds in surprising situations.

Before you ship a "security mitigation" ...

Hey everybody,

During my years doing vulnerability research and my time in Project Zero, I frequently encountered proposals for new security mitigations. Some of these were great, some of these - were not so great.

The reality is that most mitigations or "hardening" features will impose a tax on someone, somewhere, and quite possibly a heavy one. Many security folks do not have a lot of experience running reliable infrastructure, and some of their solutions can break things in production in rather cumbersome ways.

To make things worse, many mitigations are proposed and implemented with very handwavy and non-scientific justifications - "it makes attacks harder", it "raises the bar", etc., making it difficult or impossible for third parties to understand the design and trade-offs considered during the design.

Over the years, I have complained about this repeatedly, not least in this Twitter thread:

https://twitter.com/halvarflake/status/1156815950873804800

This blog post is really just a recapitulation of the Twitter thread:

Below are rules I wrote for a good mitigation a while ago: “Before you ship a mitigation...

1. Have a design doc for a mitigation with clear claims of what it intends to achieve. This should ideally be something like "make it impossible to achieve reliable exploitation of bugs like CVE1, CVE2, CVE3", or similar; claims like "make it harder" are difficult to quantify. If you can't avoid such statements, quantify them: "Make sure that development of an exploit for bugs like CVE4, CVE5 takes more than N months".
   
2. Pick a few historical bugs (ideally from the design doc) and involve someone with solid vuln-dev experience; give him a 4-8 full engineering weeks to try to bypass the mitigation when exploiting historical bugs. See if the mitigation holds, and to what extent. Most of the time, the result will be that the mitigation did not live up to the promise. This is good news: You have avoided imposing a tax on everybody (in complexity, usability, performance) that provides no or insufficient benefit.
3. When writing the code for the mitigation, *especially* when it touches kernel components, have a very stringent code review with review history. The reviewer should question any unwarranted complexity and ask for clarification.
   Follow good coding principles - avoid functions with hidden side effects that are not visible from the name etc. - the stringency of the code review should at least match the stringency of a picky C++ readability reviewer, if not exceed it.”

There are also these three slides to remember:

In short: Make it easy for people to understand the design rationale behind the mitigation. Make sure this design rationale is both accessible, and easily debated / discussed. Be precise and quantifiable in your claims so that people can challenge the mitigation on it's professed merits.

Rashomon of disclosure

In a world of changing technology, there are few constants - but if there is one constant in security, it is the rhythmic flare-up of discussions about disclosure on the social-media-du-jour (mailing lists in the past, now mostly Twitter and Facebook).

Many people in the industry have wrestled with, and contributed to, the discussions, norms, and modes of operation - I would particularly like to highlight contributions by Katie Moussouris and Art Manion, but there are many that would deserve mentioning whose true impact is unknown outside a small circle. In all discussions of disclosure, it is important to keep in mind that many smart people have struggled with the problem. There may not be easy answers.

In this blog post, I would like to highlight a few aspects of the discussion that are important to me personally - aspects which influenced my thinking, and which are underappreciated in my view.

I have been on many (but not most) sides of the table during my career:

• On the side of the independent bug-finder who reports to a vendor and who is subsequently threatened.
• On the side of the independent bug-finder that decided reporting is not worth my time.
• On the side of building and selling a security appliance that handles malicious input and that needs to be built in a way that we do not add net exposure to our clients.
• On the side of building and selling a software that clients install.
• On the side of Google Project Zero, which tries to influence the industry to improve its practices and rectify some of the bad incentives.

The sides of the table that are notably missing here are the role of the middle- or senior-level manager that makes his living shipping software on a tight deadline and who is in competition for features, and the role of the security researcher directly selling bugs to governments. I will return to this in the last section.

I expect almost every reader will find something to vehemently disagree with. This is expected, and to some extent, the point of this blog post.

The simplistic view of reporting vulnerabilities to vendors

I will quickly describe the simplistic view of vulnerability reporting / patching. It is commonly brought up in discussions, especially by folks that have not wrestled with the topic for long. The gist of the argument is:

1. Prior to publishing a vulnerability, the vulnerability is unknown except to the finder and the software vendor.
2. Very few, if any, people are at risk while we are in this state.
3. Publishing about the information prior to the vendor publishing a patch puts many people at risk (because they can now be hacked). This should hence not happen.

Variants of this argument are used to claim that no researcher should publish vulnerability information before patches are available, or that no researcher should publish information until patches are applied, or that no researcher should publish information that is helpful for building exploits.

This argument, at first glance, is simple, plausible, and wrong. In the following, I will explain the various ways in which this view is flawed.

The Zardoz experience

For those that joined Cybersecurity in recent years: Zardoz was a mailing list on which "whitehats" discussed and shared security vulnerabilities with each other so they could be fixed without the "public" knowing about them.

The result of this activity was: Every hacker and active intelligence shop at the time wanted to have access to this mailing list (because it would regularly contain important new attacks). They generally succeeded. Quote from the Wikipedia entry on Zardoz:

On the other hand, the circulation of Zardoz postings among computer hackers was an open secret, mocked openly in a famous Phrack parody of an IRC channel populated by notable security experts.^[3]

History shows, again and again, that small groups of people that share vulnerability information ahead of time always have at least one member compromised; there are always attackers that read the communication.

It is reasonably safe to assume that the same holds for the email addresses to which security vulnerabilities are reported. These are high-value targets, and getting access to them (even if it means physical tampering or HUMINT) is so useful that well-funded persistent adversaries need to  be assumed to have access to them. It is their job, after all.

(Zardoz isn't unique. Other examples are unfortunately less-well documented. Mail spools of internal mailing lists of various CERTs were circulated in hobbyist hacker circles in the early 2000s, and it is safe to assume that any dedicated intelligence agency today can reproduce that level of access.)

The fallacy of uniform risk

Risk is not uniformly distributed throughout society. Some people are more at-risk than others: Dissidents in oppressive countries, holders of large quantities of cryptocurrency, people who think their work is journalism when the US government thinks their work is espionage, political stakeholders and negotiators. Some of them face quite severe consequences from getting hacked, ranging from mild discomfort to death.

The majority of users in the world are much less at risk: The worst-case scenario for them, in terms of getting hacked, is inconvenience and a moderate amount of financial loss.

This means that the naive "counting" of victims in the original argument makes a false assumption:  Everybody has the same "things to lose" by getting hacked. This is not the case: Some people have their life and liberty at risk, but most people don't. For those that do not, it may actually be rational behavior to not update their devices immediately, or to generally not care much about security - why take precautions against an event that you think is either unlikely or largely not damaging to you?

For those at risk, though, it is often rational to be paranoid - to avoid using technology entirely for a while, to keep things patched, and to invest time and resources into keeping their things secure.

Any discussion of the pros and cons of disclosure should take into account that risk profiles vary drastically. Taking this argument to the extreme, the question arises: "Is it OK to put 100m people at risk of inconvenience if I can reduce the risk of death for 5 people?"

I do not have an answer for this sort of calculation, and given the uncertainty of all probabilities and data points in this, I am unsure whether one exists.

Forgetting about patch diffing

One of the lessons that our industry sometimes (and to my surprise) forgets is: Public availability of a patch is, from the attacker perspective, not much different than a detailed analysis of the vulnerability including a vulnerability trigger.

There used to be a cottage industry of folks that analyze patches and write reports on what the fixed bugs are, whether they were fixed correctly, and how to trigger them. They usually operated away from the spotlight, but that does not mean they do not exist - many were our customers.

People in the offensive business can build infrastructure that helps them rapidly analyze patches and get the information they need out of them. Defenders, mostly due to organizational and not technical reasons, can not do this. This means that in the absence of a full discussion of the vulnerability, defenders will be at a significant information disadvantage compared to attackers.

Without understanding the details of the vulnerability, networks and hosts cannot be monitored for its exploitation, and mitigations-other-than-patching cannot be applied.

Professional attackers, on the other hand, will have all the information about a vulnerability not long after they obtain a patch (if they did not have it beforehand already).

The fallacy of "do not publish triggers"

When publishing about a vulnerability, should "triggers", small pieces of data that hit the vulnerability and crash the program, be published?

Yes, building the first trigger is often time-consuming for an attacker. Why would we save them the time?

Well, because without a public trigger for a vulnerability, at least, it is extremely hard for defensive staff to determine whether a particular product in use may contain the bug in question. A prime example of this CVE-2012-6706: Everybody assumed that the vulnerability is only present on Sophos; no public PoC was provided. So nobody realized that the bug lived in upstream Unrar, and it wasn't until 2017 that it got re-discovered and fixed. 5 years of extra time for a bug because no trigger was published.

If you have an Antivirus Gateway running somewhere, or any piece of legacy software, you need at least a trigger to check whether the product includes the vulnerable software. If you are attempting at building any form of custom detection for an attack, you also need the trigger.

The fallacy of "do not publish exploits"

Now, should exploits be published? Clearly the answer should be no?

In my experience, even large organizations with mature security teams and programs often struggle to understand the changing nature of attacks. Many people that are now in management positions cut their teeth on (from today's perspective) relatively simple bugs, and have not fully understood or appreciated how exploitation has changed.

In general, defenders are almost always at an information disadvantage: Attackers will not tell them what they do, and gleefully applaud and encourage when the defender gets a wrong idea in his head about what to do. Read the declassified cryptolog_126.pdf Eurocrypt trip report to get a good impression of how this works.

Three of the last four sessions were of no value whatever, and indeed there was almost nothing at Eurocrypt to interest us (this is good news!). The scholarship was actually extremely good; it's just that the directions which external cryptologic researchers have taken are remarkably far from our own lines of interest. 

Defense has many resources, but many of them are misapplied: Mitigations performed that do not hold up to an attacker slightly changing strategies, products bought that do not change an attacker calculus or the exploit economics, etc.

A nontrivial part of this misapplication is information scarcity about real exploits. My personal view is that Project Zero's exploit write-ups, and the many great write-ups by the Pwn2Own competitors and other security research teams (Pangu and other Chinese teams come to mind) about the actual internal mechanisms of their exploits is invaluable to transmit understanding of actual attacks to defenders, and are necessary to help the industry stay on course.

Real exploits can be studied, understood, and potentially used by skilled defenders for both mitigation and detection, and to test other defensive measures.

The reality of software shipping and prioritization

Companies that sell software make their money by shipping new features. Managers in these organizations get promoted for shipping said features and reaching more customers. If they succeed in doing so, their career prospects are bright, and by the time the security flaws in the newly-shipped features become evident, they are four steps in the career ladder and two companies away from the risk they created.

The true cost of attack surface is not properly accounted for in modern software development (even if you have an SDLC); largely because this cost is shouldered by the customers that run the software - and even then, only by a select few that have unusual risk profiles.

A sober look at current incentive structures in software development shows that there is next to zero incentive for a team that ships a product to invest in security on a 4-5 year horizon. Everybody perceives themselves to be in breakneck competition, and velocity is prioritized. This includes bug reports: The entire reason for the 90-day deadline that Project Zero enforced was the fact that without a hard deadline, software vendors would routinely not prioritize fixing an obvious defect, because ... why would you distract yourself with doing it if you could be shipping features instead?

The only disincentive to adding new attack surface these days is getting heckled on a blog post or in a Blackhat talk. Has any manager in the software industry ever had their career damaged by shipping particularly broken software and incurring risks for their users? I know of precisely zero examples. If you know of one, please reach out, I would be extremely interested to learn more.

The tech industry as risk-taker on behalf of others

(I will use Microsoft as an example in the following paragraphs, but you can replace it with Apple or Google/Android with only minor changes. The tech giants are quite similar in this.)

Microsoft has made 248bn$+ in profits since 2005. In no year did they make less than 1bn$ in profits per month. Profits in the decade leading up to 2005 were lower, and I could not find numbers, but even in 2000 MS was raking in more than a billion in profits a quarter. And part of these profits were made by incurring risks on behalf of their customers - by making decisions to not properly sandbox the SMB components, by under-staffing security, by not deprecating and migrating customers away from insecure protocols.

The software product industry (including mobile phone makers) has reaped excess profits for decades by selling risky products and offloading the risk onto their clients and society. My analogy is that they constructed financial products that yield a certain amount of excess return but blow up disastrously under certain geopolitical events, then sold some of the excess return and *all* of the risk to a third party that is not informed of the risk.

Any industry that can make profits while offloading the risks will incur excess risks, and regulation is required to make sure that those that make the profits also carry the risks. Due to historical accidents (the fact that software falls under copyright) and unwillingness to regulate the golden goose, we have allowed 30 years of societal-risk-buildup, largely driven by excess profits in the software and tech industry.

Now that MS (and the rest of the tech industry) has sold the rest of society a bunch of toxic paper that blows up in case of some geopolitical tail events (like the resurgence of great-power competition), they really do not wish to take the blame for it - after all, there may be regulation in the future, and they may have to actually shoulder some of the risks they are incurring.

What is the right solution to such a conundrum? Lobbying, and a concerted PR effort to deflect the blame. Security researchers, 0-day vendors, and people that happen to sell tools that could be useful to 0-day vendors are much more convenient targets than admitting: All this risk that is surfaced by security research and 0-day vendors is originally created for excess profit by the tech industry.

FWIW, it is rational for them to do so, but I disagree that we should let them do it :-). 

A right to know

My personal view on disclosure is influenced by the view that consumers have a right to get all available information about the known risks of the products they use. If an internal Tobacco industry study showed that smoking may cause cancer, that should have been public from day 1, including all data.

Likewise, consumers of software products should have access to all known information about the security of their product, all the time. My personal view is that the 90 days deadlines that are accepted these days are an attempt at balancing competing interests (availability of patches vs. telling users about the insecurity of their device).

Delaying much further or withholding data from the customer is - in my personal opinion - a form of deceit; my personal opinion is that the tech industry should be much more aggressive in warning users that under current engineering practices, their personal data is never fully safe in any consumer-level device. Individual bug chains may cost a million dollars now, but that million dollars is amortized over a large number of targets, so the cost-per-individual compromise is reasonably low.

I admit that my view (giving users all the information so that they can (at least in theory) make good decisions using all available information) is a philosophical one: I believe that withholding available information that may alter someone's decision is a form of deceit, and that consent (even in business relationships) requires transparency with each other. Other people may have different philosophies.

Rashomon, or how opinions are driven by career incentives

The movie Rashomon that gave this blog post the title is a beautiful black-and-white movie from 1950, directed by the famous Akira Kurosawa. From the Wikipedia page:

The film is known for a plot device that involves various characters providing subjective, alternative, self-serving, and contradictory versions of the same incident.

If you haven't seen it, I greatly recommend watching it.

The reason why I gave this blog post the title "Rashomon of disclosure" is to emphasize the complexity of the situation. There are many facets, and my views are strongly influenced by the sides of the table I have been on - and those I haven't been on.

Everybody participating in the discussion has some underlying interests and/or philosophical views that influence their argument.

Software vendors do not want to face up to generating excess profits by offloading risk to society. 0-day vendors do not want to face up to the fact that a fraction of their clients kills people (sometimes entirely innocent ones), or at least break laws in some jurisdiction. Security researchers want to have the right to publish their research, even if they fail to significantly impact the broken economics of security.

Everybody wants to be the hero of their own story, and in their own account of the state of the world, they are.

None of the questions surrounding vulnerability disclosure, vulnerability discovery, and the trade-offs involved in it are easy. People that claim there is an easy and obvious path to go about security vulnerability disclosure have either not thought about it very hard, or have sufficiently strong incentives to self-delude that there is one true way.

After 20+ years of seeing this debate go to and fro, my request to everybody is: When you explain to the world why you are the hero of your story, take a moment to reflect on alternative narratives, and make an effort to recognize that the story is probably not that simple.