Three important steps in my maturation process

My father passed recently, and he was twice my age. I am approximately the same age that he was when I was born, and I am now "the old generation" - there's no one left in the generation above me.

At the same time, I recently joined a company that skews younger-than-me. When I joined Google in 2011, I had just turned 30, and was in the mainstream demographics of Google in 2011. There were a bunch of more senior folks, with the very senior ones being in their 50s and having completed stints at Bell Labs. I admired a lot of these "greybeards" (even though this is a sexist term - what's the right female equivalent? There were a few very senior female engineers that I would love to include).

So perhaps it is natural that I am reflecting on "what were the important realizations that I made since my early 20s that had a profound impact on the way I think about the world"? In some sense: What are the insights I had that made me "more mature", for some positive definition of "mature"?

This post tries to list them.

1. The importance of understanding your own incentive structure, and not believing everything you think.

I recently wrote a Twitter thread about the topic. Oppenheimer was very publicly guilt-ridden about the creation of the nuclear bomb, and von Neumann at some point quipped "some people profess guilt to claim credit for sin". In my young years, particularly in situations when I had 0day that nobody else had, I agonized about the responsibility that comes with having 0day. Should I fix them? Should I use them for good? Will the world be harmed this way? Or that way?

In the end, it turns out that - while individuals matter - many ideas have a "time at which they are ripe", and the actions of the individual matter less than the individual thinks in that moment. There is also almost no way to predict the ways in which what you do impacts the broader world.

If you were asked: "Would it be good if this 0day was used to apprehend a terrorist?" you would probably say "this is good". If you were asked "would it be good if this 0day is used to arrest someone and then torture and waterboard him 183 times?", you would probably say "this is bad". So if your 0day was used to capture KSM, it is probably good? Or bad? Things get very complicated very quickly.

Is closing 0days good for society, because it makes everything safer? Or is it enabling oppression, because buggy systems are easier to bypass?

There are no good answers, and your own incentive structure will greatly influence how you choose your beliefs. In the end, people want to be the heroes of their own story, and at the same time they have basal needs for recognition, for material goods, etc. - so they will try to construct a narrative that allows them to satisfy their basal needs while also remaining the hero of their saga.

Anxiety about the impact of your work is self-flattering, and you have to recognize it as such, and keep it in check - it's sugar for your ego, but history will largely route around you, because while individual decisions matter in specific situations, the overall flow of history is less sensitive to the individual than the individual thinks. The broader lesson, though, is: Do not believe everything you think. Examine your own incentive structures carefully. Ask yourself what alternative narratives for your behavior and beliefs could be, especially if they contradict the narrative of the heroic saga you're constructing for yourself. Carefully weighing the question "how might I be the villain in this story?" is an important and valuable skill.

Similarly, meta-cognition - just observing your own thoughts in a detached manner, and then being able to interpret, analyze, and contextualize them with regards to your own incentive structures, is a great skill to cultivate.

2. Monocausal determinism is an illusion, and largely does not exist outside of computer debugging.

The monocausal determinism that young computer enthusiasts get used to is an illusion that generations of electrical and process engineers spent their lives perfecting and maintaining. It is because of these engineers that computer scientists could largely get away without probabilities or any empirical grounding in the past. There is an argument that you have so many natural scientists that crossed over into AI because CS education was for a long time too focused on reasoning within the deterministic monocausal illusion.

The reality is: Computing machines are physical devices, which includes wear & tear, differences in quality between items, and "probabilistically deterministic behavior", e.g. it'll appear deterministic most of the time if not shaken too much. If pushed a bit - be it temperature, voltage, electromagnetic fields, or even rapid memory accesses to adjacent DRAM rows - determinism has a tendency to go out of the window, the illusion collapses, and we're dealing with a very different beast.

FWIW - this also makes me wonder about model alignment, because even a perfectly aligned model will be subject to random bit flips in inference, and it's hard for me to imagine that you can maintain any reasonable guarantees in the presence of bit flips to inopportune values at inopportune times.

The real world is one where very few things that happen have a single reason, and very few truly deterministic transmission mechanisms. Everything is probabilistic, and everything is multicausal.

Measurement noise is real, experiment design is difficult.

Interestingly, if you think about this carefully, you also realize that the scientific method is a classifier that is intentionally biased against accepting something as true - so that we only accept things as true that are beyond any reasonable doubt true.

A somewhat fascinating corolary of this is that there exists a large class of true things that will never be scientifically shown as true.

3. The dichotomy between reason and emotion is a cultural construct, and neither grounded in neuroscience nor in logic.

With some digging, it turns out that the western belief that reason and emotion are two ends of a spectrum is a purely cultural construct, as is the belief that "higher-order" reason needs to reign in "basal" emotions, or that "emotions" intrude on "rationality".

In most non-western cultures, achieving integration between rational deliberation and impulses and emotions is more common, and it turns out that this is much closer to the biological reality.

From a neuroscience perspective, it is clear that emotional valuation is part of a larger decision-making machinery that tends to not function properly if the emotional valuation component is damaged or removed. There is also a large component where things that your brain struggles to articulate verbally are transmitted via emotions, as well as actual feedback from your sensory organs in your body. Fun trivia: Your gut's enteric nervous system contains as many neurons as the entire cerebral cortex of a dog. Your body also forward-deploys neurons in your muscles and extremities, as a form of latency optimization. Your body is feeding you extra information, and most of this shows up in the shape of emotions.

Which brings us to the logical argument why attempting to "remove" emotions from decision-making is a bad idea: Clearly, having the ability of leveraging more information for decision-making will improve the quality of decisions. Attempting to eliminate a particular source of information almost certainly makes the quality of your decisions worse.

This is not to say one should act on impulse alone, but it is certain that integrating the full spectrum of information - which includes emotions - in your decisions is a wise idea.

I am sure that if I think more carefully, I will come up with more insights, but these three are important enough that they show up in my life with astonishing regularity.

Hope this is helpful to someone.

RL economics, morally charged terms, and "distillation"

After a number of Twitter discussions, and repeating myself a lot in these discussions, it is time to write a short note on the economics of advancing LLM capabilities through RL, about principles of propaganda and coining new words, and about my stubborn refusal to use the term "distillation" except in a specific narrow sense.

How do models advance when human-curated data has run out?

It's been a while since we ran out of human data to train LLMs on. We are training on copies of the internet, large piles of (originally pirated, then purchased-and-scanned-and-wholesale ingested) books, and whatever other data sources we can obtain. This leads to a certain performance plateau, as we haven't quite figured out how to make the models more data-efficient in training.

The advancements we have seen in coding and mathematics in the last year are mostly due to reinforcement learning. At the highest level, you pose a problem to an LLM that the LLM has a small but nontrivial chance of solving. You then run N copies of the LLM to generate solutions, and you get a small number of solutions and many failures. You can then use the successful solutions as new data to improve your model - moving the weights in a way that helps the model succeed with greater probability.

This is very elegant in a way, because you are kinda pulling yourself up by your own bootstraps. The cost is computational - if you have a 1% chance of finding a solution given your current LLM and current training data, you need to do 100s or 1000s of rollouts to get a reasonable variety of useful solutions.

Once you have a model that can generate a good solution for this problem with high probability, and you make that model available to others, you also provide a much cheaper way of producing the better training data: Third parties can now just ask your model to generate good solutions for them.

So for the second-mover that gets to use your model, improving their model from your model outputs is cheaper, as they can skip the more-or-less-random-search into a high-dimensional solution space and be guided better.

This is a fundamental part of the "closed LLM as a service" business, and it is painful for the leader of the pack because they need to spend money to advance, and others can catch up more cheaply.

Terms of service, copyright law, crimes vs. contract disputes

Copyright law imposes concrete ownership rights on copyrighted material. Pirating material and commercially exploiting it is often a crime.

The frontier labs have all argued that training on public data does not require them to obtain licenses from the copyright holders (a self-serving and somewhat dubious claim). The Llama release further muddied the waters by adding a license to the redistribution of model weights - by law, the output of an algorithm itself (such as model weights) are not a copyrightable object, and Meta just pretended they were. Other model labs followed suit, in the hope of establishing a practical precedent that can then be used to shape legislation in the future.

But a priori, model weights are not copyrightable.

There is an argument, though, that prompts, and the resulting output from the model are copyrightable to the person submitting the prompts. Certainly not to the model provider: Running an algorithm on somebody else's copyrightable work without human input does not make you the owner of the work. There is no human creativity input, which is the minimum threshold for establishing copyright in our current legal system.

Model providers have no rights to the output of their models if they provide access to these models to third parties.

What rights do model providers have? They have the right to set terms-of-service for their service - e.g. if you don't use the tool in a way we like, we revoke access to the tool.

Terms-of-service are very different from copyright law - they are essentially private law contracts about the exchange of services between entities. So if a model provider says "you may not use this service to generate training data for your competing LLM", they can say so, and they have the right to terminate your account if they catch you doing so.

That said - let's say I was to run a benchmarking service that tests the progress of LLMs against my favorite programming problems, and all I do is (a) run rollouts against these services (b) score the results (c) archive the results (d) sell access to the results to third parties so they can evaluate progress of models and the quality of their reasoning and (e) publish the positive results after a few months for free.

This is not a violation of the terms of service -- I am just measuring the capabilities of the models and have them solve problems for me. Publishing the data isn't a violation of the terms of service either.

Yet - by me publishing the positive results into the greater internet makes them part of the training corpus, so the improvement in capability that the model provider achieves will flow into other models. There is no way around this in our current legal system.

Reframing an inconvenient issue with your business model in moral terms

Imagine you've raised billions of dollars and you realize that your business model has a rather inconvenient flaw - you have a good business, but for it to become a fantastic business, you'd need to fix this flaw. And the flaw, as you perceive it, is the current legal system for intellectual property with it's old and well-tested precedents and mechanisms.

It will be easy to convince yourself that the flaw in your business model that gives your competitors a way to catch up with lesser investment is a moral outrage - it is so unjust! - and then complain about the fact that others have the right to do what they are doing.

Once you've convinced yourself of the immorality of what your competition is doing (how dare they compress your margins?), you will need to somehow re-frame what they are doing in moral terms. So "training on solved problems to improve" doesn't quite have the right ring to it. We need something malicious, like "distillation attacks".

"Distillation" is great, because it evokes bootlegging and 1920s prohibition-era intrigue. And "attack" is great because only bad people attack. So you leverage the fact that people called a technique to teach a smaller model from a larger model provided you have access to the internals of the larger model "distillation", you tack on the word "attack" to make it sound more nefarious, and you start screaming from the rooftops that evil distillation attackers are killing your morally superior business (that started by actual copyright violations, only justified ex-post by your success).

This is what happened here, and I urge every reader to not go along with it. Distillation means having access to a large model, including all the last-layer token probabilities, and training a smaller model by taking those internal last-layer probabilities into account.

Just training on model output isn't it. And you cannot have a world where people use LLMs to write code or text, and are allowed to publish that on the internet, and simultaneously prevent up-leveling other models as they train on that data. You have no legal or moral legs to stand on if you want to prevent that.

If the chinese models are distilled, so is the Cursor fine-tune of Kimi, or any model that is trained on the output of other models - and most of human output is now model-assisted.

You are free to argue that this is inconvenient for your business model, and a legal framework which allows you to prevent that would be useful in attracting more investment to advance your model, but that's about it.

This is why I don't call training on other models output "distillation"

Let's call it "training on model output", or whatever else that is not morally charged. And let's be honest that the existence of LLMs in their current form is the result of highly dubious approaches to copyright that are ex-post legitimized by the actual value these models bring to society. Let's please avoid allowing parties with particular financial interest build a moral framing around their interests, though.

Slightly safer vibecoding by adopting old hacker habits

I have seen a lot of public discussion around supply-chain attacks on the Python ecosystem, prompt injection risks when using coding agents, and general worries about the security implications of "vibe coding" for the development machine.

In some of these discussions I find myself puzzled as to what problem is being solved - and it took me a while to realize that my failure to understand lies in the development setup that I tend to use.

In this blog post I'll quickly explain my development setup.

The setup is pretty simple:

1. The actual development happens on a rented server (or a VM on that server).
   
2. In order to do development, I SSH into that server with key-forwarding for my github keys enabled.
   
3. I perform my development on the server by attaching to a screen or tmux session.
   
4. I used to just use vim with various extensions, but with the advent of coding agents I also use claude code etc. nowadays.
5. I avoid keeping secrets inside the development VM or on the development server.
   
6. I let the agent churn away on problems for extended periods of time while I am detached from the tmux/screen.

A setup like this reduces a large number of supply-chain attacks to - at worst - compromise the development VM.

There is still a significant risk of the github key forwarding being abused to compromise the upstream main repository.

The way around this is a bit cumbersome, but not much different from what many open-source projects already do: You keep a main repository, and you *fork* a development repository from it. Then you do all your development on the dev repository, and when you're done in your development branch, you issue a cross-repository pull request.

Obviously, a human needs to go through that PR with a fine comb - but this is something you want to do for insider risk etc. anyhow, so your risk profile changes only marginally.

In a setup like this, the main secret that you'll lose in a supply chain attack are your Claude credentials. And you don't need to worry about prompt injection into your coding agent too much, and can just focus on writing code.

Interestingly, the development model of "SSH into a machine and attach to a screen session" was popularized by the hacker subculture (as in "computer break-in" subculture) since historically it was never a good idea to have data on machines you physically own. SSH'ing into a random machine in a different country that law enforcement couldn't easily get access to was a reasonable way of keeping your hands clean. I mainly switched to that development model because I almost always need long-running compute and was travelling a lot, and with agent-first development the model is seeing a bit of a resurgence.