❌

Normal view

There are new articles available, click to refresh the page.
Before yesterdayMain stream

What happens to mathematics when AI becomes an oracle?

10 November 2026 at 20:30

Our modern world runs on math, from social media algorithms, package delivery, and weather prediction to medical imaging and machine learning. The rise of computers helped fuel its dominance, enabling us to work with increasingly large numbers that would otherwise be beyond the capacity of our finite human brains. But with the rise of large language models, AI chatbots, and, soon, quantum computing, so-called "big math" is beginning to outstrip its creators entirely. What that means, and how we might adapt, is the focus of a new book, Big Math: The Hidden Codes That Run Our World, by applied mathematicians Steven Strogatz and Alex Townsend.

Strogatz is perhaps best known in his field for his co-authorship of a seminal 1998 paper with Duncan Watts on the collective dynamics of small-world networks. He has also studied chaos theory as applied to physics, the geometry of supercoiled DNA, and the biological collective oscillators, such as swarms of synchronously flashing fireflies and crickets that chirp in unison. In 2005, Strogatz extended his study of synchronicity to the crowds that caused the wobbling of London's Millennium Bridge.

An award-winning communicator, Strogatz wrote three series of columns on mathematics for The New York Times in 2010, 2012, and 2025 and hosted a podcast for Quanta magazine from 2020–2021. He has written four popular books: Sync (2003), The Calculus of Friendship (2009), The Joy of X (2012), and the New York Times bestseller Infinite Powers (2019).

Read full article

Comments

© Kenishirotie via iStock / Getty Images

Scattered thoughts on how doing math feels these days

By: dynomight
26 October 2026 at 00:00

This is a weird moment. I have a hunch that we’ll look back later and tell ourselves stories in which it seemed obvious where things were going. So I wanted to lay down a little time capsule and register just how confusing everything felt at the time. This is a personal view from the trenches.

1.

If you’re a normal person, you may feel that AI has been good at math for a long time.1 But for “real” math, this is roughly what it felt like:

When Feeling
2024 LLMs are toys
Summer 2025 LLMs are interesting toys
Early Autumn 2025 LLMs can do useful stuff sometimes
Mid Autumn 2025 …a lot of stuff
Late Autumn 2025 …a lot
Early 2026 holy god

Even having been astonished by AI progress in the past, it was still astonishing to see AI move in a matter of months from a mildly-useful toy to something that many people now consider almost impossible to do without.

2.

Personally, I’d say that current AI makes me around 50% more productive at “doing math”. That’s despite the fact that AI can now often do in a few minutes things that would have taken me hours to do myself.

The main reason is that my concept of “doing math” involves various steps, like coming up with ideas for theorems, trying to prove them, writing things down, reorganizing everything to make it more clear, and so on. AI is good at some parts of this, but still pretty bad at others, so the overall speedup isn’t as high as you might expect.

(Even if planes could go 10× faster, your door-to-door travel time from your apartment in London to your friend’s place in Rome wouldn’t decrease that much, because you still have to get to the airport and wait to board.)

3.

But then, I’m still (sort of; more on this later) trying to follow my old “doing math” workflow, just automating certain parts of it. That can’t be right, can it?

If we found a way to make batteries at 1% of the current cost, we wouldn’t just say, “cool, batteries are cheap now”. We’d find all sorts of new ways to use batteries that didn’t make sense before, and change the ways we generate power to account for all these new batteries. Probably our whole energy system would end up transformed. Analogously, AI being good at proofs should lead to a very different equilibrium. But what is that?2

4.

It seems healthy, at the moment, to maintain a fluid view of what “doing math” means. But at bare minimum it surely involves understanding the math that “I” am “doing”. So long as that’s part of the workflow, it would seem to impose a limit on how quickly “I” could “do” math, no matter how good AI is.

One obvious conclusion would be that my tiny organic brain should be removed from the workflow entirely. Maybe! But it’s interesting to speculate about how much faster things could go while still keeping a human brain in the loop. I think the answer is: A lot.

5.

Contrary to what you may have heard, humans are not obsolete for math. Not yet. The most obvious reason is that AI is still bad at writing.

People often complain that AI writing overuses various patterns or lacks a unique voice or whatever. I don’t disagree, but I always felt like there was a deeper issue that I couldn’t quite put my finger on. With math, this becomes very clear: The problem is it doesn’t make any goddamned sense.

Really, I think it’s deeper than being bad “at writing”. It seems like AI is bad at thinking in certain ways that manifest as bad writing.

6.

When novices do math, they often create “spaghetti-code math”: Imagine a long document with lots of equations, but if you point at any one equation, it’s hard to say what assumptions underpin it, or how it fits into the bigger picture. Any random idea that was mentioned 17 pages ago might be used again at any time with no warning. With experience, people learn to break complex arguments down into modular pieces with a good “API”, in the sense that the assumptions and conclusions for each part are relatively simple.

Somehow, AI has developed these amazing technical skills, while still being bad at creating abstractions So the arguments it gives are long and tangled and bewildering. This is what makes AI math seem so alien.

7.

Now I know what you’re thinking. Maybe AI isn’t bad at creating abstractions. It’s just bad at creating abstractions that work for you, human with tiny organic brain.

From a distance, that seems plausible. And it might be true to some degree. I have high-bandwidth access to a human brain. It’s not crazy to imagine that this gives me an advantage in predicting how a human brain would react to a given presentation of ideas. But still, I claim that current AI is objectively bad at creating abstractions, regardless of if the consumer of those abstractions is human or artificial.

Here’s why: If I ask AI to prove something hard, it will produce a proof that starts simple and gradually descends into a rampage of complexity. If I tell it to break that proof down into self-contained pieces, it will produce lemmas that each take a full page just to state. If I ask, “Why? Why have you done this to me?”, the AI will say, “This is as simple as it gets. It’s not that bad. What’s your problem?”

But if I argue with the AI for a while, I can start to understand the basic contours of the argument. And I’ll ask, “Why don’t you first just try to prove blah using only assumptions blahblah, without worrying about blahblahblah? Wouldn’t that be easier?” Often, this doesn’t work, either because my idea was bad, or because the AI isn’t smart enough to do it. (Which one? Who knows!) But after a few attempts, I can usually find something that works, and the AI can use it to re-write the proof in a somewhat simpler way. If I repeat this many times, I can gradually chip away at the complexity.

But maybe that’s still just making the proof easier for humans to understand? No! For one, this usually makes the proof shorter, and sometimes dramatically shorter. More importantly, during this process, someone (maybe me, maybe the AI) will often realize that there’s a different proof strategy that gives a stronger result. Instead of proving x < 9.7, now we can prove x < 7.1. Thus, artisanal human abstractions are “objectively” useful.

8.

Some people might claim that it’s already possible to automate some of what I’m doing with some multi-agent Rube Goldberg thing. I’m not sure, because my experience has been that investing lots of time into clever AGENTS.MD or custom skills or harnesses isn’t a great use of time. That stuff is somewhat helpful, but things are changing so fast that the knowledge has a short half-life. Also, the majority of information about these tricks seems to come from people who are excited about the technology itself.

On the other hand, the automator told me to try typing, “ask independent agents for an adversarial review”, and I’ve found those seven words to be quite helpful. I wish I could find a source with boring best practices for people who are happy to stay away from the bleeding edge.

9.

Another reason AI doesn’t improve my productivity more is that it is often the green light, the orgastic future that recedes before me. I’ll ask AI to do something, and it will almost seem to do it. So I’ll ask for changes, and it seems to help, but it’s still not quite right. Hours later, I’ve accomplished nothing and learned nothing. Eventually, I start over, either doing it myself or with AI on a very tight leash, and I’m done in 30 minutes. Each time this happens I feel foolish and tell myself not to do it again. But it’s very hard to judge in real-time.

10.

Of course, I’m not claiming that AI is intrinsically bad at writing or thinking or abstraction-making. It’s gotten better at all of these and I expect it will continue to do so. All I’m saying is that as of Autumn 2026, humans still have a lot to contribute, and it’s not just better exposition.

11.

Did AI actually get much better at math in Autumn 2025? Or perhaps, in the grand scheme of things, human math ability falls into a narrow range?

12.

Some people are responding to the new reality by gleefully using AI to generate lots of semi-OK-ish papers and puking them up on arXiv. I understand the incentives that drive this. In some sense, it seems like the logical response to the new reality. (After inventing cars, we started taking a lot more low-value trips.) But I don’t think this is socially useful, and I tend to think everyone will soon agree that this is embarrassing. But… I’m not sure. Maybe it is socially useful? And I find myself feeling vaguely jealous of the people who are wired this way, who instead of feeling blinkered, see opportunity and plunder.

13.

Want an optimistic pro-AI, pro-progress take? OK. If I take my old “doing math” workflow, AI makes it strictly better. A lot better. It’s like having a hyper-conscientious intern that is smarter than me in many ways, works at incredible speed, and knows approximately everything.

Yes, AI sucks at writing and making abstractions. But if I’m disciplined and try to mostly do the math “myself” while asking the AI for help, I can explore ideas much faster than before, make far fewer mistakes, and produce something just as clear as before, while developing just as much understanding. I can ask, “What happens if you do a Kofuni expansion?”, and get an answer instantly. I can ask for illustrations or interactive visualizations and see them instantly. Arguably, we’re incredibly lucky to be alive at this moment, when these new tools have opened up this new territory to explore.

14.

And isn’t that what I wanted? Back in 2024, I bemoaned that what AI most seemed to struggle with was getting the facts straight:

Perhaps the old science fiction robots were a projection of our fantasies. I want my style and voice to be precious and uncopyable. I want an AI assistant that will tirelessly hunt down facts while I provide creativity and panache and magic. But that’s not how it is. So we should remember: In a scenario where AI is useful but humans retain some comparative advantage, we might not like where that comparative advantage lies.

At least with math, and at least for the moment, I get to be Kirk, while the AI is Spock. (Even with other types of writing, my impression is that since 2024, AI has improved more along Spock dimensions than along Kirk dimensions.) So that’s nice.

15.

In the last few months, it frequently happens that a collaborator will use AI to generate something, and then send it to me and ask me to check if it’s useful. I can see why this seems reasonable. (Really, I do.) But when it happens a lot, one can’t help but wonder what the point of the human collaborator is.

16.

On the other hand—with creative writing, my advice is not to use AI to write, at all. Ask questions if you want, but write every damned word yourself.3 With math, I don’t feel like such a hard line makes sense. It seems that creating something good still requires a lot of human involvement, to the point of agonizing over notation, repeatedly re-organizing things, and heavily editing individual sentences. Still, I find that if people make that effort, they can create something just as good as they could by hand, while still being perhaps 50% AI written.4

So I don’t mind if people send me “centaur math”, provided they take responsibility for the contents. The problem is that when working with AI, it takes constant paranoid vigilance to avoid letting things slip beyond your understanding. I often find myself wanting to awkwardly ask people, “OK, you obviously used AI to make this… which is OK… but how much of this do you vouch for? If we have a meeting to discuss it, will you crumble the moment I start asking questions?”

17.

When I’m in abstraction-creation mode, I feel like I’m doing something intelligent. When I yell at AI about bad notation and writing, I feel like I’m providing taste. But I often find myself giving feedback that seems incredibly basic. Things like: This is a mess, clean it up. Or: Do you need that assumption? Or: If you can’t solve this problem, why don’t you create a simpler version and try to solve it first?

On the one hand, I guess a lot of what good human managers do amounts to giving very basic feedback. On the other hand, it’s hard to imagine that I have much durable advantage in doing this kind of thing. So I wonder how much durable advantage I have in abstraction-making or taste, too.

18.

A common reaction to AI getting better at proofs is that humans need to focus more on exposition. This has an obvious Bourdieu-esque criticism: Human painters used to strongly value photo-realism, but after cameras were invented they convinced everyone that, oops, photo-realism isn’t important, what matters is expression, which coincidentally cameras can’t do. Isn’t a shift towards exposition in math just a way for humans to move the goalposts and justify their own continued relevance?

The only credible way I see to avoid that criticism seems to be to bite the bullet and say that, in fact, math should have already have focused more on exposition, even before AI. Personally, I strongly believe that this is true.

19.

I think the biggest reason I’m not more productive is: What’s the point? In the past, if you came up with an elegant proof, you’d tell yourself, “I made a thing! Other people will read and use my thing! I am advancing human progress.” You can still tell yourself that now. But you have to wonder, will anyone actually read your proof? Or will they just ask another AI to explain it to them? Will your proof help their AI explain it better? For how long? If you just waited a year, would some other AI produce that same proof independently?

  1. Alternatively, if you’re a normal person, you might think that LLMs still struggle with basic arithmetic? ↩

  2. AI progress is so fast, compared to society’s ability to adjust, that we may never actually see what “the current equilibrium” looks like. ↩

  3. This essay, if you’re wondering, was created by writing every damned word myself and not asking AI anything. ↩

  4. You could have a whole debate about what that percentage means. It’s not the number of words written by AI that really matters, in my opinion, so much as who is responsible for the overall structure of the ideas. ↩

Pseudpocalypse

By: dynomight
14 July 2026 at 00:00

Here’s a conjecture: If you put any significant amount of text on the internet under different names, those identities can be linked using only the text itself. This is possible (I conject) because of the statistical “fingerprint” you leave in everything you write.

Imagine a website where you can paste in some brand-new text someone just wrote. In return, the website provides links to all the text that writer has ever published under any name. It’s not perfect, but it’s pretty good. As far as I know, no such website exists—at least not on the public internet. But I suspect it’s possible and will soon become easy. This will pose some difficulty for pseudonymous blogging.

Note: I wrote most of this essay in mid-2025, after which I idiotically sat on it for a year tinkering with theorem statements that none of you will read.1 In the meantime, LLMs have gotten much better at guessing authors from text. (Given the first 1000 words of a draft of this post, Claude 4.8 knows it’s me.) Still, I think we’re just getting started. I expect to see increasingly obscure writers identified from increasingly small bits of text. I expect that this work even when people are writing in a different register or about unrelated subjects. And I expect that everything I’ve ever written under any pseudonym will soon be linked to my genuine-nym.2

A stronger conjecture is that we’re heading towards a sort of generalized pseudpocalypse. Perhaps, in the future, if you interact with the world through essentially any high-bandwidth channel, then you identify yourself. Say you wear a mask in public and only speak by sub-vocalizing into a voice changer. That’s fine, you’ll still be identified using your body shape, gait, or chemical signature. Or say you don’t like your car being tracked everywhere, so you stop carrying a phone and you somehow convince lawmakers to ban license plates. No problem, your car will still be tracked using tiny scratches or unique pinging sounds from the engine. Or say you don’t like being tracked on the internet, so you lock down your browser profile, buy stuff only with Monero, and connect through a chain of three VPNs. That’s OK. You’ll still be identified through how you wiggle your finger as you scroll down the page. We’re all just too unique, and the information theoretic limit is coming for us.

Starting bits

Let’s start from first principles. Imagine that at birth, everyone is assigned a random binary string. Whenever you post anything on the internet, you’re required to sign it with that string. If the strings are very short, like 0110, then lots of other people will have the same one as you. But if the strings are very long, then yours would almost certainly be unique and it would be trivial to link all your pseudonyms.

Where’s the transition point? If you only know that the author is currently alive and living somewhere in the Anglosphere, it’s around 29 bits. That’s because if there are K digits, then there are 2ᴷ possible binary strings, and if K = 28.86, then 2ᴷ ≈ 490,000,000 is the number of currently-alive Anglosphere-dwellers. If the strings have fewer than 29 bits, then someone else will probably share your string. If they have more than 29 bits, then your string is probably unique.

We don’t (yet?) have to sign the things we write with immutable government-issued strings. But the way you write still provides lots of clues about you by way of your tone, personality, word choice, and so on.

Theoretically speaking, I think it has to be possible to link the identities of anyone who writes enough. Imagine again that everyone is assigned a random binary string at birth, but instead of you needing to sign the stuff you write with your string, each time you write a word, there’s some chance that a random bit from your string is revealed and added as a signature to your message. For example, maybe a signature of bit[129]=1 is added, indicating that your string at position 129 has value 1.

Think of your string as representing all your writing style quirks, and a bit being revealed as representing when you write something that reveals a preference. For example, maybe bit 18 indicates if you prefer to write your em-dashes with hideous spaces — like this — or without spaces—like this. If you use an em-dash, that bit is revealed.

So imagine you’ve written a lot under Pseudonym A, enough that the full bit-string has been revealed. Maybe it’s this:

Pseudonym A: 
110000001111001101110000100001
010100100101011110111001101000
100111110010101001101010111010

Now say you start writing under Pseudonym B. Initially, none of the bits will be known:

Pseudonym B: 
??????????????????????????????
??????????????????????????????
??????????????????????????????

But slowly, you’ll start to leak a few bits:

Pseudonym B:
?????00???1????1????????1??0??
?????01??1?????????11????01???
???1??????1???10??????????????

And eventually you’ll leak a lot of bits:

Pseudonym B:
???0?00?1?11??110??1????10?0??
?10?001?0101?11???11100?101???
???11?1??010??10?1?01??011????

Now think about this from the perspective of an “attacker” who wants to know if A and B are the same person. Let’s assume they’ve only seen the above bits, and have no information about anyone else. Then here’s what the attacker knows:

  1. A and B have revealed K overlapping bits, which all match.
  2. Different people have a 50% chance of matching on any given revealed bit.
  3. Non-different people have a 100% chance of matching on any given revealed bit.
  4. There are 490,000,000 people.

Intuitively, if K was 5, then the fact that all bits match wouldn’t prove much, since with 490 million people, lots of people would match on those bits by chance. But if K was 70, it’s extremely unlikely that two different people would share all of them, even with such a gigantic pool to start with. It turns out that if there are N other people with random bits, and you pick K of your bits, the probability that someone exists who matches all of them is 1 - (1-2⁻ᴷ)ᴺ. When N is 490 million, that looks like this:

Look at that, 29 appears again. (Isn’t math wonderful?) In general, the transition happens around whatever number of bits K makes 2ᴷ ≈ N, namely K = log₂(N).

If you reveal significantly fewer than 29 bits under pseudonym B, then it’s almost guaranteed that there’s someone else out there who matches all of them. But if you reveal significantly more than 29 bits, then there’s almost no chance that anyone else exists who matches all of them. So the attacker essentially knows that A and B are the same person. And I stress again: They know that without needing to see anything from the other 490 million people.

Of course, we don’t literally leak bits of immutable feature strings as we write. But you can make the model more realistic, and the same issue persists. If you want to reflect that text only provides noisy information about the writer, then you can add noise to the bits before they’re revealed. If you want to reflect that some writing styles are more common than others, then you can make the distribution over bit strings non-uniform. If you want to reflect that certain quirks are more obvious than others, you can give different bits different probabilities of being revealed. All these make the math more complicated. But they don’t change the basic conclusion: If your writing style contains at least 29 bits of information, and you do enough writing, you’re done.

That’s my argument that pseudpocalypse is possible. But I don’t just want to claim that it could happen, eventually. I think it is likely to happen, soon, and that the amount of text you need to reveal isn’t very large. To make that argument, we need to get specific: What features do people have that are reflected in their writing? How many bits of information do those features contain? How accurately can those bits be guessed from written text?

Note: To avoid this turning into a giant information theory lecture, I’ll mostly use words like “bit” and “information” without being 100% fully precise about what they mean. I’m doing that because I expect that most people reading this aren’t definition-of-bit fetishists, and anyway being hyper-technical would obscure the big picture. If you’re an information theory enthusiast and/or skeptical that I know what I’m doing, I refer you to the Section For Skeptical Information Theory Enthusiasts, below. Until then, use your intuition and have faith.

Feature space

Say you knew nothing about me other than that I wrote the above words. And say you had to guess my age or religion or occupation. You could guess, right? It wouldn’t be perfect, but you’d do much better than you would without being able to read those words. Thus, somehow, those words contain information about my demographic characteristics. So I tried to make a list of similar things that you could plausibly guess from text at least somewhat better then chance. Here’s what I came up with:

  • Age
  • Education
  • Ethnicity
  • Family status
  • Income
  • Marital status
  • Mental health
  • Native language
  • Occupation
  • Physical health
  • Political leanings
  • Region
  • Religious affiliation
  • Sex

In the same spirit, if you only read the above words, could you guess how extroverted or conscientious I am? Again, not perfectly. (When I meet people who read this blog, they usually seem surprised I can survive direct sunlight.) But still, I’m sure you’d do OK. So, again, these words contain information about my personality.

What features does personality have? The HEXACO model lists six, namely honesty-humility, emotionality, extraversion, agreeableness, conscientiousness, and openness to experience. I suspect those can all be guessed with reasonable accuracy from a long-enough writing sample. But could you guess more? For each of those six factors, the HEXACO model lists four “facets”. In the abstract, trying to guess 6 × 4 = 24 different personality features from text sounds ludicrous, but just look at them:

  • Honesty-humility
    • Sincerity
    • Fairness
    • Greed avoidance
    • Modesty
  • Emotionality
    • Fearfulness
    • Anxiety
    • Dependence
    • Sentimentality
  • Extraversion
    • Social self-esteem
    • Social boldness
    • Sociability
    • Liveliness
  • Agreeableness
    • Forgivingness
    • Gentleness
    • Flexibility
    • Patience
  • Conscientiousness
    • Organization
    • Diligence
    • Perfectionism
    • Prudence
  • Openness to experience
    • Aesthetic appreciation
    • Inquisitiveness
    • Creativity
    • Unconventionality

If you think about specific people, I think you can convince yourself that these 24 represent real things, and that it’s plausible to guess them from text. (Your favorite existential angst + science blogger, for example, might score lower on “modesty” than the other honesty-humility facets.) The different sub-factors are surely correlated, but not perfectly correlated.

Of course, the biggest thing you learn from people’s writing is how they write. Do they tend to pointlessly split infinitives? Do they use hyphen-connected words? Do they, incorrectly, position their adverbial clauses?

The idea of attributing authorship using writing style features goes back to at least 1440, when Lorenzo Valla demonstrated that the Donation of Constantine—in which Emperor Constantine supposedly donated the Roman Empire to the Catholic Church—used a vernacular that came from 400 years after Constantine’s death and was therefore a forgery. In 1851, Augustus De Morgan observed that average word length tends to be stable for the same author. The first “modern” attempt seemingly came in 1964, when Mosteller and Wallace published Inference in an Authorship Problem:

This study [attempts] to solve the authorship question of The Federalist papers; […]

Word counts are the variables used for discrimination. Since the topic written about heavily influences the rate with which a word is used, care in selection of words is necessary. The filler words of the language such as an, of, and upon, and, more generally, articles, preposition, and conjunctions provide fairly stable rates, whereas more meaningful words like war, executive, and legislature do not.

After an investigation of the distribution of these counts, the authors execute an analysis […] based on Bayesian methods. The conclusions about the authorship problem are that Madison rather than Hamilton wrote all 12 of the disputed papers.

Get that? The idea is that your usage of the word war depends mostly on if you happen to be talking about war. But your usage of upon mostly depends mostly on how much you like the word upon. To demonstrate this, they took 48 papers written by Hamilton and 50 by Madison and made this table of how many times they used by, from, and to:

Madison liked by. Hamilton was more a to man. Using these kinds of statistics, they concluded that the disputed Federalist papers must have been written by Madison.

So I did some research looking for other writing style features that are believed to be stable when people write about different subjects. I found that there are a lot. There were so many that I struggle to even organize them into meaningful groups:

Low-level frequencies:

  • Word lengths
  • Sentence lengths
  • Paragraph lengths
  • Punctuation frequencies (commas, colons, dashes, parentheses)
  • Function word frequencies (the, of, and, to)
  • Adverb frequencies
    • Intensifiers (very, really, quite, pretty, so)
    • Evidential markers (apparently, evidently, obviously)
    • Downtoners (somewhat, fairly, rather)
  • Pronoun usage
    • Overall preferences (I/we vs. you vs. he/she/they)
    • Third-person singular preferences (he, she, he or she, they, one)
  • Modal verbs (can, could, might, must, should, will, would)
  • Hedges (perhaps, maybe, possibly, probably)
  • Conjunctions (and, but, yet, so)
  • Known stable ratios (the/a, this/that, these/those, I/me/my)
  • Character N-grams (3-grams and 4-grams)
  • Word N-grams (often 3-grams)

Lexical features:

  • Vocabulary size
  • Lexical diversity / type-token ratio (Number of distinct words divided by number of words.)
  • Frequencies of rare words
  • Semantic density
  • Discourse marker positions, combinations (So, anyway, so anyway)
  • Use of abbreviations and acronyms
  • Preference for latinate vs. germanic words (The majestic creature traversed the terrain vs. the mighty beast strode across the land.)

Syntactic features:

  • Syntactic complexity
    • Subordination index
    • Average parse tree depth
  • Use of passive voice.
  • Nominalization (She was shocked I ate the pizza vs. My pizza consumption shocked her)
  • Verb tense and aspect (I walk vs I walked vs I was walking vs I have walked)
  • Sentence structure preferences:
    • Branching preferences (Cursed everyone had a good time when Alice taught some cool dogs I met and brought to dinner to juggle vs. clumsy-but-readable I met some dogs and they were cool and I took them to dinner and Alice taught them to juggle and and everyone had a good time.)
    • Adverbial clause positioning (Suddenly I was hungry vs. I was, suddenly, hungry vs. I was hungry, suddenly)
    • Sentence-final weight (Your plan won’t work because of the dyslexic bears vs. Dyslexic bears mean your plan won’t work.)
  • Polysyndeton (I like dogs, cats, and ferrets vs. I like dogs and cats and ferrets.)
  • Repetition / breaking of syntactic structures.

Style features:

  • Register / formality.
  • Patterns in sentence length (long/short/long/short vs. long/long/short/short)
  • Stressed syllable interval preferences (e.g. iambic vs. trochaic)

Rule preference features:

  • Minor punctuation (I laughed—you cried vs. I laughed — you cried, “…” (three periods) vs. “…” an actual ellipsis)
  • Capitalization. (Job titles, seasons, after a colon, mistakes)
  • Apostrophes (Steve Jobs’ car vs Steve Jobs’s car, 1990’s vs 1990s)
  • Hyphenation (a highly-stable feature vs a highly stable feature)
  • Oxford commas.
  • Article omissions (Local dog was petted. vs. A local dog was petted.)
  • Relative pronoun omissions (the dog you petted vs. the dog that you petted)
  • Who vs. whom.
  • Split infinitives (To obsessively blog vs. to blog obsessively)

Idiosyncratic features:

  • Whitespace habits.
  • Spelling errors (loose instead of lose)
  • Grammar errors. (Between you and I)
  • Consistent, unique typos
  • Other consistent errors (repeated words, un-closed parentheses)

That’s a lot. There are surely more. And these are all “shallow” features that humans came up with using our tiny little brains. I strongly suspect that there are many more “deep” features that could be found by looking for statistical patterns in a sufficiently large dataset. Many of those features might not even have a coherent English-language description. But they’re still there, providing bits for those who seek them.

So we leak information about lots of different stuff when we write. But how much information? Is it possible to say how many words are needed to uniquely fingerprint someone?

No. To a first approximation, the answer is no. But to a second approximation, maybe? Within an order of magnitude? I’ll try, but it’s going to be hard.

Demographic bits

How many bits of identifying information does text provide by way of demographic features like age and sex and so on?

At first glance, this seems a perilous question, as it depends on the number of categories you consider those things to have. Take sex. For pseudpocalypse purposes, your opinion about how sex should be defined or how many sexes exist is irrelevant. Finer categorizations always provide more information, and our de-pseudonymizing attacker friends will use that information if they can. However, going beyond two categories for sex makes little difference, because the additional categories will be hard to guess and even if you could, categories with low prevalence don’t contribute much extra information.3 So, for us, two categories is the right answer.

And what about age? At first glance, converting age into a set of categories seems meaningless. If you code age by the millisecond, then there are 3.156 trillion categories for people born in the last 100 years. If you code age by the decade, there are only 10. Here, the thing to notice is that while you might be able to guess my decade of birth from how I write, you don’t have a snowball’s chance in hell of guessing the millisecond. (See what I did there? People born in certain decades are more likely to use expressions like snowball’s chance in hell?4) If we took age to have some crazy number of categories, we’d have to discount later to reflect the difficulty of guessing. My intuition is that it would be hard to guess age more accurately than around five years, so 20 categories seems reasonable.

Following this kind of logic, I chose a number of categories for each of the demographic variables, trying to hit the upper end of what could be guessed from text. (I’ll provide the actual categories below.)

Feature Number of categories
Age 20
Education level 6
Ethnicity 6
Family status 2
Income 11
Marital status 3
Mental health 3
Native language 2
Occupation 23
Physical health 3
Political leanings 3
Region 23
Religious affiliation 3
Sex 2

If each of the age bins were equally likely, then knowing what bin someone fell into would provide 4.32 bits of information, because 2ᴷ ≈ 20 when K = 4.32. Doing that same calculation for each feature gives the maximum amount of information they could contain.

Feature Number of categories Maximum bits
Age 20 4.32
Education level 6 2.58
Ethnicity 6 2.58
Family status 2 1
Income 11 3.46
Marital status 3 1.58
Mental health 3 1.58
Native language 2 1
Occupation 23 4.52
Physical health 3 1.58
Political leanings 3 1.58
Region 23 4.52
Religious affiliation 3 1.58
Sex 2 1
Total   32.88

But there’s a problem. There are more people aged 30-35 than there are people aged 90-95. So, even if you could guess those age bins perfectly, they’d provide less than 4.32 bits of information on average. However, it turns out that categories need to get pretty damned uneven before information content drops very much. A perfectly balanced 50/50 distribution provides 1 bit of information, but if you switch to a 60/40 distribution, you still get 0.971 bits, and you need to go almost to 90/10 before information content drops to 0.5 bits.5 The same basic thing is true when there are more than two categories.6

So I went through all those features, rated them by how unevenly people are distributed, and tried to discount the bits accordingly. I’ve put the full details of what the original categories are and how I discounted them in a footnote.7

Feature Number of categories Maximum bits Estimated bits
Age 20 4.32 3.9
Education level 6 2.58 2.1
Ethnicity 6 2.58 1.7
Family status 2 1 0.8
Income 11 3.46 2.5
Marital status 3 1.58 1.2
Mental health 3 1.58 0.9
Native language 2 1 0.6
Occupation 23 4.52 4.0
Physical health 3 1.58 1.3
Political leanings 3 1.58 1.5
Region 23 4.52 3.5
Religious affiliation 3 1.58 1.5
Sex 2 1 1
Total   32.88 26.5

But there’s another problem. Female 65 to 70 year-old Asians living in Scotland tend to have different {occupations, family statuses, religious affiliations} than 15 to 20 year-old Latinos living in Southeast Australia. That is, the above features are correlated. So as you look at more of them, they gradually become less surprising and thus contribute less information.

How much less? Answering that the right way would require us to estimate how likely someone is to fall into each of the 20 × 6 × 6 × 2 × 11 × 3 × 3 × 2 × 23 × 3 × 3 × 23 × 3 × 2 = 8,144,737,920 joint categories. That seems hard. But a not-completely-ridiculous approximation is that if a group of variables are all pairwise correlated at a level of ρ>0, then the total information might be reduced by a fraction of ρ.8

So how correlated are those features? In the social sciences, a correlation of 0.5 is considered quite high. That’s plausible for some pairs of variables, e.g. age vs. health or political leaning vs. religious affiliation. But many of those correlations are are probably quite weak, e.g. age vs. native language or region vs. sex vs. marital status.9

Overall, my guess is that correlations reduce the total information by at least 10% but I doubt they reduce it by more than 60%. So I’d think the total information in the above features (if you could guess the categories perfectly) is somewhere between 10.6 and 23.9 bits. Let’s take the average and call it 17.2 bits.

Personality bits

What about personality features? Let’s use the same same recipe we used for demographic features, but faster: To start, let’s give each of the 24 personality features five bins, in deference to dynomight personality notation. That would correspond to 24 × 2.32 = 55.68 bits total, because 2ᴷ ≈ 5 when K = 2.32.

Then we need to discount for correlations. The six main HEXACO personality factors are designed to be uncorrelated, but the different “facets” inside each factor are correlated (usually with a coefficient between 0.3 and 0.6). It seems reasonable to use an overall discount factor of 0.3 to reflect strong intra-factor correlations but weak inter-factor correlations. That suggests 39.0 bits overall.

Style bits

And what about writing style features? How much information do they contain?

This seems hard. Some of the features, like character n-grams are actually themselves long lists of features. (Frequency of typing aaa, frequency of typing aab, etc.) However, many of those features contain little information, since almost everyone types zqx around 0% of the time. And, of course, writing style features are correlated, since people who write realise instead of realize are less likely to put spaces around their em-dashes.

In absence of a better idea, I’m going to give one bit for each leaf node in the above list of style features. I think of this as giving each feature two bins, and then assuming that uneven distributions of features and correlations (which reduce information) are canceled out by the fact that many features deserve more than one bin and that there are probably more “deep” features that aren’t listed (which increase information). This gives us the suspiciously round number of 50.0 bits.

Guessing bits

If you believe the above numbers, then we have at least 17.2 + 39.0 + 50.0 = 106.2 bits of identifying information that we leave clues about when we write. That’s a lot. If you could see all those features, it would be enough to identify people even on a planet with 93 million trillion trillion people.

But to argue that the pseudpocalypse is nigh, it’s not enough to argue that those bits exist. We need to argue that they can and will be guessed from a relatively small amount of text.

So obviously we need to talk about nuclear weapons. In a nuclear detonation, many unstable atoms are created. These spontaneously decay into more-stable atoms, in the process emitting radiation. Some types of atoms are very eager to decay, meaning they release a lot of radiation but stop existing within a few weeks (iodine-131). Others are reluctant to decay, meaning they don’t release as much radiation but they stick around for decades (strontium-90). Others stick around for millions of years, but they produce so little radiation that they’re not a big problem (cesium-135).10 So, the residual radiation produced after a nuclear detonation is the sum of many different exponential curves, one for each isotope created during the detonation.

I suspect that identifying bits in text are sort of like that. Your level of formality and your average sentence length are revealed almost immediately. Your preference for latinate vs. germanic words takes a while to come through. And your social boldness and the fact that you live in Queensland rather than Southeast Australia are revealed very slowly, perhaps so slowly that it’s effectively not revealed at all.

Right. So if you start with 106.2 bits, how many of those do you reveal after writing a given number of words?

I will answer that question through the noble method of making up numbers. But first, let’s calibrate. You just read 4500 words written by me. How well could you guess my demographic and personality features? As a sanity check, I gave the above words to an LLM and asked it to guess. It did unnervingly well. It wasn’t always right, but it usually was, and it did a great job of rating the confidence of the individual predictions.

I don’t think there’s any magical explanation for this. The fact is, if you look at the individual personality and demographic features, guessing them just isn’t that hard. So I’m sure you could do just as well. And given enough time, I’m pretty sure you’d do even better for writing style features.

Even so, you’re probably bad at it. Take the example of GeoGuessr, where people guess a location in the world from a random photo. Random people are sort of OK, but if you pick the top natural talents and have them practice obsessively, they’re really good. I don’t think LLMs are particularly good at guessing features from text, either. They weren’t trained for it. It’s just an emergent property of their general intelligence. The information-theoretic limit is surely much higher.

So here’s a very rough cut: After 4500 words, I’d think it’s possible to guess around:

  • 60% of the demographic features
  • 70% of the personality features
  • 80% writing style features

If we model each of those with a separate exponential, and start them at 17.2 / 39.0 / 50.0 bits, then the total number of identifying bits that remain hidden after writing a given number of words is as plotted here:11

Et voilà, pseudonymity is compromised when you leak 29 bits, which happens after 1071 words.

Seriously?

Of course not. The above figure stands on a creaking tower of tenuous assumptions. I’ve gone through the details of deriving that curve not because you should trust it, but because I think seeing the calculations makes the following points hard to argue with:

  • You have far more than 29 bits of identifying information that you leak into your writing.
  • Some of those bits take a long time to get revealed, but others are revealed pretty quickly.
  • There are enough “fast leaking bits” that you can be identified from a writing sample that’s “pretty small”.

I’ve made lots of debatable choices in terms of choosing features, assigning numbers of categories, estimating distributions across those categories, discounting for correlations, and guessing how many bins can be guessed. Those choices are all individually suspect. But the above points are supported by a pretty wide margin of error. You can make different choices, but it seems very hard to avoid concluding that the above three points are true.12

How would this work?

You might be wondering why I’m using so many made-up numbers. After all, there’s a whole field devoted to identifying authors from text, usually called “stylometry” or “authorship attribution”. They have research papers and competitions and all that. However, as best I can tell, state of the art published results look something like this:

  1. Take 50 people.
  2. Get a few hundred writing samples from each author, each 1000-2000 words long.
  3. Now, take a new writing sample from one of those authors.
  4. Do some standard machine learning stuff.
  5. Hey look, the author can be identified with ~95% accuracy!

That sounds OK, but that’s only identifying people against a pool of ~50 authors. For my claim to be true, similar accuracy would have to be possible with 490 million people. That’s seven orders of magnitude more.

The thing is, the methods those papers are using are extremely weak. All the above math assumes that you’re operating at the “information-theoretic limit”, making perfect use of all available information. If you want to get close to that, we now have some idea how to do it: You apply the “modern” machine learning recipe of gigantic dataset + gigantic neural network + gigantic fortune spent on GPUs. My guess is that for us, that would require something on the order of “all the words ever written” + “tens of billions of parameters” + “tens of millions of dollars”. I couldn’t find a single paper that came remotely close to attempting that.

So I don’t think those papers tell us much, for the same reason that a 3rd-order Markov model trained on a few books doesn’t tell us much about how good computers could be at writing text. LLMs have shown that if you use the above recipe, then computers can get close to the information-theoretic limit for generating text.13 So, I suspect that an LLM-level effort could achieve the same thing for identifying authors.

You might also wonder: Why am I talking about this as some possible future technology? Isn’t that technology just LLMs?

I suspect the technology will be quite LLM-like in how it models human language. But current general-purpose LLMs aren’t trained for this task. They’re good at it “by accident”. So, just like specialized chess AIs can crush LLMs at chess, I suspect specialized stylometry methods could crush general-purpose LLMs at stylometry. It’s just that those specialized stylometry methods don’t seem to exist yet, or at least aren’t public.14 So we shouldn’t imagine that current LLMs are anything close to what’s possible, even if you assume that generic LLM progress stopped today.15

Countermeasures

If this is all true, what could be done about it?

The most obvious “countermeasure” would be to get used to it. I mean, imagine that we did live in a world in which everyone literally had to sign everything they wrote with a unique immutable string. What would happen? I’d expect a mixture of:

  1. People become more comfortable with their “full selves” being public, with less compartmentalization.
  2. People pull back from communicating in public channels, relying more on group chats and the like.
  3. People self-censor.

There are strong historical analogies here, since over the past 20 years many governments and tech companies have in fact decreed that people must sign the things they write with their real names.

The effects seem to vary quite a lot based on the ambient culture and political system. Overall, my impression is that people are already much more comfortable with the idea that their work colleagues might read their dating profile or learn that they go to furry conventions. I’m optimistic that culture will continue to adapt to respect the fact that we all encompass multitudes. This seems healthy.

Some effects seem clearly positive. Self-censoring is not necessarily bad. For example, on the margin, real-names surely stop some teenagers from engaging in cyber-bullying. On the other hand, were you ever a teenager? I’m pretty sure that for anyone who is “different”, having those differences broadcast to the world creates a much larger “bullying surface area”. So the effects are mixed. And adults aren’t as different from teenagers as we might like to think.

Twenty years ago, I might have predicted that real names would discourage people from expressing controversial political ideas online. Superficially, that seems completely wrong. At least in the West, lots of people are very happy to express minority political views, and if you disagree at all, then you can go to hell. But I also tend to think this hides a lot of self-censorship, where most people don’t want engage in political mortal combat and so are cowed by a feisty minority. And, obviously, people in certain countries know that it’s unwise to criticize the Party. So, getting used to it seems like an imperfect solution at best.

Another countermeasure would be to not build this technology, or not make it widely available. In the short term, this seems plausible. As far as I can tell, it’s been possible for years for a modestly-funded group to build a phone app that would identify most people on the street from a photo. And yet, almost no one reading this has access to such an app. If general-purpose LLMs continue to get better at stylometry, it seems entirely possible that AI companies might decide it’s a safety issue and train their AIs to refuse to do it.16 This could work for a while.

But if the technology is possible, it seems certain that governments will build it and use it. They might try to keep it out of the hands of normal people. Certain governments might restrict their own use. My privacy-minded allies always seem very jaded, but it wouldn’t surprise me at all if the Supreme Court declared that a warrant was needed before the FBI could de-pseudonymize a U.S. citizen. But when/if that technology becomes sufficiently cheap, it seems like it would be very difficult to keep it out of the hands of normal people and/or bad actors. My guess is that it’s possible to create a program that’s a few hundred gigabytes large and can run (slowly) on most modern laptops. If that program is made public, it would be hard to put the genie back in the bottle.

There are also technological countermeasures. Most obviously, you could run your writing through a “filter” to try to remove identifying bits, e.g. by asking an LLM to rewrite it. It’s hard to be sure how well this would work, since we don’t have accurate estimates of how many bits you’re starting with or how many bits this would remove. But I’d guess this would be pretty effective if done carefully. The reason is that the number of identifying bits you leave in writing probably isn’t that large, relative to the number needed to identify you. If you “homogenize” your writing to remove all style and personality, you should be able to remove most of those bits. Theoretically, you’ll still leak some information. But I’d think this would substantially increase the amount you could write while remaining pseudonymous.17

But after thinking about it, this makes me sad. Effectively, this countermeasure would preserve pseudonymity by taking writing and destroying all traces of humanity. It seems like this would work well for the “bad” uses of pseudonymity, like cyber-bullying or coordinated violence, but it wouldn’t work at all for the “good” uses, like for example someone who likes to write pseudonymously because they feel like it allows them to be more honest and vulnerable and more fully themselves, damn it.

Generalized pseudpocalypse

Maybe this isn’t just true for writing. Maybe it’s just a feature of our universe that if you interact with the world in any significant way, then you leave traces that make it possible to identify you.

  • If you walk around in public, then you can likely be identified by your face, your gait, your voice, your DNA, your retinas, or your literal fingerprints.

  • Or say you use the internet. Even if you lock down your browser fingerprint and hide your IP address using a VPN or Tor, a sufficiently powerful adversary could still identify you by analyzing global packet flow.

  • Or say you use any phone or computer. You might be identified through keystroke dynamics or the way you jiggle your finger or mouse.

  • Say you buy food at the grocery store, but you pay with cash and somehow shop at a grocery store with no cameras. If you buy more than a handful of items, I’d bet you can still be identified through the patterns in the stuff you buy.

  • (Incidentally, did you ever notice that cash has serial numbers on it? And did you know that more and more ATMs are starting to track those numbers?)

  • Or say you don’t like your car being tracked, so you stop carrying a phone and somehow get lawmakers to outlaw license plates. Still, your car surely has a few small unique scratches, and the engine probably doesn’t sound exactly the same as other cars, even from the same model and year. So if there’s any high-resolution video or audio, that’s still enough to track you.

  • Say you plug your headphones into a charging station at the airport. Your headphones have eccentricities in their analog charging circuits. If someone really wanted to, they could track that.

  • Or say you use electricity. Given high-resolution power-usage data, what can be said about how many people live with you? And what devices you’re using? Probably a lot?

  • Or say you use a toilet. Many places already test sewage and know, at a population level, what drugs people are using and how prevalent various diseases are. Imagine this was upgraded to test many places in the system, with high temporal resolution, possibly correlated with flow measurements from individual houses. That would be exciting.

  • Or say you are a country and you have submarines. Can they be detected by adversaries using distributed acoustic sensing? What about satellite-based synthetic aperture radar? Gravity Gradiometers? Quantum magnetometry?

As far as I can tell, the general trend is that without countermeasures, almost everything can be identified. Countermeasures can make it harder, but they’re costly, and on the whole, the arms race seems to favor the identifier, not the person who doesn’t want to be identified.

I stress: This is not all bad. The goodness / badness of a generalized pseudpocalypse depends on how society is structured. After all, the foundation of civilization is finding ways for people to make deals, and arguably less privacy makes that easier. The degree that we live in a vulnerable world where it’s easy to create civilization-destroying technologies, perhaps we’re very lucky to find ourselves in a non-private world. Still, I do worry that privacy has long provided a kind of “slack” from laws and norms. Historically, that slack has limited the power of institutions to enforce their rules. If privacy is going away, we need to think about how to preserve slack, particularly when institutions don’t want to.

Appendix: Section for skeptical information theory enthusiasts

Above, I tried to estimate the number of bits of identifying information in writing. But what is a “bit”? In general, if x is a discrete random variable, then the Shannon entropy of x in bits is H(x) = ∑ₓ p(x) log₂(1/p(x)), where the sum is over all the values x can take. This is always bounded between zero and the logarithm of the number of values x can take.

That’s fine, but “writing style” is not a discrete variable with a discrete number of categories. So how can I estimate the entropy of writing style? The short answer is that I can’t. What I’ve actually estimated above is the mutual information between writing and writing style.

Let s be a random variable representing writing style. Think of this as some sort of high dimensional continuous vector representing all the quirks of how different people write. And let x be a writing sample of some length. This is discrete because we can represent writing on digital computers. Then what I’ve estimated above is the mutual information I(x;s) = H(x) - H(x|s), where H(x|s) is the conditional entropy of x given s. This can be measured in bits because both H(x) and H(x|s) can be measured in bits. So that’s what my estimate above really says: I(x;s) ≈ 106.2 bits.

Now, you still might be skeptical. Above, I’ve implicitly assumed something like the following was true:

It’s possible to identify one person out of N possibilities with low accuracy if and only if the mutual information between identifying features and writing is at least log₂(N) bits.

That’s how I justified pseudonymity being compromised around 29 bits. But is it really true? Strictly speaking, no. Actually, even more strictly speaking, it’s “not even untrue” because it’s not precise enough to be true or false. But as far as I can tell, basically any precise version of that statement is false. However, it’s possible to find versions of that statement that are true, provided you add some extra not-too-crazy assumptions.

To start, let’s consider an extremely simple model of information leakage.

Theorem. Suppose the world consists of you plus N other people, and suppose each person has a binary identity string, drawn at uniform from the distribution over M-bit binary strings. All these strings are known to the attacker. Suppose you pick some subset of K bits and reveal them. Then the probability that this identifies you is

  (1-2⁻ᴷ)ᴺ.

Furthermore, in order to hold the probability of being identified below

  (1-1/N)ᴺ ≈ exp(-1) ≈ 36.7%,

it is necessary that K ≤ log₂(N).

Proof. The probability that all K observed features collide with any random person in the crowd is 2⁻ᴷ. Thus, the probability of no collisions after checking the crowd of N people (meaning you are the only one matching the observed features) is (1-2⁻ᴷ)ᴺ. □

That’s simple. But it’s not realistic at all, since it assumes that people have immutable binary strings that they leak into their writing. Can we make it more realistic?

Well, there is a simple lower bound. That is, we can say in general that if the mutual information is significantly less than log₂(N), then it’s not possible to reliably identify someone.

Theorem. Suppose N random people are selected and their full writing style features are made public. One person from that group is chosen and produces a writing sample. Then, the attacker must guess who produced it. The average success rate of the attacker (averaged over the random pool, the random choice of author, and the random writing sample) is at most (I(x;s)+1)/log₂(N).

Proof. Let S=(s₁, s₂, s₃, …) be the pool of N styles and let n be a random variable indicating which person was chosen. Fano’s inequality says that the highest possible success rate is bounded by the conditional mutual information between the writing sample x and the identity n, conditioning on the pool of writing styles, i.e. the probability of success is at most

  (I(x;n|S)+1)/log₂(N).

However, we can bound that conditional mutual information as

  I(x;n|S) ≤ I(x;n,S) = I(x;n,sₙ) = I(x;sₙ) = I(x;s).

The first inequality is standard. The second step uses the fact that given n, the writing x is conditionally independent of all styles except the chosen writer. The third step uses the fact that n is conditionally independent of x given sₙ. The last step uses that (x,sₙ) is distributed as (x,s). Substituting this bound gives the claimed result. □

So, if mutual information is much less than log₂(N), reliable identification is impossible, even if the attacker knows all the style vectors perfectly. So, provided you don’t leak that many bits, you’re definitely safe.

But is the converse true? Does leaking more than log₂(N) bits always identify you? The general answer is no. The basic problem is that I(x;s) is the average information that an average person leaks in an average writing sample. Without further assumptions, you can construct scenarios where some rare people and writing samples contain gigantic amounts of information, but most people usually leak nothing. That would mean that the attacker is very certain in some cases but usually learns nothing.

So, to get a guarantee that identification is actually possible, you need to make some kind of additional assumption that the information leakage rate doesn’t vary too much between different writers or between different things they write.

Suppose that p(x,s) is the joint distribution over writing styles s and writing samples x. Let’s suppose that the attacker knows the true style vector ŝ for some person. Then, they will be given a writing sample x that either came from that person or came from a randomly chosen person, and must decide which. Formally, the attacker’s goal is to guess if x was sampled from the writing distribution for that person, p(x|ŝ) or from the population marginal p(x). Intuition suggests that the attacker’s best strategy will be to look at the ratio

  p(x|ŝ)/p(x),

and “accept” x as coming from ŝ if above some threshold, and reject it otherwise. In fact, the Neyman-Pearson lemma guarantees that this is the optimal strategy, in a very strong sense: That ratio contains all the information that’s useful for making that decision.

Now here’s something interesting: Instead of looking at the ratio, the attacker could look at the logarithm of the ratio. It makes no difference since it’s monotonic. But if you take the logarithm of that ratio, and take the expectation over people and over texts, what do you get? Well:

  𝔼 ln (p(x|s)/p(x)) = 𝔼 ln (p(x,s)/(p(x) p(s))) = I(x;s)

It’s the mutual information! So, intuitively, the mutual information is how much an attacker learns about the style of the writer “on average”, where that average is over both writers and text.

The following theorem will look at the average information in text for a writer with a particular style. I’ll define this as

  D(s) = KL(p(X|s) || p(X)).

Intuitively, this is how different the writing of someone with style s is from the population average. That’s because if you take the average of this value over different styles, you get the mutual information. That is, I(x;s) = 𝔼[D(s)].18

Theorem (informal). Suppose that the attacker will observe some text and wishes to classify it as either coming from a writer with specific known style ŝ, or coming from someone with a random style. Suppose that the attacker is only willing to tolerate some small risk ε of a false positive. Provided that D(ŝ) is significantly larger than -ln(ε), the attacker can achieve that, while also keeping the risk of false negatives very low, provided that the variance of how much information is revealed in a random writing sample is bounded.

Theorem. Let D(ŝ) = KL(p(X|ŝ) || p(X)) to be the divergence between the target’s writing distribution and the marginal distribution. Also, define qₜ(x) ∝ p(x|ŝ)ᵗ p(x)¹⁻ᵗ to be the family that interpolates between those two distributions. To formalize the idea that “information leakage” for ŝ doesn’t vary that much, we assume that some constant V exists such that for 0 < t < 1, the variance of log(p(x|ŝ)/p(x)) under qₜ is bounded by V.

Then for any ε satisfying exp(-D) < ε < exp(-D + ½ V), it is possible for the attacker to simultaneously achieve a false positive rate of FPR ≤ ε and a false negative rate of FNR ≤ exp( - ½ (D+ ln ε)² / V). This false positive rate reflects the mistake rate provided the writing sample x came from a randomly chosen other person, while the false negative rate reflects the mistake rate provided the writing sample x actually came from the person with style ŝ.

Proof sketch. Let f be the distribution of l(x) = log(p(x|ŝ)/p(x)) with respect to p(x|ŝ) and let g be the distribution of l(x) with respect to p(x). The stated variance assumption implies a quadratic bound K(u) ≤ D u +½ V u^2 for -1 < u < 0, where K is the cumulant generating function of f. Observe that g is an exponential tilting of f. The attacker’s strategy must be to “accept” x as coming from ŝ if l is above some threshold c and “reject” it otherwise. Use K in a Chernoff bound on the probability l is less than c under f to upper-bound FNR ≤ exp( - ½ (D-c)²/V). Now, using that g(l) = exp(-l) f(l), again use K in a Chernoff bound on the probability l exceeds c under g to upper-bound FPR ≤ exp( -c - ½ (D-c)²/V). Both of these bounds are simultaneously valid when D-V < c < D. Setting c to make the false-positive bound equal to ε gives FPR ≤ ε and FNR ≤ exp( -½ (V - √(V² - 2V(D + ln ε)))²/V). The latter can be relaxed into the stated result using that √(1-x) ≤1-x/2 for 0 ≤ x ≤ 1. □

Now, if we suppose that the attacker wants to find a particular person, with a particular known style s. And suppose that the attacker has a pool of N people and will see one writing sample from each, but wants to limit the total probability of a false positive to δ after seeing one sample from each person. Then, they will need that

  (1-ε)ᴺ ≈ exp(-εN) = (1-δ),

which is satisfied by ε ≈ δ/N. Substituting this into the previous result says that the attacker can hold the total risk of a false positive to δ while achieving a false-negative risk of

  FNR ≤ exp( - ½ (D(s) + ln δ - ln N)² / V).

These results use natural logarithms because the math is easier if you measure information in nats. If you measure information in bits then you would get log₂ δ and log₂ N. (Rescaling D and V appropriately.)

So, again, as long as the average information for user s is significantly larger than log₂ N, the attacker can identify that user with minimal risk of false positives.

Some writers might leak more information (higher D(s)) and some writers might leak less information (lower D(s)). But remember, I(x;s)=𝔼 D(s). So as long as information leakage doesn’t vary too much between people, and assuming that I(x;s) is much larger than log₂ N (and assuming that variance condition), almost everyone can be identified.

  1. Editor’s note: After this sentence was written, many additional hours were devoted to further idiotic tinkering. ↩

  2. It’s fine. ↩

  3. A standard binary variable that is 0 or 1 with 50% probability conveys 1 bit of information, while a variable that is 0 / 1 / 2 with probability 49.8% / 49.8% / 0.4% conveys 1.0336 bits. ↩

  4. People born in certain decades are also presumably more likely to employ see what I did there gambits. ↩

  5. For example, here is the information content for seven different “bent coins”:

    Probability of landing heads Information
    0.50 (fair coin) 1.000
    0.60 0.971
    0.70 0.881
    0.80 0.722
    0.90 0.469
    0.95 0.286
    0.99 0.081

    ↩

  6. Here’s a more formal looking version of the table from the previous footnote:

    p(A) p(B) Information
    0.50 0.50 1.000
    0.60 0.40 0.971
    0.70 0.30 0.881
    0.80 0.20 0.722
    0.90 0.10 0.469
    0.95 0.05 0.286
    0.99 0.01 0.081

    You can generate that table by running this code:

    from scipy.stats import entropy
     
    dists = ([0.5, 0.5], [0.6, 0.4], [0.7, 0.3], [0.8, 0.2], [0.9, 0.1], [0.95, 0.05], [0.99, 0.01])
    entropies = [entropy(p, base=2) for p in dists]
    
    print("| p(A) | p(B) | Entropy |")
    print("|------|------|---------|")
    for i in range(len(dists)):
        print(f"| {dists[i][0]:<4.3f} | {dists[i][1]:<4.3f} | {entropies[i]:<7.3f} |")
    

    With three categories, the story is much the same. Things need to get quite uneven before information drops too much:

    p(A) p(B) p(C) Entropy
    0.333 0.333 0.333 1.585
    0.400 0.300 0.300 1.571
    0.500 0.250 0.250 1.500
    0.600 0.200 0.200 1.371
    0.700 0.150 0.150 1.181
    0.800 0.100 0.100 0.922
    0.900 0.050 0.050 0.569
    0.950 0.025 0.025 0.336
    0.990 0.005 0.005 0.091

    You can generate that with this code:

    from scipy.stats import entropy
    
    dists = (
        [1/3, 1/3, 1/3],
        [.4, .3, .3],
        [.5, .25, .25],
        [.6, .2, .2],
        [.7, .15, .15],
        [.8, .1, .1],
        [.9, .05, .05],
        [.95, .025, .025],
        [.99, .005, .005]
    )
    entropies = [entropy(p, base=2) for p in dists]
    
    print("| p(A) | p(B) | p(C) | Entropy |")
    print("|------|------|------|---------|")
    for i in range(len(dists)):
        print(f"| {dists[i][0]:<4.3f} | {dists[i][1]:<4.3f} | {dists[i][2]:<4.3f} | {entropies[i]:<7.3f} |")
    

    ↩

  7. Roughly speaking, we we should discount those maximum bits as follows:

    • Near even: No discount.
    • “Mildly uneven” (E.g. 70/30 with two categories) Discount by 10%.
    • “Quite uneven” (E.g. 90/10 with two categories) Discount by 50%.
    • “Extremely uneven” (E.g. 99/1 with two categories) Discount by 90%.

    The Shannon entropy of a categorical distribution is - Σᵢ pᵢ log₂ pᵢ. Or, in python:

    import math
    def entropy(probs):
    	return sum(-p * math.log2(p) for p in probs)
    

    Age: It’s hard for me to imagine you could guess age from text with accuracy higher than 5 years. If you assume an age between 0 and 100, that would be 20 categories and log2(20)=4.32 bits. These are mildly non-uniform so I’ll reduce to 3.9.

    Education: I’m assuming 6 categories: less than high school, high school, some college, finished college, master’s degree, doctorate. That would be log2(6)=2.58 bits, but fairly uneven, so I’ll reduce by 20% to reflect that.

    Ethnicity: Assuming 62% white, 11% black, 16% latino, 6% asian, 1.5% indigenous, 3.5% mixed/other, and actually using the entropy formula.

    Family status: I’m using two categories: Children / no children, on the logic that guessing the number of children would be very hard. These are mildly non-uniform, so I’ll drop to 0.8 bits. You could have a third category for having children that are grown and that had left home, but this would be heavily redundant with age.

    Income: The US census gives 11 income brackets. That seems as good a way of discretizing as anything. That would be log2(11) = 3.459 bits, but these are again moderately non-uniform, so I’ll reduce to 2.5.

    Marital status: I’m taking 3 categories (single, married, divorced / widowed / etc). That would be log2(3)=1.58 bits at maximum, but again these are somewhat non-uniform, so I dropped that to 1.2.

    Mental health: I’m using 3 categories: “Healthy”, “chronic condition”, and “severe issues”. Assuming 73% healthy 25% chronic condition, 2% “severe issues”, and using the entropy formula gives 0.9 bits.

    Native language: I’m using 2 categories, namely “English native”, and “non-English native”. These are pretty uneven inside the Anglosphere, so I’ll drop from 1 bit to 0.6 bits.

    Occupation. The BLS classification gives 23 major groups. That would be log2(23)=4.523 bits, but it’s moderately non-uniform, so I’ll reduce to 4 bits.

    Physical health: Assuming 60% “healthy” 30% “chronic condition” 10% “severe issues” and using the entropy formula.

    Political leanings: I’m using three categories (left, center, right). These are fairly uniform so I’m using 1.58 bits.

    Region: I asked an LLM to divide the Anglosphere up into a number of regions with reasonable granularity. With some tinkering, it gave 23 regions: South East England, South West England, Midlands, Northern England, Scotland, Wales, Republic of Ireland, Northern Ireland, Quebec, Ontario, Western Canada, Atlantic Canada, Northeast US, Southern US, Midwest US, Western US, Alaska, Hawaii, Southeast Australia, Western Australia, Queensland, Central & Southern Australia, New Zealand. With LLM-generated population estimates (which looked reasonable) and plugging into the entropy formula, this gave 3.5481 bits.

    # Region Pop (M) pi (Pop/Total) log2⁡(pi) pilog2⁡(pi)
    1 South East England 20.0 0.04062 -4.617 -0.1875
    2 South West England 6.0 0.01219 -6.353 -0.0774
    3 Midlands 11.0 0.02234 -5.485 -0.1225
    4 Northern England 20.0 0.04062 -4.617 -0.1875
    5 Scotland 5.5 0.01117 -6.484 -0.0724
    6 Wales 3.0 0.00609 -7.359 -0.0448
    7 Republic of Ireland 5.0 0.01015 -6.626 -0.0673
    8 Northern Ireland 2.0 0.00406 -7.949 -0.0323
    9 Quebec 9.0 0.01828 -5.774 -0.1055
    10 Ontario 16.0 0.03250 -4.943 -0.1606
    11 Western Canada 13.0 0.02640 -5.247 -0.1385
    12 Atlantic Canada 2.5 0.00508 -7.625 -0.0387
    13 Northeast US 56.0 0.11373 -3.136 -0.3568
    14 Southern US 130.0 0.26401 -1.922 -0.5074
    15 Midwest US 69.0 0.14013 -2.836 -0.3973
    16 Western US 80.0 0.16247 -2.624 -0.4264
    17 Alaska 0.7 0.00142 -9.467 -0.0134
    18 Hawaii 1.4 0.00284 -8.790 -0.0249
    19 Southeast Australia 16.0 0.03250 -4.943 -0.1606
    20 Western Australia 3.0 0.00609 -7.359 -0.0448
    21 Queensland 5.5 0.01117 -6.484 -0.0724
    22 Central & Southern Australia 2.5 0.00508 -7.625 -0.0387
    23 New Zealand 5.3 0.01076 -6.539 -0.0703
      Sum of pilog2⁡(pi)       -3.5481

    Religious affiliation: 3 categories (christian, other religion, atheist / agnostic). These are uniform-ish.

    Sex: 2 categories, near-even ↩

  8. Consider a set of binary random variables, each of which is equally likely to be 0 and 1, yet all are correlated with a pairwise correlation coefficient of ρ. There are many distributions that satisfy this condition, but a natural choice is an Ising model. If there are many variables, then the entropy per-variable in an Ising model with pairwise correlations of ρ tends to h((1+√ρ)/2), where h is the binary entropy function. We can print out those numbers:

    ρ h((1+√ρ)/2)
    0.0000 1.00000000
    0.1000 0.92661216
    0.2000 0.85048963
    0.3000 0.77121926
    0.4000 0.68826012
    0.5000 0.60087604
    0.6000 0.50801160
    0.7000 0.40803633
    0.8000 0.29811751
    0.9000 0.17212786
    1.0000 0.00000000

    As you can see, the entropy per-variable is always a bit more than 1-ρ. But the Ising model is optimistic, in the sense that it has the highest entropy of all distributions meeting the given conditions. So, screw it, let’s estimate the entropy per-variable to just be 1-ρ. ↩

  9. If it means anything to you, I asked Kimi 2.6 to hallucinate some numbers:

      Age Edu Eth Fam Inc Mar Mhe Nlg Occ Phe Pol Reg Rel Sex
    Age 1.0 -0.2 0.0 0.6 0.1 0.5 -0.1 0.0 0.2 -0.5 0.1 0.0 0.2 -0.1
    Edu -0.2 1.0 0.3 0.2 0.6 0.2 0.1 0.1 0.7 0.3 0.3 0.2 -0.2 -0.1
    Eth 0.0 0.3 1.0 0.2 0.3 0.1 -0.1 0.7 0.3 -0.3 0.2 0.4 0.4 0.0
    Fam 0.6 0.2 0.2 1.0 0.2 0.7 -0.1 0.0 0.1 0.0 0.1 0.0 0.2 0.1
    Inc 0.1 0.6 0.3 0.2 1.0 0.3 -0.2 0.1 0.7 0.3 0.1 0.2 0.0 -0.1
    Mar 0.5 0.2 0.1 0.7 0.3 1.0 0.2 0.0 0.1 0.2 0.1 0.0 0.2 0.0
    Mhe -0.1 0.1 -0.1 -0.1 -0.2 0.2 1.0 0.0 -0.2 0.4 0.0 0.0 -0.1 0.1
    Nlg 0.0 0.1 0.7 0.0 0.1 0.0 0.0 1.0 0.1 0.0 0.1 0.5 0.3 0.0
    Occ 0.2 0.7 0.3 0.1 0.7 0.1 -0.2 0.1 1.0 0.1 0.2 0.2 0.0 0.3
    Phe -0.5 0.3 -0.3 0.0 0.3 0.2 0.4 0.0 0.1 1.0 0.0 0.1 0.0 0.1
    Pol 0.1 0.3 0.2 0.1 0.1 0.1 0.0 0.1 0.2 0.0 1.0 0.5 0.4 0.1
    Reg 0.0 0.2 0.4 0.0 0.2 0.0 0.0 0.5 0.2 0.1 0.5 1.0 0.2 0.0
    Rel 0.2 -0.2 0.4 0.2 0.0 0.2 -0.1 0.3 0.0 0.0 0.4 0.2 1.0 0.1
    Sex -0.1 -0.1 0.0 0.1 -0.1 0.0 0.1 0.0 0.3 0.1 0.1 0.0 0.1 1.0

    Personally, this doesn’t mean very much to me… ↩

  10. It’s more complicated than this, because some atoms (e.g. strontium-90) emit more energy per decay than others. And some types of radiation are more harmful to human life than others. ↩

  11. In general, if you want an exponential curve f(n) that starts at 1 for n=0 and decays to 1-X for n=N, you should choose f(n) = exp(n × ln(1-X) / N). So for demographic features we’re using X=0.6 and N = 4500, meaning f(n) = exp(-0.00020362 × n). For personality features, we’re using X=0.7, meaning f(n) = exp(-0.00026755 × n), and for writing style features, we’re using X = 0.8, meaning f(n) = exp(-0.000357653 × n). So the total number of bits remaining hidden is 17.2 × exp(-0.00020362 × n) + 39.0 × exp(-0.00026755 × n) + 50.0 × exp(-0.000357653 × n). ↩

  12. OK, what’s the most likely reason I might be wrong? Above, I used math to estimate the information in features, and then I basically made up numbers for how much of that information can be guessed from text. Even so, my greatest concern is that the first part. I’m a bit worried that I might be overestimating the amount of information in the features themselves due to inadequately discounting for correlations. For one thing, there are probably correlations between feature groups. (For example, I’d bet that people who are high in perfectionism are less likely to use lose and loose interchangeably, and that people who live in Northern England are more likely to use the character string colour than people who live in Hawaii.) Also, my crude method of discounting information by ρ due to pairwise correlations of ρ might not discount enough: I used an estimate based on an Ising model, which is the maximum-entropy (highest information) distribution given the correlation constraints. I haven’t been able to figure out how much lower the information could be in the worst-case. ↩

  13. People debate if this is true for “intelligence”, but it’s definitely true in terms of bit-rate. ↩

  14. Also, arguably, stylometry is about language. This means that large language models probably have much of what they need baked in. That might explain why they’re pretty good at it just “by accident”. But to do this optimally I think they’d need self-reflection (e.g. access to probabilities of text given different contexts) that current LLMs aren’t typically capable of, and wouldn’t know how to manipulate correctly without task-specific training. ↩

  15. You could conjecture that near-optimal stylometry abilities are some kind of “emergent property”. But the general lesson so far is that LLMs mostly don’t have emergent properties but are just good at what they’re trained at. (Edit: I withdraw this sentence!) ↩

  16. (Meta-joke about you—person who works at an AI company—thinking, “maybe we should do that”, coming to this footnote, and seeing this meta-joke.) ↩

  17. Instead of “homogenizing” writing by imposing a generic style, perhaps it would be better to “camouflage” it by enforcing a very strong but random style. ↩

  18. Be a little careful here: Typically, the KL-divergence is understood to be measured in nats. But in this article, I’ve measured mutual information in bits. That’s fine, but you need to convert. For example, 106.2 bits = 73.60 nats. ↩

Life with hazard ratios

By: dynomight
6 July 2026 at 00:00

If you read anything about health or longevity, you’ll soon find yourself in a world of hazard ratios. Some study might say that eating more fiber might change your risk of dying by a factor of HR = 0.90. Another might say that occasional smoking might change it by HR = 1.30.

But how much should you care about that? Is HR = 0.90 or HR = 1.30 a lot? What if you don’t want to eat more fiber? What if you like smoking?

Instead of staring at a ratio1, a more sensible thing to do is think about life expectancy.2 But is it possible to convert a hazard ratio to a change in life expectancy? You might reason as follows: Baseline life expectancy is around 75 years. And HR = 0.90 corresponds to a 10% decrease in mortality. So perhaps that hazard ratio corresponds to something like 7.5 extra years of life expectancy?

Unfortunately, that’s completely wrong. To see why, imagine that humans only die by playing Russian roulette. They start playing this once per day at the age of 75, with a revolver containing two bullets and six chambers. If you were to remove one of those two bullets, that would drop the person’s risk of death by HR = 0.5. (One bullet versus two.) But life expectancy would barely change, because even with just one bullet, almost nobody would survive for any significant amount of time past 75.

For contrast, imagine again that humans only die via Russian roulette, but now they do this once per day from birth with a revolver with 2 bullets and 54,786 chambers. (Newborns emerge and instinctively reach for this gigantic gun.) You can show that these people also live 75 years on average. But now, if you remove one of the bullets, life expectancy doubles, because when someone is spared, it takes a long time before they get unlucky again.3

Neither of those is a good model for humans. We’re somewhere between the two, with heart disease and so on instead of revolvers and risks slowly rising as we age instead of suddenly starting at age 75 or staying constant throughout life. But you get the point: If you want to convert a hazard ratio for some intervention to a change in life expectancy, the impact depends on how “spread out” baseline mortality risk is over time. Baseline life expectancy is simply not enough information.

That’s one problem. Here’s another: What even is a hazard ratio? The technical definition is something like:

The hazard ratio at a given time is the rate of an event in the treatment group divided by the rate of that event in the control group.

Hazard ratios are often confused with their more beloved siblings, relative risks. Say you run a trial for 10 years and at the end, 10% of the control group died and 8% of the treatment group. Then the relative risk is RR = 0.8, nice and simple. But relative risks have problems, most notably that if you run a long enough trial, then no one will be alive at the end no matter the intervention, meaning RR = 1.0. That’s not helpful. Intuitively, you can think of the hazard ratio at age 40 as sort of like the relative risk for people between the ages of 39.99 and 40.01.

In real life, interventions have different hazard ratios at different ages. Chemotherapy tends to have better results in younger patients who are more able to endure the side-effects. Having a slightly higher BMI (25-30 rather than 20-25) is associated with an increased risk of mortality in young people, but a decreased risk in the elderly. You may remember from 2020 that COVID’s mortality risk had a different age curve than baseline mortality, meaning the hazard ratio of getting COVID was different at different ages.

This is important, because hazard ratios at different ages have different impacts on life expectancy. A hazard ratio of 0.9 at age 80 prevents more deaths than at age 20, because baseline mortality is higher at 80. But at the same time, if you save the life of a 20 year-old, they have more years in front of them. Beyond that, the hazard ratios at different ages interact: If some intervention decreases mortality at younger ages, that allows more people to reach older ages, increasing how much hazard ratios matter at older ages.4

If we knew the hazard ratio at all ages, we could account for those dynamics. But we don’t, because when estimating hazard ratios, people almost always assume that the hazard ratio is constant.5 We’re quasi-forced to do this because there’s not enough data to estimate a whole time-series of ratios. That’s why papers contain single numbers like HR = 0.90.

So even though Intervention A (say, more fiber) and Intervention B (say, light jogging) might have the same hazard ratio in a paper, those numbers could be the product of different underlying age-dependent effects, meaning those interventions could conceivably lead to vastly different changes in life expectancy.

So is this all hopeless? Are single hazard ratio numbers just too far removed from what we care about to tell us anything meaningful?

Surprisingly, no. It’s mostly OK. If we were a different species, it might be hopeless. But for modern humans in rich countries, mortality happens to be distributed in a way that produces a sort of lucky coincidence: When people estimate constant hazard ratio numbers, they’re implicitly sorta-kinda taking a weighted average of hazard ratios at different ages. And those weights happen to (sorta-kinda) reflect how much changes in mortality at different ages change.

So, I will argue, even if the true intervention has a varying effect, it’s sorta-mostly OK to just take a hazard ratio from a paper and convert it to a change in life expectancy using this curve:

dl_vs_hr_log

If a paper showed that eating more fiber produces a hazard ratio of HR = 0.75, that corresponds to an increase of around 3.7 years. If a paper says that occasional smoking produces a hazard ratio of HR = 1.25, that corresponds to a decrease of around 2.9 years.

This isn’t exact. If the intervention is better (or less bad) for older people this will tends to overestimate the increase (or underestimate the decrease) in life expectancy. If the intervention is worse (or less good) for older people, it will tend to underestimate the increase (or overestimate the decrease) in life expectancy. But as long as the hazard ratio doesn’t vary too much by age, it’s probably not off by more than around 30% in either direction.

The easy case

Say there’s some intervention (eating more fiber or whatever) that multiplies your risk of dying at age t by a factor of HR(t). Then it can be shown that this changes life expectancy by approximately

  ΔL ≈ ∑ₜ ΔHR(t) × P(t) × L(t).

Here, P(t) is the baseline probability of dying at age t. For males in the United States, it looks like this:

Meanwhile, L(t) is conditional life expectancy at age t. That’s the average number of additional years left for someone who reaches age t. For males in the United States, it looks like this:

Finally, ΔHR(t) is the decrease in hazard at age t. You can think of that as just ΔHR(t) = 1 - HR(t). Though if you’re OK with logarithms, there’s a somewhat better approximation that uses logarithms, which I’ve quarantined in a footnote.6

Let’s start with the easy case. What if your intervention has the same effect on mortality at all ages, so HR(t)=HR is just a constant? Then, the above equation simplifies into

  ΔL ≈ ΔHR × L̄,

where

  L̄ = ∑ₜ P(t) × L(t).

This makes sense! Again, P(t) is the baseline probability of dying at age t and L(t) is conditional life expectancy at age t. These are constant, so when you add them up, L̄ is just a number. For males in the United States, it happens to be 12.93 years. This quantity has a specific meaning: The average remaining life expectancy for US males when they die. That sounds a bit odd, but think of picking a random death and asking how many additional years people who reach that age live on average. That number is 12.93 years.

So, if an intervention has a constant hazard ratio, the mean change in life expectancy for US males is just

  ΔL ≈ ΔHR × 12.93 years.

Now we’re getting somewhere! If you prevent a fraction ΔHR of deaths, then you increase life expectancy by ΔHR times 12.93 years.

Now remember the naive calculation we started with: Life expectancy for US males is 75.8 years. You might hope that if eating more fiber drops your risk of death by 10%, that would save 7.58 years. Sadly, the above equation says that a 10% drop in risk only increases life expectancy by around 1.293 years—only 0.17 times as much.

This is essentially the observation Keyfitz made in his 1977 paper, “What Difference Would It Make if Cancer Were Eradicated?” Cancer is responsible for 18 percent of deaths, so does that mean eradicating it would increase lifespan by 18 percent, or around 13.6 years? Nope, Keyfitz says, it’s only 2.3 years.

If a cure for cancer were discovered and made available today, 350,000 cancer deaths would be avoided in the next year. The overall death rate would be lower by nearly 18 percent. If the cure were quick and inexpensive, a large fraction of the country’s hospital beds and medical personnel would be released for treatment of other ailments. Patients would be spared untold suffering. Such an implicit analysis underlies government proposals for eradication of cancer. The argument is sound for first effects on mortality but wholly misleading for the long term.

The first effects would soon be offset by more mortality from diseases other than cancer. As a result of the cancer cures, the population would include a higher proportion of people subject to other causes of death. […]

At the extreme, it might be said that everyone dies of something sooner or later, so that, when the effects of the eradication of cancer had shaken down, the same number of deaths would occur as before, and the only benefit would be the substitution of heart and other diseases for cancer. A cure for cancer would only have the effect of giving people the opportunity to die of heart disease.

Cheerful stuff! We can also write our approximation in terms of baseline life expectancy as

  ΔL ≈ ΔHR × 0.17 × 75.8 years,

which makes explicit that 12.93 years is only 0.17 times as large as a naive estimate using baseline life expectancy. The discount factor of 0.17 is sometimes called the “Keyfitz entropy”. You can think of it as measuring how close some population is to playing Russian roulette with 2 bullets in 6 chambers starting at age 75 (a discount factor of just above 0) and playing Russian roulette from birth with 2 bullets and 54,786 chambers (a discount factor of 1.0). It’s typically around 0.15 in rich countries today, though it was historically much higher.

Keyfitz entropy is also much higher in other species like mice (perhaps 0.45). You could argue that this explains why nothing that increases lifespan in mice ever translates to humans. Say caloric restriction or whatever produced the same constant hazard ratio in mice and humans. Then it’s mathematically guaranteed that the percentage increase in life expectancy will be three times smaller in humans, because Keyfitz entropy is three times smaller in humans. It’s harder to increase life expectancy when the baseline mortality distribution is more compressed.7

But that’s all assuming the hazard ratio is the same at all ages. Which it surely isn’t.

The interesting case

Here again is our equation for the change in life expectancy in response to taking some action that changes the risk of mortality at age t by a factor of HR(t):

  ΔL ≈ ∑ₜ ΔHR(t) × P(t) × L(t),

Basically, for each age t, we multiply together three numbers:

  1. ΔHR(t) is the decrease in the chance of dying at age t as a result of whatever intervention you’ve made (e.g. eating more fiber). This reflects that larger decreases in risk lead to larger increases in life expectancy.
  2. P(t) is the baseline probability of dying at age t. This reflects that the hazard ratio is a ratio, so you prevent more deaths when you apply that ratio to ages where the baseline rate is higher.
  3. L(t) is conditional life expectancy at age t. This reflects that you miss out on more years of life if you die when you’re young.

Now notice: The impact of a change ΔHR(t) at age t is the product of the baseline risk of death P(t) and remaining life expectancy L(t). So what really matters is their product, P(t) × L(t):

This shows how sensitive life expectancy is to changes in hazard ratios at different ages. It would be nice if this were constant. Then, the shape of HR(t) wouldn’t matter at all, only the average value. That’s not quite true, but it’s not terribly far from being true.

An equivalent way of writing our equation for the change in life expectancy is

  ΔL ≈ avg(ΔHR) × L̄,

where L̄ is still mean “life expectancy at death” (12.93 years for US males) and avg(ΔHR) is the average change in hazard, weighted by the P(t) × L(t) sensitivity curve at different ages.8 While that sensitivity curve isn’t constant, it’s not too curvy, either. Intuitively, it gives a lot of weight to ages between 50 and 90, somewhat less weight to ages between 20 and 50, and little weight to other ages.9

So that’s not too bad. But let’s remember our original problem: You see some number like HR = 0.90 in a paper, and you want to convert it to a change in life expectancy. If the true underlying hazard ratio were constant, then there’s no problem. But if it’s not constant, then what does that HR = 0.90 number even mean?

Numbers in papers

Unfortunately, you almost never get to see the underlying time-dependent HR(t), because there’s almost never enough data to estimate it. So it’s almost never possible to compute the weighted average avg(ΔHR). In reality what you have is probably a single number in a paper. Let’s call that number est(HR). The obvious thing to do would be to plug the change into the above equation in place of avg(ΔHR) and approximate the change in life expectancy as

  ΔL ≈ est(ΔHR) × L̄.

Again, you can just think of est(ΔHR) = 1-est(HR) as being the estimated reduction in hazard. Although, again, I’d prefer you use logarithms if you’re OK with logarithms.10 So the question is: Will that be accurate? How close are est(ΔHR) and avg(ΔHR)?

Well, how do people actually estimate those scalar hazard ratio numbers in papers? Somehow, they’re aggregating together information about hazards at different ages into a single number. But how? Well, it’s complicated. But if there’s a lot of data, you can show that the estimated scalar hazard ratio is approximately11

  est(HR) ≈ Πₜ HR(t)ᵖ⁽ᵗ⁾.

(Pardon the hideous typsetting.) That is, the estimated hazard ratio is the geometric average of age-dependent hazard ratios, weighted by the probability of dying at each age. It follows12 that the estimated change in hazard is approximately

  est(ΔHR) ≈ ∑ₜ P(t) ΔHR(t).

So ideally, we’d estimate life expectancy using avg(ΔHR), which averages the changes ΔHR(t) based on the weights P(t) × L(t). But we can’t do that, because we don’t have access to the ΔHR(t) numbers. What we can do is read a hazard ratio number in a paper, call it est(HR) and then compute the change est(ΔHR). The above equation says that if you do that, you are implicitly (and approximately) averaging the changes ΔHR(t) based on the weights P(t) alone.

The “right” weights used by avg(ΔHR) and the “wrong” weights implicitly used by est(ΔHR) aren’t the same. But they’re not that different. Here’s P(t) × L(t), the weights that we’d like to use to compute avg(ΔHR) and estimate changes in life expectancy accurately:

And here’s P(t), the weights you’re implicitly using if we take a hazard ratio number from a paper and compute est(ΔHR):

They’re different. In particular, the latter weights give more weight to people aged 80-95 and less weight to people aged 20-50. But they’re not terribly different.

Enough math, let’s try it

To start, imagine some intervention that decreases risk by HR(t)=0.9 for all ages.

Here are the results:

Thing Formula Years
Original life expectancy L 75.7769
New life expectancy L’ 76.4127
Exact ΔL ΔL = L - L’ 0.6358
Ideal approximation ΔL ≈ avg(ΔHR) × L̄ 0.6409
Use number from paper ΔL ≈ est(ΔHR) × L̄ 0.6409

Let me explain what’s happening here. I made a simulator that takes actuarial data for how likely US males are to die at various ages. From this, it’s a simple spreadsheet calculation to compute life expectancy L.13 Then I applied a hazard ratio to change the probability of dying at each age, and re-ran the simulator to compute a new life expectancy L’ and the exact difference ΔL. Then I’m showing two approximations of ΔL: The first is the “ideal approximation” using avg(ΔHR), which I’m including mostly to show that my math is good. Finally, I’m showing the approximation you get if you actually fit a Cox proportional hazards model and use the resulting number in est(ΔHR). This corresponds to what you’d get if you plug in a number from a paper.

So, with the above constant hazard ratio HR = 0.90, both approximations are very good. This remains true if you switch to some other constant.

What if the hazard ratio varies? At first, you might think that something like this would be very problematic:

But it’s basically fine:

Thing Formula Years
Original life expectancy L 75.7769
New life expectancy L’ 77.4373
Exact ΔL ΔL = L - L’ 1.6604
Ideal approximation ΔL ≈ avg(ΔHR) × L̄ 1.7451
Use number from paper ΔL ≈ est(ΔHR) × L̄ 1.7121

The reason this is fine is that the changes in the hazard ratio are relatively “high frequency”, meaning they sort of locally average out. To demonstrate this, suppose the hazard ratio is chosen randomly for each 1-year bin:

Then the approximations are even better:

Thing Formula Years
Original life expectancy L 75.7769
New life expectancy L’ 77.4218
Exact ΔL ΔL = L - L’ 1.6449
Ideal approximation ΔL ≈ avg(ΔHR) × L̄ 1.7059
Use number from paper ΔL ≈ est(ΔHR) × L̄ 1.7123

What causes trouble is if the hazard ratio varies systematically between the young and the old. For example, suppose the intervention is useless for newborns, but gradually becomes more helpful as you get older:

My “ideal approximation” would still be pretty accurate, if you could compute it. (Which you can’t, in the real world.) But using a number from a paper leads to an overestimate:

Thing Formula Number
Original life expectancy L 75.7769 years
New life expectancy L’ 77.9031 years
Exact ΔL ΔL = L - L’ 2.1261 years
Ideal approximation ΔL ≈ avg(ΔHR) × L̄ 2.0962 years
Use number from paper ΔL ≈ est(ΔHR) × L̄ 2.7645 years

This happens because est(ΔHR) is implicitly weighted by P(t) which is heavily weighted towards older people, whereas we’d like to use something more like avg(ΔHR) which is weighted by P(t) × L(t) which is somewhat less weighted towards older people. Even so, the error isn’t terrible.

Now, it is possible that plugging in a hazard ratio from a paper could give wildly inaccurate estimates of life expectancy. One such scenario would be an intervention which is amazing for people aged 85-95, but does nothing for anyone else:

Now, the hazard ratio looks good exactly at the ages where est(ΔHR) has the most weight, leading it to hugely overestimate the impact on life expectancy:

Thing Formula Number
Original life expectancy L 75.7769 years
New life expectancy L’ 76.1741 years
Exact ΔL ΔL = L - L’ 0.3972 years
Ideal approximation ΔL ≈ avg(ΔHR) × L̄ 0.3840 years
Use number from paper ΔL ≈ est(ΔHR) × L̄ 1.0989 years

Another nightmare case is an intervention that starts out harmful, but then switches to being helpful at older ages:

Now, using a number from a paper doesn’t even give an estimate with the right sign.

Thing Formula Number
Original life expectancy L 75.7769 years
New life expectancy L’ 75.5006 years
Exact ΔL ΔL = L - L’ -0.2764 years
Ideal approximation ΔL ≈ avg(ΔHR) × L̄ -0.2348 years
Use number from paper ΔL ≈ est(ΔHR) × L̄ +0.2709 years

That’s bad. But I think most interventions probably aren’t like that? My guess is that most real interventions vary somewhat with age, but they do so gradually and without switching sign. In those cases, it’s quite difficult to find cases where plugging in the number from a paper is off by more than 30% or so. If you don’t believe me, just try it.14

TLDR

If we were another species, it might be very hard to convert from hazard ratios to changes in life expectancy. But for modern people in rich countries, there are three lucky coincidences:

  1. Mortality risk happens to be distributed so that you can approximate changes in life expectancy through a simple weighted sum of hazard ratios at different ages, ignoring interactions.
  2. The statistical method that people use to estimate scalar hazard ratios can also be approximated as a weighted sum of hazard ratios at different ages, ignoring interactions.
  3. The weights that you need to estimate life expectancy (from #1) and the weights that are implicitly used to compute hazard ratio numbers (from #2) aren’t the same. But they’re fairly close.

These facts justify taking an estimated hazard ratio number HR from a paper and approximating the change in life expectancy as ΔL ≈ ln(1/HR) × 12.93 years or, if the hazard ratio is close to one and you hate logarithms, as ΔL ≈ (1-HR) × 12.93 years.

dl_vs_hr_both

The number 12.93 years is for US males. It’s the product of Keyfitz entropy (0.17) and baseline life expectancy (75.8 years). It will vary a bit in other populations.

If the true underlying hazard ratio:

  • …is constant across ages, then the above approximation will be extremely good.
  • …decreases as people get older, that approximation will overestimate ΔL. That is, it will make helpful interventions look better than they actually are, and it will make harmful interventions look less bad than they actually are.
  • …increases as people get older, that approximation will underestimate ΔL. That is, it will make helpful interventions look less good than they actually are, and it will make harmful interventions look worse than they actually are.

But as long as the true underlying hazard ratio isn’t too crazy, there’s probably not more than ~30% error in either direction.

Finally, two major caveats: First, the above discussion assumes that the hazard ratio was estimated by running a trial on people of all ages. In general, est(ΔHR) implicitly gives weight to different ages proportional to how many deaths occur at those ages in the baseline population in the trial. If there’s a minimum age of, say, 50 years old, that won’t change too much because most of the mass of P(t) is above the age of 50 anyway. But if there’s a minimum age of 70, or a maximum age of 50, that could make a huge difference if the true hazard ratio is different at the ages that weren’t seen.

Second, these are estimates for the life expectancy for a population. But you are not a population. In some sense, your genetics and lifestyle mean you have your own “personal Keyfitz entropy”, reflecting how spread out your mortality would be for you if you led millions random lives. If you drive safely and use an air purifier and eat well and get exercise and don’t smoke, that likely means your personal life expectancy is higher than average. But it also probably means that your personal Keyfitz entropy is lower than average.15 So, if you make your lifestyle even better by eating more fiber or whatever, even if that produces the same hazard ratio for you as for other people, it would still likely lead to smaller increases in life expectancy, for the same reason that the same hazard ratio produces smaller changes in lifespan in humans compared to mice. What we really need is some interventions strong enough to break the math behind these approximations and free us from Keyfitz tyranny.

  1.  ↩

  2. I know, I know, you care about quality of life, not just years of life. I agree, some number that measures health and vitality, maybe disability-adjusted life years or quality-adjusted life years, would be better. But these are hard to estimate and so are rarely reported. Anyway, in practice most interventions that make you more vital tend to make you live longer and vice versa, so focusing on life expectancy isn’t too bad. ↩

  3. In this model, the number of days of life follows a geometric distribution with p = (number of bullets) / (number of chambers). So the mean life expectancy is 1/p days or (number of chambers) / (number of bullets) days. With 54,786 chambers and 2 bullets, that works out to 75 years. And if you drop down to one bullet, then it increases to 150 years. ↩

  4. If some intervention would have reduce mortality among people aged ≥ 60 in prehistorical tribal bands, that wouldn’t have increased life expectancy very much, because most people didn’t make it to 60. But compared to prehistorical tribal bands, we have in fact vastly reduced mortality at younger ages. And so, today, reducing mortality for people aged ≥ 60 will increase life expectancy a lot. ↩

  5. You might think this is stupid. Why change a relative risk into a hazard ratio if you’re just going to assume it’s constant? Isn’t that pointless? Well, no. Remember how relative risks always go to 1.0 for long enough trials as everyone in both the treatment and control groups departs our coil? That doesn’t happen with constant hazard ratios. ↩

  6. It’s usually (though not always) better to use ΔHR(t) = ln(1/HR(t)). This correctly reflects, for example, that if all hazard ratios go to zero, then life expectancy goes to infinity, yay. These two approximations are almost identical for hazard ratios that are close to one because ln(1/r) ≈ (1-r) when r is close to one. So if you are terrified of logarithms but you’ve made it to the end of this footnote anyway, you’re not missing out on too much. ↩

  7. There’s a degree of circularity to this argument. It assumes that hazard ratios transfer better between species than changes in life expectancy. That might be true, but it would be an empirical / biological fact, not something that’s guaranteed by logic. ↩

  8. To see this, note that ΔL ≈ ∑ₜ ΔHR(t) × P(t) × L(t) = L̄ × ∑ₜ ΔHR(t) × (P(t) × L(t) / L̄) = L̄ × avg(ΔHR). ↩

  9. A pretty decent approximation turns out to be

      avg(ΔHR) ≈ 0.27 × avg₂₀₋₅₀(ΔHR) + 0.73 × avg₅₀₋₉₀(ΔHR),

    where avg₂₀₋₅₀(ΔHR) represents a flat average of the change over the ages 20 to 50 and avg₅₀₋₉₀(ΔHR) represents a flat average over the ages 50 to 90. ↩

  10. That is, it’s better to use est(ΔHR) = ln(1/est(HR)). This is close to 1-est(HR) when est(HR) is close to one. ↩

  11. If there is an infinite amount of data, the typical method reduces to solving

      ∑ₜ (P(t) + P’(t)) × π(t, HR) = ∑ₜ P’(t),

    for HR. Here, P’(t) is the chance of dying at age t after the hazard ratio has been applied, and π(t, HR) is the probability that, if a death occurred at time t, it was in the treatment group. Of course, the true probability that a death is in the treatment group is P’(t) / (P(t) + P’(t)). The standard “proportional Cox” model assumes that the hazard ratio is constant and so replaces this raw fraction with a model-based one, namely

      π(t, HR) = S’(t) × HR / (S(t) + S’(t) × HR).

    This reflects the fact that at age t, a fraction S(t) of controls are alive and each of these have some chance μ(t) of dying, so P(t)=S(t) × μ(t). Meanwhile, a fraction S’(t) of the treatment group is alive, and these each have a chance HR × μ(t) of dying, meaning that P’(t) = S’(t) × HR × μ(t). If you substitute these equations for P(t) and P’(t) into the second equation above, the factor of μ(t) conveniently cancels and you get π(t, HR) as written.

    In effect, the hazard ratio’s job is to attribute deaths to the treatment versus the control group. Now, if the true time-varying HR(t) is close to one, then it can be shown that the estimated hazard ratio est(HR) approximately satisfies

      ln(est(HR)) ≈ ∑ₜ P(t) ln(HR(t)). ↩

  12. The geometric average is equivalent to the condition that

      ln(est(HR)) ≈ ∑ₜ P(t) ln(HR(t))

    Using the “better” approximation that ΔHR(t) = ln(1/HR(t)) and *est(ΔHR)=ln(1/est(HR)), it follows that

      est(ΔHR) ≈ ∑ₜ P(t) ΔHR(t).

    You can justify interpreting that same equation using est(ΔHR) = 1-est(HR) and ΔHR(t)=1-HR(t) from the fact that these are almost the same when HR(t) is close to one. ↩

  13. This simulator pretends that people live for integer numbers of years. That’s not true in reality, of course, but it makes the simulator easier to implement and understand and makes little difference in practice. ↩

  14. In the simulation, “true ΔL” is what I called “exact ΔL” above, while “approximation (log)” is what I called “ideal approximation” and “Cox fitted” is what I called “Use number from paper”. ↩

  15. The way modern human mortality is distributed, even if your healthy lifestyle were to reduce mortality by a constant factor at all ages, that still has the effect of decreasing Keyfitz entropy. ↩

Good if make prior after data instead of before

By: dynomight
18 December 2025 at 00:00

They say you’re supposed to choose your prior in advance. That’s why it’s called a “prior”. First, you’re supposed to say say how plausible different things are, and then you update your beliefs based on what you see in the world.

For example, currently you are—I assume—trying to decide if you should stop reading this post and do something else with your life. If you’ve read this blog before, then lurking somewhere in your mind is some prior for how often my posts are good. For the sake of argument, let’s say you think 25% of my posts are funny and insightful and 75% are boring and worthless.

OK. But now here you are reading these words. If they seem bad/good, then that raises the odds that this particular post is worthless/non-worthless. For the sake of argument again, say you find these words mildly promising, meaning that a good post is 1.5× more likely than a worthless post to contain words with this level of quality.

If you combine those two assumptions, that implies that the probability that this particular post is good is 33.3%. That’s true because the red rectangle below has half the area of the blue one, and thus the probability that this post is good should be half the probability that it’s bad (33.3% vs. 66.6%)

(Why half the area? Because the red rectangle is ⅓ as wide and ³⁄₂ as tall as the blue one and ⅓ × ³⁄₂ = ½. If you only trust equations, click here for equations.)

It’s easiest to calculate the ratio of the odds that the post is good versus bad, namely

P[good | words] / P[bad | words]
 = P[good, words] / P[bad, words]
 = (P[good] × P[words | good])
 / (P[bad] × P[words | bad])
 = (0.25 × 1.5) / (0.75 × 1)
 = 0.5.

It follows that

P[good | words] = 0.5 × P[bad | words],

and thus that

P[good | words] = 1/3.

Alternatively, if you insist on using Bayes’ equation:

P[good | words]
 = P[good] × P[words | good] / P[words]
 = P[good] × P[words | good]
 / (P[good] × P[words | good] + P[bad] × P[words | bad])
 = 0.25 × 1.5 / (0.25 × 1.5 + 0.75)
 = (1/3)

Theoretically, when you chose your prior that 25% of dynomight posts are good, that was supposed to reflect all the information you encountered in life before reading this post. Changing that number based on information contained in this post wouldn’t make any sense, because that information is supposed to be reflected in the second step when you choose your likelihood p[good | words]. Changing your prior based on this post would amount to “double-counting”.

In theory, that’s right. It’s also right in practice for the above example, and for the similar cute little examples you find in textbooks.

But for real problems, I’ve come to believe that refusing to change your prior after you see the data often leads to tragedy. The reason is that in real problems, things are rarely just “good” or “bad”, “true” or “false”. Instead, truth comes in an infinite number of varieties. And you often can’t predict which of these varieties matter until after you’ve seen the data.

Aliens

Let me show you what I mean. Say you’re wondering if there are aliens on Earth. As far as we know, there’s no reason aliens shouldn’t have emerged out of the random swirling of molecules on some other planet, developed a technological civilization, built spaceships, and shown up here. So it seems reasonable to choose a prior it’s equally plausible that there are aliens or that there are not, i.e. that

P[aliens] ≈ P[no aliens] ≈ 50%.

Meanwhile, here on our actual world, we have lots of weird alien-esque evidence, like the Gimbal video, the Go Fast video, the FLIR1 video, the Wow! signal, government reports on unidentified aerial phenomena, and lots of pilots that report seeing “tic-tacs” fly around in physically impossible ways. Call all that stuff data. If aliens weren’t here, then it seems hard to explain all that stuff. So it seems like P[data | no aliens] should be some low number.

On the other hand, if aliens were here, then why don’t we ever get a good image? Why are there endless confusing reports and rumors and grainy videos, but never a single clear close-up high-resolution video, and never any alien debris found by some random person on the ground? That also seems hard to explain if aliens were here. So I think P[data | aliens] should also be some low number. For the sake of simplicity, let’s call it a wash and assume that

P[data | no aliens] ≈ P[data | aliens].

Since neither the prior nor the data see any difference between aliens and no-aliens, the posterior probability is

P[no aliens | data] ≈ P[aliens | data] ≈ 50%.

See the problem?

(Click here for math.)

Observe that

P[aliens | data] / P[no aliens | data]
 = P[aliens, data] / P[no aliens, data]
 = (P[aliens] × P[data | aliens])
 / (P[no aliens] × P[data | no aliens])
 ≈ 1,

where the last line follows from the fact that P[aliens] ≈ P[no aliens] and P[data | aliens] ≈ P[data | no aliens]. Thus we have that

P[aliens | data] ≈ P[no aliens | data] ≈ 50%.

We’re friends. We respect each other. So let’s not argue about if my starting assumptions are good. They’re my assumptions. I like them. And yet the final conclusion seems insane to me. What went wrong?

Assuming I didn’t screw up the math (I didn’t), the obvious explanation is that I’m experiencing cognitive dissonance as a result of a poor decision on my part to adopt a set of mutually contradictory beliefs. Say you claim that Alice is taller than Bob and Bob is taller than Carlos, but you deny that Alice is taller than Carlos. If so, that would mean that you’re confused, not that you’ve discovered some interesting paradox.

Perhaps if I believe that P[aliens] ≈ P[no aliens] and that P[data | aliens] ≈ P[data | no aliens], then I must accept that P[aliens | data] ≈ P[no aliens | data]. Maybe rejecting that conclusion just means I have some personal issues I need to work on.

I deny that explanation. I deny it! Or, at least, I deny that’s it’s most helpful way to think about this situation. To see why, let’s build a second model.

More aliens

Here’s a trivial observation that turns out to be important: “There are aliens” isn’t a single thing. There could be furry aliens, slimy aliens, aliens that like synthwave music, etc. When I stated my prior, I could have given different probabilities to each of those cases. But if I had, it wouldn’t have changed anything, because there’s no reason to think that furry vs. slimy aliens would have any difference in their eagerness to travel to ape-planets and fly around in physically impossible tic-tacs.

But suppose I had divided up the state of the world into these four possibilities:

possibility description
No aliens + normal people There are no aliens. Meanwhile, people are normal and not prone to hallucinating evidence for things that don’t exist.
No aliens + weird people There are no aliens. Meanwhile, people are weird and do tend to hallucinate evidence for things that don’t exist.
Normal aliens There are aliens. They may or may not have cool spaceships or enjoy shooting people with lasers. But one way or another, they leave obvious, indisputable evidence that they’re around.
Weird aliens There are aliens. But they stay hidden until humans get interested in space travel. And after that, they let humans take confusing grainy videos, but never a single good video, never ever, not one.

If I had broken things down that way, I might have chosen this prior:

P[no aliens + normal people] ≈ 41%
P[no aliens + weird people] ≈ 9%
P[normal aliens] ≈ 49%
P[weird aliens] ≈ 1%

Now, let’s think about the empirical evidence again. It’s incompatible with no aliens + normal people, since if there were no aliens, then normal people wouldn’t hallucinate flying tic-tacs. The evidence is also incompatible with normal aliens since is those kinds of aliens were around they would make their existence obvious. However, the evidence fits pretty well with weird aliens and also with no aliens + weird people.

So, a reasonable model would be

P[data | normal aliens] ≈ 0	
P[data | no aliens + normal people] ≈ 0
P[data | weird aliens] ≈ P[data | no aliens + weird people].

If we combine those assumptions, now we only get a 10% posterior probability of aliens.

P[no aliens + normal people | data] ≈ 0
P[no aliens + weird people | data] ≈ 90%
P[normal aliens | data] ≈ 0
P[weird aliens | data] ≈ 10%

Now the results seem non-insane.

(math)

To see why, first note that

P[normal aliens | data]
 ≈ P[data | no aliens + normal people]
 ≈ 0,

since both normal aliens and no aliens + normal people have near-zero probability of producing the observed data.

Meanwhile,

P[no aliens + weird people | data] / P[weird aliens | data]
 = P[no aliens + weird people, data] / P[weird aliens, data]
 ≈ P[no aliens + weird people] / P[weird aliens]
 ≈ .09 / .01
 = 9,

where the second equality follows from the fact that the data is assumed to be equally likely under no aliens + weird people and weird people

It follows that

P[no aliens + normal people | data]
 ≈ 9 × P[weird aliens | data],

and so

P[no aliens + weird people | data] ≈ 90%
P[weird aliens | data] ≈ 10%.

Huh?

I hope you are now confused. If not, let me lay out what’s strange: The priors for the two above models both say that there’s a 50% chance of aliens. The first prior wasn’t wrong, it was just less detailed than the second one.

That’s weird, because the second prior seemed to lead to completely different predictions. If a prior is non-wrong and the math is non-wrong, shouldn’t your answers be non-wrong? What the hell?

The simple explanation is that I’ve been lying to you a little bit. Take any situation where you’re trying to determine the truth of anything. Then there’s some space of things that could be true.

In some cases, this space is finite. If you’ve got a single tritium atom and you wait a year, either the atom decays or it doesn’t. But in most cases, there’s a large or infinite space of possibilities. Instead of you just being “sick” or “not sick”, you could be “high temperature but in good spirits” or “seems fine except won’t stop eating onions”.

(Usually the space of things that could be true isn’t easy to map to a small 1-D interval. I’m drawing like that for the sake of visualization, but really you should think of it as some high-dimensional space, or even an infinite dimensional space.)

In the case of aliens, the space of things that could be true might include, “There are lots of slimy aliens and a small number of furry aliens and the slimy aliens are really shy and the furry aliens are afraid of squirrels.” So, in principle, what you should do is divide up the space of things that might be true into tons of extremely detailed things and give a probability to each.

Often, the space of things that could be true is infinite. So theoretically, if you really want to do things by the book, what you should really do is specify how plausible each of those (infinite) possibilities is.

After you’ve done that, you can look at the data. For each thing that could be true, you need to think about the probability of the data. Since there’s an infinite number of things that could be true, that’s an infinite number of probabilities you need to specify. You could picture it as some curve like this:

(That’s a generic curve, not one for aliens.)

To me, this is the most underrated problem with applying Bayesian reasoning to complex real-world situations: In practice, there are an infinite number of things that can be true. It’s a lot of work to specify prior probabilities for an infinite number of things. And it’s also a lot of work to specify the likelihood of your data given an infinite number of things.

So what do we do in practice? We simplify, usually by limiting creating grouping the space of things that could be true into some small number of discrete categories. For the above curve, you might break things down into these four equally-plausible possibilities.

Then you might estimate these data probabilities for each of those possibilities.

Then you could put those together to get this posterior:

That’s not bad. But it is just an approximation. Your “real” posterior probabilities correspond to these areas:

That approximation was pretty good. But the reason it was good is that we started out with a good discretization of the space of things that might be true: One where the likelihood of the data didn’t vary too much for the different possibilities inside of A, B, C, and D. Imagine the likelihood of the data—if you were able to think about all the infinite possibilities one by one—looked like this:

This is dangerous. The problem is that you can’t actually think about all those infinite possibilities. When you think about four four discrete possibilities, you might estimate some likelihood that looks like this:

If you did that, that would lead to you underestimating the probability of A, B, and C, and overestimating the probability of D.

This is where my first model of aliens went wrong. My prior P[aliens] was not wrong. (Not to me.) The mistake was in assigning the same value to P[data | aliens] and P[data | no aliens]. Sure, I think the probability of all our alien-esque data is equally likely given aliens and given no-aliens. But that’s only true for certain kinds of aliens, and certain kinds of no-aliens. And my prior for those kinds of aliens is much lower than for those kinds of non-aliens.

Technically, the fix to the first model is simple: Make P[data | aliens] lower. But the reason it’s lower is that I have additional prior information that I forgot to include in my original prior. If I just assert that P[data | aliens] is much lower than P[data | no aliens] then the whole formal Bayesian thing isn’t actually doing very much—I might as well just state that I think P[aliens | data] is low. If I want to formally justify why P[data | aliens] should be lower, that requires a messy recursive procedure where I sort of add that missing prior information and then integrate it out when computing the data likelihood.

(math)

Mathematically,

P[data | aliens]
 = ∫ P[wierd aliens | aliens]
 × P[data | wierd aliens] d(weird aliens)
 + ∫ P[normal aliens | aliens]
 × P[data | normal aliens] d(normal aliens).

But now I have to give a detailed prior anyway. So what was the point of starting with a simple one?

I don’t think that technical fix is very good. While it’s technically correct (har-har) it’s very unintuitive. The better solution is what I did in the second model: To create a finer categorization of the space of things that might be true, such that the probability of the data is constant-ish for each term.

The thing is: Such a categorization depends on the data. Without seeing the actual data in our world, I would never have predicted that we would have so many pilots that report seeing tic-tacs. So I would never have predicted that I should have categories that are based on how much people might hallucinate evidence or how much aliens like to mess with us. So the only practical way to get good results is to first look at the data to figure out what categories are important, and then to ask yourself how likely you would have said those categories were, if you hadn’t yet seen any of the evidence.

Futarchy’s fundamental flaw — the market — the blog post

By: dynomight
14 August 2025 at 00:00

Here’s our story so far:

  1. Markets are a good way to know what people really think. When India and Pakistan started firing missiles at each other on May 7, I was concerned, what with them both having nuclear weapons. But then I looked at world market prices: MSCI prices See how it crashes on May 7? Me neither. I found that reassuring.

  2. But we care about lots of stuff that isn’t always reflected in stock prices, e.g. the outcomes of elections or drug trials. So why not create markets for those, too? If you create contracts that pay out $1 only if some drug trial succeeds, then the prices will reflect what people “really” think.

  3. In fact, why don’t we use markets to make decisions? Say you’ve invented two new drugs, but only have enough money to run one trial. Why don’t you create markets for both drugs, then run the trial on the drug that gets a higher price? Contracts for the “winning” drug are resolved based on the trial, while contracts in the other market are cancelled so everyone gets their money back. That’s the idea of Futarchy, which Robin Hanson proposed in 2007.

  4. Why don’t we? Well, maybe it won’t work. In 2022, I wrote a post arguing that when you cancel one of the markets, you screw up the incentives for how people should bid, meaning prices won’t reflect the causal impact of different choices. I suggested prices reflect “correlation” rather than causation, for basically the same reason this happens with observational statistics. This post, it was magnificent.

  5. It didn’t convince anyone.

  6. Years went by. I spent a lot of time reading Bourdieu and worrying about why I buy certain kinds of beer. Gradually I discovered that essentially the same point about futarchy had been made earlier by, e.g., Anders_H in 2015, abramdemski in 2017, and Luzka in 2021.

  7. In early 2025, I went to a conference and got into a bunch of (friendly) debates about this. I was astonished to find that verbally repeating the arguments from my post did not convince anyone. I even immodestly asked one person to read my post on the spot. (Bloggers: Do not do that.) That sort of worked.

  8. So, I decided to try again. I wrote another post called ”Futarky’s Futarchy’s fundamental flaw”. It made the same argument with more aggression, with clearer examples, and with a new impossibility theorem that showed there doesn’t even exist any alternate payout function that would incentivize people to bid according to their causal beliefs.

That post… also didn’t convince anyone. In the discussion on LessWrong, many of my comments are upvoted for quality but downvoted for accuracy, which I think means, “nice try champ; have a head pat; nah.” Robin Hanson wrote a response, albeit without outward evidence of reading beyond the first paragraph. Even the people who agreed with me often seemed to interpret me as arguing that futarchy satisfies evidential decision theory rather than causal decision theory. Which was weird, given that I never mentioned either of those, don’t accept the premise the futarchy satisfies either of them, and don’t find the distinction helpful in this context.

In my darkest moments, I started to wonder if I might fail to achieve worldwide consensus that futarchy doesn’t estimate causal effects. I figured I’d wait a few years and then launch another salvo.

But then, legendary human Bolton Bailey decided to stop theorizing and take one of my thought experiments and turn it into an actual experiment. Thus, Futarchy’s fundamental flaw — the market was born. (You are now reading a blog post about that market.)

June 25

I gave a thought experiment where there are two coins and the market is trying to pick the one that’s more likely to land heads. For one coin, the bias is known, while for the other coin there’s uncertainty. I claimed futarchy would select the worse / wrong coin, due to this extra uncertainty.

Bolton formalized this as follows:

  1. There are two markets, one for coin A and one for coin B.

  2. Coin A is a normal coin that lands heads 60% of the time.

  3. Coin B is a trick coin that either always lands heads or always lands tails, we just don’t know which. There’s a 59% it’s an always-heads coin.

  4. Twenty-four hours before markets close, the true nature of coin B is revealed.

  5. After the markets closes, whichever coin has a higher price is flipped and contracts pay out $1 for heads and $0 for tails. The other market is cancelled so everyone gets their money back.

Get that? Everyone knows that there’s a 60% chance coin A will land heads and a 59% chance coin B will land heads. But for coin A, that represents true “aleatoric” uncertainty, while for coin B that represents “epistemic” uncertainty due to a lack of knowledge. (See Bayes is not a phase for more on “aleatoric” vs. “epistemic” uncertainty.)

Bolton created that market independently. At the time, we’d never communicated about this or anything else. To this day, I have no idea what he thinks about my argument or what he expected to happen.

June 26-27

In the forum for the market, there was a lot of debate about “whalebait”. Here’s the concern: Say you’ve bought a lot of contracts for coin B, but it emerges that coin B is always-tails. If you have a lot of money, then you might go in at the last second and buy a ton of contracts on coin A to try to force the market price above coin B, so the coin B market is cancelled and you get your money back.

The conversation seemed to converge towards the idea that this was whalebait. Though notice that if you’re buying contracts for coin A at any price above $0.60, you’re basically giving away free money. It could still work, but it’s dangerous and everyone else has an incentive to stop you. If I was betting in this market, I’d think that this was at least unlikely.

June 27

Bolton posted about the market. When I first saw the rules, I thought it wasn’t a valid test of my theory and wasted a huge amount of Bolton’s time trying to propose other experiments that would “fix” it. Bolton was very patient, but I eventually realized that it was completely fine and there was nothing to fix.

At the time, this is what the prices looked like:

That is, at the time, both coins were priced at $0.60, which is not what I had predicted. Nevertheless, I publicly agreed that this was a valid test of my claims.

I think this is a great test and look forward to seeing the results.

Let me reiterate why I thought the markets were wrong and coin B deserved a higher price. There’s a 59% chance coin B would turns out to be all-heads. If that happened, then (absent whales being baited) I thought the coin B market would activate, so contracts are worth $1. So thats 59% × $1 = $0.59 of value. But if coin B turns out to be all-tails, I thought there is a good chance prices for coin B would drop below coin A, so the market is cancelled and you get your money back. So I thought a contract had to be worth more than $0.59.

You can quantify this with a little math. Even if you think the coin B market only has a 50% chance of being cancelled if B is revealed to be all tails, a contract would still be worth $0.7421.

If you buy a contract for coin B for $0.70, then I think that’s worth

P[all-heads] × P[market activates | all-heads] × $1
   + P[all-tails] × P[market cancelled | all-tails] × $0.70
 = 0.59  × $1
   + 0.41 × P[market cancelled | all-tails] × $0.70
 = $0.59
   + $0.287 × P[market cancelled | all-tails]

Surely P[market cancelled | all-tails] isn’t that low. So surely this is worth more than $0.59.

More generally, say you buy a YES contract for coin B for $M. Then that contract would be worth

P[all-heads] × $1 × P[market activates | all-heads]
   + P[all-tails] × $M × P[market cancelled | all-tails]
 = $0.59
   + 0.41 × $M × P[market cancelled | all-tails]

It’s not hard to show that the breakeven price is

M = $0.59 /  (1 - 0.41 × P[market cancelled | all-tails]).

Even if you thought P[market cancelled | all-tails] was only 50%, then the breakeven price would still be $0.7421.

June 27 (later)

Within a few hours, a few people bought contracts on coin B, driving up the price.

Then, Quroe proposed creating derivative markets.

In theory, if there was a market asking if coin A was going to resolve YES, NO, or N/A, supposedly people could arbitrage their bets accordingly and make this market calibrated.

Same for a similar market on coin B.

Thus, Futarchy’s Fundamental Fix - Coin A and Futarchy’s Fundamental Fix - Coin B came to be. These were markets in which people could bid on the probability that each coin would resolve YES, meaning the coin was flipped and landed heads, NO, meaning the coin was flipped and landed tails, or N/A, meaning the market was cancelled.

Honestly, I didn’t understand this. I saw no reason that these derivative markets would make people bid their true beliefs. If they did, then my whole theory that markets reflect correlation rather than causation would be invalidated.

June 28 - July 5

Prices for coin B went up and down, but mostly up.

Eventually, a few people created large limit orders, which caused things to stabilize.

Meanwhile, the derivatives markets did not cause the price of coin B to drop. They basically didn't change anything.

Here was the derivative market for coin A.

And here it was market for coin B.

July 6 - July 24

During this period, not a whole hell of a lot happened.

This brings us up to the moment of truth, when the true nature of coin B was to be revealed. At this point, coin B was at $0.90, even though everyone knows it only has a 59% chance of being heads.

July 25

The nature of the coin was revealed. To show this was fair, Bolton did this by asking a bot to publicly generate a random number.

Thus, coin B was determined to be always-heads.

July 26-27

There were still 24 hours left to bid. At this point, a contract for coin B was guaranteed to pay out $1. The market quickly jumped to $1.

Summary

I was right. Everyone knew coin A had a higher chance of being heads than coin B, but everyone bid the price of coin B way above coin A anyway.

It's a bit sad that B wasn't revealed to be all-tails. If it had, we could have seen if the price for coin B actually crashed to below that for coin A. But given that coin B had a price of $0.90, we know the *market* expected it to crash. In fact, if you do a little math, you can put a number on this and say the market thought there was an 84% chance the market for coin B would indeed crash if revealed to be all-tails.

In the previous math box, we saw that the breakeven price should satisfy

M = $0.59 /  (1 - 0.41 × P[market cancelled | all-tails]).

If you invert this and plug in M=$0.90, then you get

P[market cancelled | all-tails]
 = (1 - $0.59 / M) / 0.41
 = (1 - $0.59 / $0.9) / 0.41
 = 84.01%

I’ll now open the floor for questions.

Questions

Isn’t this market unrealistic?

Yes, but that’s kind of the point. I created the thought experiment because I wanted to make the problem maximally obvious, because it’s subtle and everyone is determined to deny that it exists.

Isn’t this just a weird probability thing? Why does this show futarchy is flawed?

The fact that this is possible is concerning. If this can happen, then futarchy does not work in general. If you want to claim that futarchy works, then you need to spell out exactly what extra assumptions you’re adding to guarantee that this kind of thing won’t happen.

But prices did reflect causality when the market closed! Doesn’t that mean this isn’t a valid test?

No. That’s just a quirk of the implementation. You can easily create situations that would have the same issue all the way through market close. Here’s one way you could do that:

  1. Let coin A be heads with probability 60%. This is public information.
  2. Let coin B be an ALWAYS HEADS coin with probability 59% and ALWAYS TAILS coin with probability 41%. This is a secret.
  3. Every day, generate a random integer between 1 and 30.
    1. If it’s 1, immediately resolve the markets.
    2. It it’s 2, reveal the nature of coin B.
    3. If it’s between 3 and 30, do nothing.

On average, this market will run for 30 days. (The length follows a geometric distribution). Half the time, the market will close without the nature of coin B being revealed. Even when that happens, I claim the price for coin B will still be above coin A.

If futarchy is flawed, shouldn’t you be able to show that without this weird step of “revealing” coin B?

Yes. You should be able to do that, and I think you can. Here’s one way:

  1. Let coin A be heads with probability 60%. This is public information.
  2. Sample 20 random bits, e.g. 10100011001100000001. Let coin B be heads with probability (49+N)% where N is the number of 1 bits. do not reveal these bits publicly.
  3. Secretly send these bits to the first 20 people who ask.
(You could implement this in a forum by using public and private keys.)

First, have users generate public keys by running this command:

openssl genrsa 1024 > private.pem
openssl rsa -in private.pem -pubout > public.pem

Second, they should post the contents of the public_key.pem when asking for their bit. For example:

Hi, can you please send me a bit? Here's my public key:

-----BEGIN PUBLIC KEY-----
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOlesWS+mnvHJOD2osUkbrxE+Y
PMqAUYqwemOwML0LlWLq5RobZRSeyssQhg0i3g2GsMZFMsvjindz6mxccdyP4M8N
mQVCK1Ovs1Z4+DxwmLf/y8vaGC3vfZBOhJDdaNdpRyUiQFaBW99We4cafVnmirRN
Py2lRe+CFgP3kSp4dQIDAQAB
-----END PUBLIC KEY-----

Third, whoever is running the market should save that key as public.pem, pick a pit, and encrypt it like this:

% echo "your secret bit is 1" | openssl pkeyutl -encrypt -pubin -inkey public.pem | base64

OuHt25Jwc1xYq63Ub8gOLKaZEJwwGHWDL0UGfydmvBapQNKf3l6Akol2Z2XHtCAC8G/lPJsCjb1dN878tU0aCMjbO5EvpMUTuohb0OczaCqAMld8uFL+j+uEZsIjKFT3Q52VumdVqMntJYG6Br6QeUs1vAL2HA6Nvych+Ao2e8M=

Users can then decrypt like this:

% echo "OuHt25Jwc1xYq63Ub8gOLKaZEJwwGHWDL0UGfydmvBapQNKf3l6Akol2Z2XHtCAC8G/lPJsCjb1dN878tU0aCMjbO5EvpMUTuohb0OczaCqAMld8uFL+j+uEZsIjKFT3Q52VumdVqMntJYG6Br6QeUs1vAL2HA6Nvych+Ao2e8M=" | base64 -d | openssl pkeyutl -decrypt -inkey private.pem

your secret bit is 1

Or you could use email…

I think this market captures a dynamic that’s present in basically any use of futarchy: You have some information, but you know other information is out there.

I claim that this market—will be weird. Say it just opened. If you didn’t get a bit, then as far as you know, the bias for coin B could be anywhere between 49% and 69%, with a mean of 59%. If you did get a bit, then it turns out that the posterior mean is 58.5% if you got a 0 and 59.5% if you got a 1. So either way, your best guess is very close to 59%.

However, the information for the true bias of coin B is out there! Surely coin B is more likely to end up with a higher price in situations where there are lots of 1 bits. This means you should bid at least a little higher than your true belief, for the same reason as the main experiment—the market activating is correlated with the true bias of coin B.

Of course, after the markets open, people will see each other’s bids and… something will happen. Initially, I think prices will be strongly biased for the above reasons. But as you get closer to market close, there’s less time for information to spread. If you are the last person to trade, and you know you’re the last person to trade, then you should do so based on your true beliefs.

Except, everyone knows that there’s less time for information to spread. So while you are waiting till the last minute to reveal your true beliefs, everyone else will do the same thing. So maybe people sort of rush in at the last second? (It would be easier to think about this if implemented with batched auctions rather than a real-time market.)

Anyway, while the game theory is vexing, I think there’s a mix of (1) people bidding higher than their true beliefs due to correlations between the final price and the true bias of coin B and (2) people “racing” to make the final bid before the markets close. Both of these seem in conflict with the idea of prediction markets making people share information and measuring collective beliefs.

Why do you hate futarchy?

I like futarchy. I think society doesn’t make decisions very well, and I think we should give much more attention to new ideas like futarchy that might help us do better. I just think we should be aware of its imperfections and consider variants (e.g. commiting to randomization) that would resolve them.

If I claim futarchy does reflect causal effects, and I reject this experiment as invalid, should I specify what restrictions I want to place on “valid” experiments (and thus make explicit the assumptions under which I claim futarchy works) since otherwise my claims are unfalsifiable?

Possibly?

Heritability puzzlers

By: dynomight
7 August 2025 at 00:00

The heritability wars have been a-raging. Watching these, I couldn’t help but notice that there’s near-universal confusion about what “heritable” means. Partly, that’s because it’s a subtle concept. But it also seems relevant that almost all explanations of heritability are very, very confusing. For example, here’s Wikipedia’s definition:

Any particular phenotype can be modeled as the sum of genetic and environmental effects:

   Phenotype (P) = Genotype (G) + Environment (E).

Likewise the phenotypic variance in the trait – Var (P) – is the sum of effects as follows:

   Var(P) = Var(G) + Var(E) + 2 Cov(G,E).

In a planned experiment Cov(G,E) can be controlled and held at 0. In this case, heritability, H², is defined as

   H² = Var(G) / Var(P)

H² is the broad-sense heritability.

Do you find that helpful? I hope not, because it’s a mishmash of undefined terminology, unnecessary equations, and borderline-false statements. If you’re in the mood for a mini-polemic:

  1. Phenotype (P) is never defined. This is a minor issue, since it just means “trait”.
  2. Genotype (G) is never defined. This is a huge issue, since it’s very tricky and heritability makes no sense without it.
  3. Environment (E) is never defined. This is worse than it seems, since in heritability, different people use “environment” and E to refer to different things.
  4. When we write P = G + E, are we assuming some kind of linear interaction? The text implies not, but why? What does this equation mean? If this equation is always true, then why do people often add other stuff like G × E on the right?
  5. The text states that if you do a planned experiment (how?) and make Cov(G, E) = 0, then heritability is Var(G) / Var(P). But in fact, heritability is always defined that way. You don’t need a planned experiment and it’s fine if Cov(G, E) ≠ 0.
  6. And—wait a second—that definition doesn’t refer to environmental effects at all. So what was the point of introducing them? What was the point of writing P = G + E? What are we doing?

Reading this almost does more harm than good. While the final definition is correct, it never even attempts to explain what G and P are, it gives an incorrect condition for when the definition applies, and instead mostly devotes itself to an unnecessary digression about environmental effects. The rest of the page doesn’t get much better. Despite being 6700 words long, I think it would be impossible to understand heritability simply by reading it.

Meanwhile, some people argue that heritability is meaningless for human traits like intelligence or income or personality. They claim that those traits are the product of complex interactions between genes and the environment and it’s impossible to disentangle the two. These arguments have always struck me as “suspiciously convenient”. I figured that the people making them couldn’t cope with the hard reality that genes are very important and have an enormous influence on what we are.

But I increasingly feel that the skeptics have a point. While I think it’s a fact that most human traits are substantially heritable, it’s also true the technical definition of heritability is really weird, and simply does not mean what most people think it means.

In this post, I will explain exactly what heritability is, while assuming no background. I will skip everything that can be skipped but—unlike most explanations—I will not skip things that can’t be skipped. Then I’ll go through a series of puzzles demonstrating just how strange heritability is.

What is heritability?

How tall you are depends on your genes, but also on what you eat, what diseases you got as a child, and how much gravity there is on your home planet. And all those things interact. How do you take all that complexity and reduce it to a single number, like “80% heritable”?

The short answer is: Statistical brute force. The long answer is: Read the rest of this post.

It turns out that the hard part of heritability isn’t heritability. Lurking in the background is a slippery concept known as a genotypic value. Discussions of heritability often skim past these. Quite possibly, just looking at the words “genotypic value”, you are thinking about skimming ahead right now. Resist that urge! Genotypic values are the core concept, and without them you cannot possibly understand heritability.

For any trait, your genotypic value is the “typical” outcome if someone with your DNA were raised in many different random environments. In principle, if you wanted to know your genotypic height, you’d need to do this:

  1. Create a million embryonic clones of yourself.
  2. Implant them in the wombs of randomly chosen women around the world who were about to get pregnant on their own.
  3. Convince them to raise those babies exactly like a baby of their own.
  4. Wait 25 years, find all your clones and take their average height.

Since you can’t / shouldn’t do that, you’ll never know your genotypic height. But that’s how it’s defined in principle—the average height someone with your DNA would grow to in a random environment. If you got lots of food and medical care as a child, your actual height is probably above your genotypic height. If you suffered from rickets, your actual height is probably lower than your genotypic height.

Comfortable with genotypic values? OK. Then (broad-sense) heritability is easy. It’s the ratio

   heritability = var[genotype] / var[height].

Here, var is the variance, basically just how much things vary in the population. Among all adults worldwide, var[height] is around 50 cm². (Incidentally, did you know that variance was invented for the purpose of defining heritability?)

Meanwhile, var[genotype] is how much genotypic height varies in the population. That might seem hopeless to estimate, given that we don’t know anyone’s genotypic height. But it turns out that we can still estimate the variance using, e.g., pairs of adopted twins, and it’s thought to be around 40 cm². If we use those numbers, the heritability of height would be

   heritability ≈ (40 cm²) / (50 cm²) ≈ 0.8.

People often convert this to a percentage and say “height is 80% heritable”. I’m not sure I like that, since it masks heritability’s true nature as a ratio. But everyone does it, so I’ll do it too. People who really want to be intimidating might also say, “genes explain 80% of the variance in height”.

Of course, basically the same definition works for any trait, like weight or income or fondness for pseudonymous existential angst science blogs. But instead of replacing “height” with “trait”, biologists have invented the ultra-fancy word “phenotype” and write

   heritability = var[genotype] / var[phenotype].

The word “phenotype” suggests some magical concept that would take years of study to understand. But don’t be intimidated. It just means the actual observed value of some trait(s). You can measure your phenotypic height with a tape measure.

On meaning

Let me make two points before moving on.

First, this definition of heritability assumes nothing. We are not assuming that genes are independent of the environment or that “genotypic effects” combine linearly with “environmental effects”. We are not assuming that genes are in Hardy-Weinberg equilibrium, whatever that is. No. I didn’t talk about that stuff because I don’t need to. There are no hidden assumptions. The above definition always works.

Second, many normal English words have parallel technical meanings, such as “field”, “insulator”, “phase”, “measure”, “tree”, or “stack”. Those are all nice, because they’re evocative and it’s almost always clear from context which meaning is intended. But sometimes, scientists redefine existing words to mean something technical that overlaps but also contradicts the normal meaning, as in “salt”, “glass”, “normal”, “berry”, or “nut”. These all cause confusion, but “heritability” must be the most egregious case in all of science.

Before you ever heard the technical definition of heritability, you surely had some fuzzy concept in your mind. Personally, I thought of heritability as meaning how many “points” you get from genes versus the environment. If charisma was 60% heritable, I pictured each person has having 10 total “charisma points”, 6 of which come from genes, and 4 from the environment:

  Genes       ★★★☆☆☆
  Environment ★☆☆☆
  Total       ★★★★☆☆☆☆☆☆

If you take nothing else from this post, please remember that the technical definition of heritability does not work like that. You might hope that if we add some plausible assumptions, the above ratio-based definition would simplify into something nice and natural, that aligns with what “heritability” means in normal English. But that does not happen. If that’s confusing, well, it’s not my fault.

Intermission

Not sure what’s happening here, but it seems relevant.

Heritability puzzles

So “heritability” is just the ratio of genotypic and phenotypic variance. Is that so bad?

I think… maybe?

How heritable is eye color?

Close to 100%.

This seems obvious, but let’s justify it using our definition that heritability = var[genotype] / var[phenotype].

Well, people have the same eye color, no matter what environment they are raised in. That means that genotypic eye color and phenotypic eye color are the same thing. So they have the same variance, and the ratio is 1. Nothing tricky here.

How heritable is speaking Turkish?

Close to 0%.

Your native language is determined by your environment. If you grow up in a family that speaks Turkish, you speak Turkish. Genes don’t matter.

Of course, there are lots of genes that are correlated with speaking Turkish, since Turks are not, genetically speaking, a random sample of the global population. But that doesn’t matter, because if you put Turkish babies in Korean households, they speak Korean. Genotypic values are defined by what happens in a random environment, which breaks the correlation between speaking Turkish and having Turkish genes.

Since 1.1% of humans speak Turkish, the genotypic value for speaking Turkish is around 0.011 for everyone, no matter their DNA. Since that’s basically constant, the genotypic variance is near zero, and heritability is near zero.

How heritable is speaking English?

Perhaps 30%. Probably somewhere between 10% and 50%. Definitely more than zero.

That’s right. Turkish isn’t heritable but English is. Yes it is. If you ask an LLM, it will tell you that the heritability of English is zero. But the LLM is wrong and I am right.

Why? Let me first acknowledge that Turkish is a little bit heritable. For one thing, some people have genes that make them non-verbal. And there’s surely some genetic basis for being a crazy polyglot that learns many languages for fun. But speaking Turkish as a second language is quite rare, meaning that the genotypic value of speaking Turkish is close to 0.011 for almost everyone.

English is different. While only 1 in 20 people in the world speak English as a first language, 1 in 7 learn it as a second language. And who does that? Educated people.

Most people say educational attainment is around 40% heritable. My guess is that speaking English as a second language is similar. But since there's a minority of native speakers (where genes don't really matter), I'm dropping my estimate to 30%.

Some argue the heritability of educational attainment is much lower. I’d like to avoid debating the exact numbers, but note that these lower numbers are usually estimates of “narrow-sense” heritability rather than “broad-sense” heritability as we’re talking about. So they should be lower. (I’ll explain the difference later.) It’s entirely possible that broad-sense heritability is lower than 40%, but everyone agrees it’s much larger than zero. So the heritability of English is surely much larger than zero, too.

Say there’s an island where genes have no impact on height. How heritable is height among people on this island?

0%.

There’s nothing tricky here.

Say there’s an island where genes entirely determine height. How heritable is height?

100%.

Again, nothing tricky.

Say there’s an island where neither genes nor the environment influence height and everyone is exactly 165 cm tall. How heritable is height?

It’s undefined.

In this case, everyone has exactly the same phenotypic and genotypic height, namely 165 cm. Since those are both constant, their variance is zero and heritability is zero divided by zero. That’s meaningless.

Say there’s an island where some people have genes that predispose them to be taller than others. But the island is ruled by a cruel despot who denies food to children with taller genes, so that on average, everyone is 165 ± 5 cm tall. How heritable is height?

0%.

On this island, everyone has a genotypic height of 165 cm. So genotypic variance is zero, but phenotypic variance is positive, due to the ± 5 cm random variation. So heritability is zero divided by some positive number.

Say there’s an island where some people have genes that predispose them to be tall and some have genes that predispose them to be short. But, the same genes that make you tall also make you semi-starve your children, so in practice everyone is exactly 165 cm tall. How heritable is height?

∞%. Not 100%, mind you, infinitely heritable.

To see why, note that if babies with short/tall genes are adopted by parents with short/tall genes, there are four possible cases.

Baby genes Parent genes Food Height
Short Short Lots 165 cm
Short Tall Semi-starvation Less than 165 cm
Tall Short Lots More than 165 cm
Tall Tall Semi-starvation 165 cm

If a baby with short genes is adopted into random families, they will be shorter on average than if a baby with tall genes. So genotypic height varies. However, in reality, everyone is the same height, so phenotypic height is constant. So genotypic variance is positive while phenotypic variance is zero. Thus, heritability is some positive number divided by zero, i.e. infinity.

(Are you worried that humans are “diploid”, with two genes (alleles) at each locus, one from each biological parent? Or that when there are multiple parents, they all tend to have thoughts on the merits of semi-starvation? If so, please pretend people on this island reproduce asexually. Or, if you like, pretend that there’s strong assortative mating so that everyone either has all-short or all-tall genes and only breeds with similar people. Also, don’t fight the hypothetical.)

Say there are two islands. They all live the same way and have the same gene pool, except people on island A have some gene that makes them grow to be 150 ± 5 cm tall, while on island B they have a gene that makes them grow to be 160 ± 5 cm tall. How heritable is height?

It’s 0% for island A and 0% for island B, and 50% for the two islands together.

Why? Well on island A, everyone has the same genotypic height, namely 150 cm. Since that’s constant, genotypic variance is zero. Meanwhile, phenotypic height varies a bit, so phenotypic variance is positive. Thus, heritability is zero.

For similar reasons, heritability is zero on island B.

But if you put the two islands together, half of people have a genotypic height of 150 cm and half have a genotypic height of 160 cm, so suddenly (via math) genotypic variance is 25 cm². There’s some extra random variation so (via more math) phenotypic variance turns out to be 50 cm². So heritability is 25 / 50 = 50%.

(Math)

If you combine the populations, then genotypic variance is

Var[150 cm + 10 cm × Bernoulli(0.5)]
 = (10 cm)² × Var[Bernoulli(0.5)]
 = (10 cm)² × 0.25
 = 25 cm².

Meanwhile phenotypic variance is

Var[150 cm + 10 cm × Bernoulli(0.5) + 5 cm × Normal(0,1)]
 = (10 cm)² × Var[Bernoulli(0.5)] + (5 cm)² × Var[Normal(0,1)]
 = (10 cm)² × 0.25 + (5 cm)² × 1
 = 50 cm².

Say there’s an island where neither genes nor the environment influence height. Except, some people have a gene that makes them inject their babies with human growth hormone, which makes them 5 cm taller. How heritable is height?

0%.

True, people with that gene will tend be taller. And the gene is causing them to be taller. But if babies are adopted into random families, it’s the genes of the parents that determine if they get injected or not. So everyone has the same genotypic height, genotypic variance is zero, and heritability is zero.

Suppose there’s an island where neither genes nor the environment influence height. Except, some people have a gene that makes them, as babies, talk their parents into injecting them with human growth hormone. The babies are very persuasive. How heritable is height?

We’re back to 100%.

The difference with the previous scenario is that now babies with that gene get injected with human growth hormone no matter who their parents are. Since nothing else influences height, genotype and phenotype are the same, have the same variance, and heritability is 100%.

Suppose there’s an island where neither genes nor the environment influence height. Except, there are crabs that seek out blue-eyed babies and inject them with human growth hormone. The crabs, they are unstoppable. How heritable is height?

Again, 100%.

Babies with DNA for blue eyes get injected. Babies without DNA for blue eyes don’t. Since nothing else influences height, genotype and phenotype are the same and heritability is 100%.

Note that if the crabs were seeking out parents with blue eyes and then injecting their babies, then height would be 0% heritable.

It doesn’t matter that human growth hormone is weird thing that’s coming from outside the baby. It doesn’t matter if we think crabs should be semantically classified as part of “the environment”. It doesn’t matter that heritability would drop to zero if you killed all the crabs, or that the direct causal effect of the relevant genes has nothing to do with height. Heritability is a ratio and doesn’t care.

What good is heritability?

So heritability can be high even when genes have no direct causal effect on the trait in question. It can be low even when there is a strong direct effect. It changes when the environment changes. It even changes based on how you group people together. It can be larger than 100% or even undefined.

Even so, I’m worried people might interpret this post as a long way of saying heritability is dumb and bad, trolololol. So I thought I’d mention that this is not my view.

Say a bunch of companies create different LLMs and train them on different datasets. Some of the resulting LLMs are better at writing fiction than others. Now I ask you, “What percentage of the difference in fiction writing performance is due to the base model code, rather than the datasets or the GPUs or the learning rate schedules?”

That’s a natural question. But if you put it to an AI expert, I bet you’ll get a funny look. You need code and data and GPUs to make an LLM. None of those things can write fiction by themselves. Experts would prefer to think about one change at a time: Given this model, changing the dataset in this way changes fiction writing performance this much.

Similarly, for humans, I think what we really care about is interventions. If we changed this gene, could we eliminate a disease? If we educate children differently, can we make them healthier and happier? No single number can possibly contain all that information.

But heritability is something. I think of it as saying how much hope we have to find an intervention by looking at changes in current genes or current environments.

  1. If heritability is high, then given current typical genes, you can’t influence the trait much through current typical environmental changes. If you only knew that eye color was 100% heritable, that means you won’t change your kid’s eye color by reading to them, or putting them on a vegetarian diet, or moving to higher altitude. But it’s conceivable you could do it by putting electromagnets under their bed or forcing them to communicate in interpretive dance.

  2. If heritability is high, that also means that given current typical environments you can influence the trait through current typical genes. If the world was ruled by an evil despot who forced red-haired people to take pancreatic cancer pills, then pancreatic cancer would be highly heritable. And you could change the odds someone gets pancreatic cancer by swapping in existing genes for black hair.

  3. If heritability is low, that means that given current typical environments, you can’t cause much difference through current typical genetic changes. If we only knew that speaking Turkish was ~0% heritable, that means that doing embryo selection won’t much change the odds that your kid speaks Turkish.

  4. If heritability is low, that also means that given current typical genes, you might be able change the trait through current typical environmental changes. If we only know that speaking Turkish was 0% heritable, then that means there might be something you could do to change the odds your kid speaks Turkish, e.g. moving to Turkey. Or, it’s conceivable that it’s just random and moving to Turkey wouldn’t do anything.

Heritability Influenced by typical genes? Influenced by typical environments?
High Yes No
Low No Maybe

But be careful. Just because heritability is high doesn’t mean that changing genes is easy. And just because heritability is low doesn’t mean that changing the environment is easy.

And heritability doesn’t say anything about non-typical environments or non-typical genes.

If an evil despot is giving all the red-haired people cancer pills, perhaps we could solve that by intervening on the despot. And if you want your kid to speak Turkish, it’s possible that there’s some crazy genetic modifications that would turn them into unstoppable Turkish learning machine.

Heritability has no idea about any of that, because it’s just an observational statistic based on the world as it exists today.

Recommended reading

Appendix

The other heritability

Appendix: Narrow heritability

This post focused on “broad-sense” heritability. But there a second heritability out there, called “narrow-sense”. Like broad-sense heritability, we can define the narrow-sense heritability of height as a ratio:

   narrow heritability = var[additive height] / var[phenotype]

The difference is that rather than having height in the numerator, we now have “additive height”. To define that, imagine doing the following for each of your genes, one at a time:

  1. Find a million random women in the world who just became pregnant.
  2. For each of them, take your gene and insert it into the embryo, replacing whatever was already at that gene’s locus.
  3. Convince everyone to raise those babies exactly like a baby of their own.
  4. Wait 25 years, find all the resulting people, and take the difference of their average height from overall average height.

For example, say overall average human height is 150 cm, but when you insert gene #4023 from yourself into random embryos, their average height is 149.8 cm. Then the additive effect of your gene #4023 is -0.2 cm.

Your “additive height” is average human height plus the sum of additive effects for each of your genes. If the average human height is 150 cm, you have one gene with a -0.2 cm additive effect, another gene with a +0.3 cm additive effect and the rest of your genes have no additive effect, then your “additive height” is 150 cm - 0.2 cm + 0.3 cm = 150.1 cm.

Note: This terminology of “additive height” is non-standard. People usually define narrow-sense heritability using “additive effects”, which are the same thing but without including the mean. This doesn’t change anything since adding a constant doesn’t change the variance. But it’s easier to say “your additive height is 150.1 cm” rather than “the additive effect of your genes on height is +0.1 cm” so I’ll do that.

Honestly, I don’t think the distinction between “broad-sense” and “narrow-sense” heritability is that important. We’ve already seen that broad-sense heritability is weird, and narrow-sense heritability is similar but different. So it won’t surprise you to learn that narrow-sense heritability is differently-weird.

Appendix: Narrow heritability puzzles

But if you really want to understand the difference, I can offer you some more puzzles.

Say there’s an island where people have two genes, each of which is equally likely to be A or B. People are 100 cm tall if they have an AA genotype, 150 cm tall if they have an AB or BA genotype, and 200 cm tall if they have a BB genotype. How heritable is height?

Both broad and narrow-sense heritability are 100%.

The explanation for broad-sense heritability is like many we’ve seen already. Genes entirely determine someone’s height, and so genotypic and phenotypic height are the same.

For narrow-sense heritability, we need to calculate some additive heights. The overall mean is 150 cm, each A gene has an additive effect of -25 cm, and each B gene has an additive effect of +25 cm. But wait! Let’s work out the additive height for all four cases:

genotype phenotypic height additive height
AA 100 cm 150 cm - 25 cm - 25 cm = 100 cm
AB 150 cm 150 cm - 25 cm + 25 cm = 150 cm
BA 150 cm 150 cm + 25 cm - 25 cm = 150 cm
BB 200 cm 150 cm + 25 cm + 25 cm = 200 cm

Since additive height is also the same as phenotypic height, narrow-sense heritability is also 100%.

In this case, the two heritabilities were the same. At a high level, that’s because the genes act independently. When there are “gene-gene” interactions, you tend to get different numbers.

Say there’s an island where people have two genes, each of which is equally likely to be A or B. People with AA or BB genomes are 100 cm, while people with AB or BA genomes are 200 cm. How heritable is height?

Broad-sense heritability is 100%, while narrow-sense heritability is 0%.

You know the story for broad-sense heritability by now. For narrow-sense heritability, we need to do a little math.

  1. The overall mean height is 150 cm.
  2. If you take a random embryo and replace one gene with A, then the there’s a 50% chance the other gene is A, so they’re 100 cm, and there’s a 50% chance the other gene is B, so they’re 200 cm, for an average of 150 cm. Since that’s the same as the overall mean, the additive effect of an A gene is +0 cm.
  3. By similar logic, the additive effect of a B gene is also +0 cm.

So everyone has an additive height of 150 cm, no matter their genes. That’s constant, so narrow-sense heritability is zero.

Appendix: Why are there two heritabilities?

I think basically for two reasons:

First, for some types of data (twin studies) it’s much easier to estimate broad-sense heritability. For other types of data (GWAS) it’s much easier to estimate narrow-sense heritability. So we take what we can get.

Second, they’re useful for different things. Broad-sense heritability is defined by looking at what all your genes do together. That’s nice, since you are the product of all your genes working together. But combinations of genes are not well-preserved by reproduction. If you have a kid, then they breed with someone, their kids breed with other people, and so on. Generations later, any special combination of genes you might have is gone. So if you’re interested in the long-term impact of you having another kid, narrow-sense heritability might be the way to go.

(Sexual reproduction doesn’t really allow for preserving the genetics that make you uniquely “you”. Remember, almost all your genes are shared by lots of other people. If you have any unique genes, that’s almost certainly because they have deleterious de-novo mutations. From the perspective of evolution, your life just amounts to a tiny increase or decrease in the per-locus population frequencies of your individual genes. The participants in the game of evolution are genes. Living creatures like you are part of the playing field. Food for thought.)

Futarchy’s fundamental flaw

By: dynomight
12 June 2025 at 00:00

Say you’re Robyn Denholm, chair of Tesla’s board. And say you’re thinking about firing Elon Musk. One way to make up your mind would be to have people bet on Tesla’s stock price six months from now in a market where all bets get cancelled unless Musk is fired. Also, run a second market where bets are cancelled unless Musk stays CEO. If people bet on higher stock prices in Musk-fired world, maybe you should fire him.

That’s basically Futarchy: Use conditional prediction markets to make decisions.

People often argue about fancy aspects of Futarchy. Are stock prices all you care about? Could Musk use his wealth to bias the market? What if Denholm makes different bets in the two markets, and then fires Musk (or not) to make sure she wins? Are human values and beliefs somehow inseparable?

My objection is more basic: It doesn’t work. You can’t use conditional predictions markets to make decisions like this, because conditional prediction markets reveal probabilistic relationships, not causal relationships. The whole concept is faulty.

There are solutions—ways to force markets to give you causal relationships. But those solutions are painful and I get the shakes when I see everyone acting like you can use prediction markets to conjure causal relationships from thin air, almost for free.

I wrote about this back in 2022, but my argument was kind of sprawling and it seems to have failed to convince approximately everyone. So thought I’d give it another try, with more aggression.

Conditional prediction markets are a thing

In prediction markets, people trade contracts that pay out if some event happens. There might be a market for “Dynomight comes out against aspartame by 2027” contracts that pay out $1 if that happens and $0 if it doesn’t. People often worry about things like market manipulation, liquidity, or herding. Those worries are fair but boring, so let’s ignore them. If a market settles at $0.04, let’s assume that means the “true probability” of the event is 4%.

(I pause here in recognition of those who need to yell about Borel spaces or von Mises axioms or Dutch book theorems or whatever. Get it all out. I value you.)

Right. Conditional prediction markets are the same, except they get cancelled unless some other event happens. For example, the “Dynomight comes out against aspartame by 2027” market might be conditional on “Dynomight de-pseudonymizes”. If you buy a contract for $0.12 then:

  • If Dynomight is still pseudonymous at the end of 2027, you’ll get your $0.12 back.
  • If Dynomight is non-pseudonymous, then you get $1 if Dynomight came out against aspartame and $0 if not.

Let’s again assume that if a conditional prediction market settles at $0.12, that means the “true” conditional probability is 12%.

A non-causal kind of thing

But hold on. If we assume that conditional prediction markets give flawless conditional probabilities, then what’s left to complain about?

Simple. Conditional probabilities are the wrong thing. If P(A|B)=0.9, that means that if you observe B, then there’s a 90% chance of A. That doesn’t mean anything about the chances of A if you do B.

In the context of statistics, everyone knows that correlation does not imply causation. That’s a basic law of science. But really, it’s just another way of saying that conditional probabilities are not what you need to make decisions. And that’s true no matter where the conditional probabilities come from.

For example, people with high vitamin D levels are only ~56% as likely to die in a given year as people with low vitamin D levels. Does that mean taking vitamin D halves your risk of death? No, because those people are also thinner, richer, less likely to be diabetic, less likely to smoke, more likely to exercise, etc. To make sure we’re seeing the effects of vitamin D itself, we run randomized trials. Those suggest it might reduce the risk of death a little. (I take it.)

Futarchy has the same flaw. Even if you think vitamin D does nothing, if there’s a prediction market for if some random person dies, you should pay much less if the market is conditioned on them having high vitamin D. But you should do that mostly because they’re more likely to be rich and thin and healthy, not because of vitamin D itself.

If you like math, conditional prediction markets give you P(A|B). But P(A|B) doesn’t tell you what will happen if you do B. That’s a completely different number with a different notation, namely P(A|do(B)). Generations of people have studied the relationship between P(A|B) and P(A|do(B)). We should pay attention to them.

This is not hypothetical

Say people bet for a lower Tesla stock price when you condition on Musk being fired. Does that mean they think that firing Musk would hurt the stock price? No, because there could be reverse causality—the stock price dropping might cause him to be fired.

You can try to fight this using the fact that things in the future can’t cause things in the past. That is, you can condition on Musk being fired next week and bet on the stock price six months from now. That surely helps, but you still face other problems.

Here’s another example of how lower prices in Musk-fired world may not indicate that firing Musk hurts the stock price. Suppose:

  1. You think Musk is a mildly crappy CEO. If he’s fired, he’ll be replaced with someone slightly better, which would slightly increase Tesla’s stock price.

  2. You’ve heard rumors that Robyn Denholm has recently decided that she hates Musk and wants to dedicate her life to destroying him. Or maybe not, who knows.

If Denholm fired Musk, that would suggest the rumors are true. So she might try to do other things to hurt him, such as trying to destroy Tesla to erase his wealth. So in this situation, Musk being fired leads to lower stock prices even though firing Musk itself would increase the stock price.

Or suppose you run prediction markets for the risk of nuclear war, conditional on Trump sending the US military to enforce a no-fly zone over Ukraine (or not). When betting in these markets, people would surely consider the risk that direct combat between the US and Russian militaries could escalate into nuclear war.

That’s good (the considering), but people would also consider that no one really knows exactly what Trump is thinking. If he declared a no-fly zone, that would suggest that he’s feeling feisty and might do other things that could also lead to nuclear war. The markets wouldn’t reflect the causal impact of a no-fly zone alone, because conditional probabilities are not causal.

Putting markets in charge doesn’t work

So far nothing has worked. But what if we let the markets determine what action is taken? If we pre-commit that Musk will be fired (or not) based on market prices, you might hope that something nice happens and magically we get causal probabilities.

I’m pro-hope, but no such magical nice thing happens.

Thought experiment. Imagine there’s a bent coin that you guess has a 40% chance of landing heads. And suppose I offer to sell you a contract. If you buy it, we’ll flip the coin and you get $1 if it’s heads and $0 otherwise. Assume I’m not doing anything tricky like 3D printing weird-looking coins. If you want, assume I haven’t even seen the coin.

You’d pay something like $0.40 for that contract, right?

(Actually, knowing my readers, I’m pretty sure you’re all gleefully formulating other edge cases. But I’m also sure you see the point that I’m trying to make. If you need to put the $0.40 in escrow and have the coin-flip performed by a Cenobitic monk, that’s fine.)

Now imagine a variant of that thought experiment. It’s the same setup, except if you buy the contract, then I’ll have the coin laser-scanned and ask a supercomputer to simulate millions of coin flips. If more than half of those simulated flips are heads, the bet goes ahead. Otherwise, you get your money back.

Now you should pay at least $0.50 for the contract, even though you only think there’s a 40% chance the coin will land heads.

Why? This is a bit subtle, but you should pay more because you don’t know the true bias of the coin. Your mean estimate is 40%. But it could be 20%, or 60%. After the coin is laser-scanned, the bet only activates if there’s at least a 50% chance of heads. So the contract is worth at least $0.50, and strictly more as long as you think it’s possible the coin has a bias above 50%.

(Math for people who like math.)

Suppose b is the true bias of the coin (which the supercomputer will compute). Then your expected return in this game is

  𝔼[max(b, 0.50)] = 0.50 + 𝔼[max(b-0.50, 0)],

where the expectations reflect your beliefs over the true bias of the coin. Since 𝔼[max(b-0.50, 0)] is never less than zero, the contract is always worth at least $0.50. If you think there’s any chance the bias is above 50%, then the contract is worth strictly more than $0.50.

To connect to prediction markets, let’s do one last thought experiment, replacing the supercomputer with a market. If you buy the contract, then I’ll have lots of other people bid on similar contracts for a while. If the price settles above $0.50, your bet goes ahead. Otherwise, you get your money back.

You should still bid more than $0.40, even though you only think there’s a 40% chance the coin will land heads. Because the market acts like a (worse) laser-scanner plus supercomputer. Assuming prediction markets are good, the market is smarter than you, so it’s more likely to activate if the true bias of the coin is 60% rather than 20%. This changes your incentives, so you won’t bet your true beliefs.

No, order is not preserved

I hope you now agree that conditional prediction markets are non-causal, and choosing actions based on the market doesn’t magically make that problem go away.

But you still might have hope! Maybe the order is still preserved? Maybe you’ll at least always pay more for coins that have a higher probability of coming up heads? Maybe if you run a market with a bunch of coins, the best one will always earn the highest price? Maybe it all works out?

Nope. You can create examples where you'll pay more for a contract on a coin that you think has a lower probability.

Suppose there’s a conditional prediction market for two coins. After a week of bidding, the markets will close, whichever coin had contracts trading for more money will be flipped and $1 paid to contract-holders for head. The other market is cancelled.

Suppose you’re sure that coin A, has a bias of 60%. If you flip it lots of times, 60% of the flips will be heads. But you’re convinced coin B, is a trick coin. You think there’s a 59% chance it always lands heads, and a 41% chance it always lands tails. You’re just not sure which.

We want you to pay more for a contract for coin A, since that’s the coin you think is more likely to be heads (60% vs 59%). But if you like money, you’ll pay more for a contract on coin B. You’ll do that because other people might figure out if it’s an always-heads coin or an always-tails coin. If it’s always heads, great, they’ll bid up the market, it will activate, and you’ll make money. If it’s always tails, they’ll bid down the market, and you’ll get your money back.

You’ll pay more for coin B contracts, even though you think coin A is better in expectation. Order is not preserved. Things do not work out.

No, it’s not easily fixable

Naive conditional prediction markets aren’t causal. Using time doesn’t solve the problem. Having the market choose actions doesn’t solve the problem. But maybe there’s still hope? Maybe it’s possible to solve the problem by screwing around with the payouts?

Theorem. Nope. You can’t solve the problem by screwing around with the payouts. There does not exist a payout function that will make you always bid your true beliefs.

(Click here for a version of that theorem with math. Warning: Math.)

Suppose you run a market where if you pay x and the final market price is y and z happens, then you get a payout of f(x,y,z) dollars. The payout function can be anything, subject only to the constraint that if the final market price is below some constant c, then bets are cancelled, i.e. f(x,y,z)=x for y < c.

Now, take any two distributions ℙ₁ and ℙ₂. Assume that:

  • ℙ₁[Y<c] = ℙ₂[Y<c] > 0
  • ℙ₁[Y≥c] = ℙ₂[Y≥c]
  • 𝔼₁[Z | Y≥c] = 𝔼₂[Z | Y≥c] ℙ₁[(Y,Z) | Y≥c] = ℙ₂[(Y,Z) | Y≥c] (h/t Baram Sosis)
  • 𝔼₁[Z | Y<c] ≠ 𝔼₂[Z | Y<c]

Then the expected return under ℙ₁ and ℙ₂ is the same. That is,

𝔼₁[f(x,Y,Z)]
  = x ℙ₁[Y<c] + ℙ₁[Y≥c] 𝔼₁[f(x,Y,Z) | Y≥c]
  = x ℙ₂[Y<c] + ℙ₂[Y≥c] 𝔼₂[f(x,Y,Z) | Y≥c]
  = 𝔼₂[f(x,Y,Z)].

Thus, you would be willing to pay the same amount for a contract under both distributions.

Meanwhile, the difference in expected values is

𝔼₁[Z] - 𝔼₂[Z]
  = ℙ₁[Y<c] 𝔼₁[Z | Y<c] - ℙ₂[Y<c] 𝔼₂[Z | Y<c]
    + ℙ₁[Y≥c] 𝔼₁[Z | Y≥c] - ℙ₂[Y≥c] 𝔼₂[Z | Y≥c]
  = ℙ₁[Y<c] (𝔼₁[Z | Y<c] - 𝔼₂[Z | Y<c])
  ≠ 0.

The last line uses our assumptions that ℙ₁[Y<c] > 0 and 𝔼₁[Z | Y<c] ≠ 𝔼₂[Z | Y<c].

Thus, we have simultaneously that

𝔼₁[f(x,Y,Z)] = 𝔼₂[f(x,Y,Z)],

yet

𝔼₁[Z] ≠ 𝔼₂[Z].

This means that you should pay the same amount for a contract if you believe ℙ₁ or ℙ₂, even though these entail different beliefs about how likely Z is to happen. Since we haven’t assumed anything about the payout function f(x,y,z), this means that no working payout function can exist. This is bad.

It’s not that bad

Just because conditional prediction markets are non-causal does not mean they are worthless. On the contrary, I think we should do more of them! But they should be treated like observational statistics—just one piece of information to consider skeptically when you make decisions.

Also, while I think these issues are neglected, they’re not completely unrecognized. For example, in 2013, Robin Hanson pointed out that confounding variables can be a problem:

Also, advisory decision market prices can be seriously distorted when decision makers might know things that market speculators do not. In such cases, the fact that a certain decision is made can indicate hidden info held by decision makers. Market estimates of outcomes conditional on a decision then become estimates of outcomes given this hidden info, instead of estimates of the effect of the decision on outcomes.

This post from Anders_H in 2015 is the first I’m aware of that points out the problem in full generality.

Finally, the flaw can be fixed. In statistics, there’s a whole category of techniques to get causal estimates out of data. Many of these methods have analogies as alternative prediction market designs. I’ll talk about those next time. But here’s a preview: None are free.

DumPy: NumPy except it’s OK if you’re dum

By: dynomight
22 May 2025 at 00:00

What I want from an array language is:

  1. Don’t make me think.
  2. Run fast on GPUs.
  3. Really, do not make me think.
  4. Do not.

I say NumPy misses on three of these. So I’d like to propose a “fix” that—I claim—eliminates 90% of unnecessary thinking, with no loss of power. It would also fix all the things based on NumPy, for example every machine learning library.

I know that sounds grandiose. Quite possibly you’re thinking that good-old dynomight has finally lost it. So I warn you now: My solution is utterly non-clever. If anything is clever here, it’s my single-minded rejection of cleverness.

To motivate the fix, let me give my story for how NumPy went wrong. It started as a nice little library for array operations and linear algebra. When everything has two or fewer dimensions, it’s great. But at some point, someone showed up with some higher-dimensional arrays. If loops were fast in Python, NumPy would have said, “Hello person with ≥3 dimensions, please call my ≤2 dimensional functions in a loop so I can stay nice and simple, xox, NumPy.”

But since loops are slow, NumPy instead took all the complexity that would usually be addressed with loops and pushed it down into individual functions. I think this was a disaster, because every time you see some function call like np.func(A,B), you have to think:

  1. OK, what shapes do all those arrays have?
  2. And what does np.func do when it sees those shapes?

Different functions have different rules. Sometimes they’re bewildering. This means constantly thinking and constantly moving dimensions around to appease the whims of particular functions. It’s the functions that should be appeasing your whims!

Even simple-looking things like A*B or A[B,C] do quite different things depending on the starting shapes. And those starting shapes are often themselves the output of previous functions, so the complexity spirals.

Worst of all, if you write a new ≤2 dimensional function, then high-dimensional arrays are your problem. You need to decide what rules to obey, and then you need to re-write your function in a much more complex way to—

Voice from the back: Python sucks! If you used a real language, loops would be fast! This problem is stupid!

That was a strong argument, ten years ago. But now everything is GPU, and GPUs hate loops. Today, array packages are cheerful interfaces that look like Python (or whatever) but are actually embedded languages that secretly compile everything into special GPU instructions that run on whole arrays in parallel. With big arrays, you need GPUs. So I think the speed of the host language doesn’t matter so much anymore.

Python’s slowness may have paradoxically turned out to be an advantage, since it forced everything to be designed to work without loops even before GPUs took over.

Still, thinking is bad, and NumPy makes me think, so I don’t like NumPy.

So what’s the fix?

Here’s my extremely non-clever idea: Let’s just admit that loops were better. In high dimensions, no one has yet come up with a notation that beats loops and indices. So, let’s do this:

  1. Bring back the syntax of loops and indices.
  2. But don’t actually execute the loops. Just take the syntax and secretly compile it into vectorized operations.
  3. Also, let’s get rid of all the insanity that’s been added to NumPy because loops were slow.

That’s basically the whole idea. If you take those three bullet-points, you could probably re-derive everything I do below. I told you this wasn’t clever.

What does it look like?

Suppose that X and Y are 2D arrays, and A is a 4D array. And suppose you want to find a 2D array Z such that Zij = (Yj)T (Aij)-1 Xi. If you could write loops, this would be easy:

import numpy as np
Z = np.empty((X.shape[0], Y.shape[0]))
for i in range(X.shape[0]):
    for j in range(Y.shape[0]):
        Z[i,j] = Y[j] @ np.linalg.solve(A[i,j], X[i])

That’s not pretty. It’s not short or fast. But it is easy!

Meanwhile, how do you do this efficiently in NumPy? Like this:

AiX = np.linalg.solve(A.transpose(1,0,2,3),
                      X[None,...,None])[...,0]
Z = np.sum(AiX * Y[:,None], axis=-1).T

If you’re not a NumPy otaku, that may look like outsider art. Rest assured, it looks like that to me too, and I just wrote it. Why is it so confusing? At a high level, it’s because np.linalg.solve and np.sum and multiplication (*) have complicated rules and weren’t designed to work together to solve this particular problem nicely. That would be impossible, because there are an infinite number of problems. So you need to mash the arrays around a lot to make those functions happy.

Without further ado, here’s how you solve this problem with DumPy (ostensibly Dynomight NumPy):

import dumpy as dp
A = dp.Array(A)
X = dp.Array(X)
Y = dp.Array(Y)
Z = dp.Slot()
Z['i','j'] = Y['j',:] @ dp.linalg.solve(A['i','j',:,:], X['i',:])

Yes! If you prefer, you can also use this equivalent syntax:

Z = dp.Slot()
with dp.Range(X.shape[0]) as i:
    with dp.Range(Y.shape[0]) as j:
        Z[i,j] = Y[j,:] @ dp.linalg.solve(A[i,j,:,:], X[i,:])

Those are both fully vectorized. No loops are executed behind the scenes. They’ll run on a GPU if you have one.

But how?

While it looks magical, the way this actually works is fairly simple:

  1. If you index a DumPy array with a string (or a dp.Range object), it creates a special “mapped” array that pretends to have fewer dimensions.

  2. When a DumPy function is called (e.g. dp.linalg.solve or dp.matmul (called with @)), it checks if any of the arguments have mapped dimensions. If so, it automatically vectorizes the computation, matching up mapped dimensions that share labels.

  3. When you assign an array with mapped dimensions to a dp.Slot, it “unmaps” them into the positions you specify.

No evil meta-programming abstract syntax tree macro bytecode interception is needed. When you run this code:

Z = dp.Slot()
Z['i','j'] = Y['j',:] @ dp.linalg.solve(A['i','j',:,:], X['i',:])

This is what happens behind the scenes:

a = A.map_axes([0, 1], ['i', 'j'])
x = X.map_axes([0], ['i'])
y = Y.map_axes([0], ['j'])
z = y @ dp.linalg.solve(a, x)
Z = z.unmap('i','j')
(Click here for a version with a million asserts and comments.)
# first map A
a = A.map_axes([0, 1], ['i', 'j'])
assert A.ndim == 4
assert a.ndim == 2             # pretends to have fewer dims
assert a.data.shape == A.shape          # secret mapped data
assert a.axes == ('i', 'j', None, None) # secret mapped axes
assert a.shape == (a.data.shape[2], a.data.shape[3])
                # shape determined by non-mapped (None) axes

# now map X
x = X.map_axes([0], ['i'])
assert X.ndim == 2
assert x.ndim == 1
assert x.data.shape == X.shape
assert x.axes == ('i', None)
assert x.shape == (x.data.shape[1], )

# now map Y
y = Y.map_axes([0], ['j'])
assert Y.ndim == 2
assert y.ndim == 1
assert y.shape == (Y.shape[1],)
assert y.axes == ('j', None)
assert y.data.shape == Y.shape
assert y.shape == (y.data.shape[1],)

# Actually do the computation. It happens that the 'j'
# dimension is stored first because its found first (in y).
# But you never need to think about that!
z = y @ dp.linalg.solve(a, x)
assert z.ndim == 0
assert z.shape == ()
assert z.axes == ('j','i')
assert z.data.shape == (Y.shape[0], X.shape[0])

# unmap the mapped axes
Z = z.unmap('i','j')
assert Z.ndim == 2
assert Z.shape == (X.shape[0], Y.shape[0])

Wait, but how?

It might seem like I’ve skipped the hard part. How does dp.linalg.solve know how to vectorize over any combination of input dimensions? Don’t I need to do that for every single function that DumPy includes? Isn’t that hard?

It is hard, but jax.vmap did it already. This takes a function defined using (JAX’s version of) NumPy and vectorizes it over any set of input dimensions. DumPy relies on this to do all the actual vectorization. (If you prefer your vmap janky and broken, I heartily recommend PyTorch’s torch.vmap.)

But hold on. If vmap already exists, then why do we need DumPy? Here’s why:

import jax
from jax import numpy as jnp
Z = jax.vmap(
        jax.vmap(
            lambda x, y, a: y @ jnp.linalg.solve(a, x),
            in_axes=[None, 0, 0]
        ),
        in_axes=[0, None, 0]
    )(X, Y, A)

That’s how you solve the same problem with vmap. (And basically what DumPy does behind the scenes.)

I think vmap is one of the best parts of the NumPy ecosystem. The above code seems genuinely better than the base NumPy version. But it still involves a lot of thinking! Why put in_axes=[None, 0, 0] in the inner vmap and in_axes=[0, None, 0] in the outer one? Why are all the axes 0 even though you need to vectorize over the second dimension of A? There are answers, but they require thinking. Loops and indices are better.

A tiny bit of cleverness

OK, I did do one thing that’s a little clever. Say you want to create a Hilbert matrix with Hij = 1/(1+i+j). In base NumPy you’d have to do this:

X = 1 / (1 + np.arange(5)[:,None] + np.arange(5)[None,:]) # hurr?

In DumPy, you can just write:

X = dp.Slot()
with dp.Range(5) as i:
    with dp.Range(5) as j:
        X[i,j] = 1 / (i + j + 1)

Yes! That works! It works because a dp.Range acts both like a string and like an array mapped along that string. So the above code is roughly equivalent to:

I = dp.Array([0,1,2,3,4])
J = dp.Array([0,1,2,3,4])
X = dp.Slot()
X['i','j'] = 1 / (1 + I['i'] + J['j'])
See? Still no magic.

In reality, the dp.Range choose random strings. (The class maintains a stack of active ranges to prevent collisions.) So in more detail, the above code becomes something like this:

I = dp.Array([0,1,2,3,4])
J = dp.Array([0,1,2,3,4])
i = I.map_axes([0],'range_EZaW')
j = J.map_axes([0],'range_ailw')
x = 1 / (1 + i + j) # vectorized
X = x.unmap('range_EZaW','range_ailw')

OK but is it actually better?

To test if DumPy is actually better in practice, I took six problems of increasing complexity and implemented each of them using loops, NumPy, JAX (with vmap), and DumPy.

Hilbert matrices

Note that in these examples, I always assume the input arrays are in the class of the system being used. If you try running them, you’ll need to add some conversions with np.array / jnp.array / dp.Array.

# loops
H = np.empty((N, N))
for i in range(N):
    for j in range(N):
        H[i, j] = 1 / (i + j + 1)

# NumPy
i = np.arange(N)
j = np.arange(N)
H = 1 / (i[:, None] + j[None, :] + 1)

# JAX
indices = jnp.arange(N)
H = jax.vmap(
        jax.vmap(
            lambda i, j: 1 / (i + j + 1),
            [0, None]),
        [None, 0]
    )(indices, indices)

# DumPy
H = dp.Slot()
with dp.Range(N) as i:
    with dp.Range(N) as j:
        H[i, j] = 1 / (i + j + 1) # Yes! This works!
Batched covariance
# Loops
C = np.zeros((X.shape[0],X.shape[1],X.shape[1]))
for n in range(X.shape[0]):
    C[n] = np.cov(X[n])

# NumPy
mu = np.mean(X, axis=2)[:, :, None]    # hurr?
C = np.sum((X - mu)[:, None, :, :] *
           (X - mu)[:, :, None, :],
           axis=3) / (X.shape[2] - 1)  # hurrr??

# JAX
C = jax.vmap(jnp.cov)(X)

# DumPy
C = dp.Slot()
with dp.Range(X.shape[0]) as n:
    C[n, :, :] = dp.cov(X[n, :, :])

# DumPy (alternate)
C = dp.Slot()
C['n',:,:] = dp.cov(X['n',:,:])
Moving average

(Pretending numpy.lib.stride_tricks doesn’t exist.)

# Loops
B = np.empty(N - window + 1)
for i in range(N - window + 1):
    B[i] = np.mean(A[i:i + window])

# NumPy
i = np.arange(N - window + 1)[:, None]
j = np.arange(window)[None, :] 
B = np.mean(A[i+j], axis=-1)

# JAX
idx = jnp.arange(window)
B = jax.vmap(
        lambda i: jnp.mean(A[i+idx]),
    )(jnp.arange(N - window + 1))

# DumPy
windowed = dp.Slot()
B = dp.Slot()
with dp.Range(N - window + 1) as i:
    with dp.Range(window) as j:
        windowed[i, j] = A[i + j]
    B[i] = dp.mean(windowed[i, :])
    # Note: B[i] = dp.mean(A[i:i+window])
    # would not work because dp.Range can't be used in slice
Indexing

The goal is to create E with

  E[i1, i2, :, i3] = A[B[i1], C[i1, i2], ::2, D[i2, i3]].

# Setup
K = 4
L = 5
M = 6
N = 7

A = np.random.randn(K, L, M, N)
B = np.random.randint(0, K, size=(9,))
C = np.random.randint(0, L, size=(9, 10))
D = np.random.randint(0, N, size=(10, 11))

# Loops
E = np.empty((9, 10, M // 2, 11))
for i1 in range(9):
    for i2 in range(10):
        for i3 in range(11):
            E[i1,i2,:,i3] = A[B[i1],C[i1, i2],::2,D[i2, i3]]

# NumPy
E = A[B[:, None, None],
      C[:, :, None],
      ::2,
      D[None, :, :]
    ].transpose((0, 1, 3, 2))

# JAX
E = jax.vmap(
        jax.vmap(
            jax.vmap(
                lambda b, c, d: A[b, c, ::2, d],
                in_axes=[None,None,0]),
            in_axes=[None,0,0]),
        in_axes=[0,0,None]
    )(B,C,D).transpose(0,1,3,2)

# DumPy
E = dp.Slot()
with dp.Range(9) as i1:
    with dp.Range(10) as i2:
        with dp.Range(11) as i3:
            E[i1,i2,:,i3] = A[B[i1],C[i1, i2],::2,D[i2, i3]]

# DumPy (alternative)
E = dp.Slot()
E['i1','i2',:,'i3'] = A[B['i1'], C['i1','i2'], ::2, D['i2','i3']]

Gaussian Densities

The goal of this problem is, given a list of vectors and a list of Gaussians parameters, and arrays mapping each vector to a list of parameters, evaluate each corresponding vector/parameter combination. Formally, given 2D X, B, C, and means and 3D covs, the goal is to create A with

  Aij = log N( Xi | meansBij, covsCij) .

# Setup
ndims = 3
ndata = 10
neval = 5
ndist = 7

X = np.random.randn(ndata, ndims)
B = np.random.randint(0, ndist, size=(ndata, neval))
C = np.random.randint(0, ndist, size=(ndata, neval))
means = np.random.randn(ndist, ndims)
scales = np.array(np.random.randn(ndist, ndims, ndims))
covs = np.array([scale @ scale.T for scale in scales])
# Loops
def log_prob(x, mean, cov):
    diff = x - mean
    y = np.linalg.solve(cov, diff)
    quad = diff @ y
    logdet = np.linalg.slogdet(2 * np.pi * cov)[1]
    return -0.5 * (quad + logdet)

A = np.empty((ndata, neval))
for i in range(ndata):
    for j in range(neval):
        A[i, j] = log_prob(X[i, :],
                           means[B[i, j], :],
                           covs[C[i, j], :, :])

# NumPy
diff = X[:, None, :] - means[B]
y = np.linalg.solve(covs[C], diff[..., None])
quad = np.sum(diff * y[..., 0], axis=-1)
logdet = np.linalg.slogdet(2 * np.pi * covs[C])[1]
A = -0.5 * (quad + logdet)

# JAX
A = jax.vmap(
        jax.vmap(
            log_prob_gauss,
            in_axes=[None, 0, 0]
        ),
    )(X, means[B], covs[C])

# DumPy
def log_prob(x, mean, cov):
    diff = x - mean
    quad = diff @ dp.linalg.solve(cov, diff)
    logdet = dp.linalg.slogdet(2 * jnp.pi * cov)[1]

A = dp.Slot()
with dp.Range(ndata) as i:
    with dp.Range(neval) as j:
        A[i, j] = log_prob(X[i,:],
                           means[B[i,j],:],
                           covs[C[i,j],:,:])

# DumPy (alternate)
A = dp.Slot()
with dp.Range(ndata) as i:
    with dp.Range(neval) as j:
        mean = means[B[i,j],:]
        cov = covs[C[i,j],:,:]
        diff = X[i,:] - mean
        quad = diff @ dp.linalg.solve(cov, diff)
        logdet = dp.linalg.slogdet(2 * jnp.pi * cov)[1]
        A[i,j] = -0.5 * (quad + logdet)
Multi-head self-attention

See also the discussion in the previous post.

# Setup
input_dim = 4
seq_len = 4
d_k = 5
d_v = input_dim
n_head = 2
X = np.random.randn(seq_len, input_dim)
W_q = np.random.randn(n_head, input_dim, d_k)
W_k = np.random.randn(n_head, input_dim, d_k)
W_v = np.random.randn(n_head, input_dim, d_v)
W_o = np.random.randn(n_head, d_v, input_dim // n_head)

# Loops
def softmax_numpy(x, axis=-1):
    e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return e_x / np.sum(e_x, axis=axis, keepdims=True)

def attention(X, W_q, W_k, W_v):
    Q = X @ W_q
    K = X @ W_k
    V = X @ W_v
    scores = Q @ K.T / np.sqrt(d_k)
    attention_weights = softmax_numpy(scores, axis=-1)
    output = attention_weights @ V
    return output

def multi_head_attention_loops(X, W_q, W_k, W_v, W_o):
    projected = []
    for n in range(n_head):
        my_output = attention(X,
                                W_q[n, :, :],
                                W_k[n, :, :],
                                W_v[n, :, :])
        my_proj = my_output @ W_o[n, :, :]
        projected.append(my_proj)
    projected = np.array(projected)

    final = []
    for i in range(seq_len):
        my_final = np.ravel(projected[:, i, :])
        final.append(my_final)
    return np.array(final)

# NumPy
def softmax_numpy(x, axis=-1): # repeat
    e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return e_x / np.sum(e_x, axis=axis, keepdims=True)

def multi_head_attention_numpy(X, W_q, W_k, W_v, W_o):
    Q = np.einsum('si,hij->hsj', X, W_q)
    K = np.einsum('si,hik->hsk', X, W_k)
    V = np.einsum('si,hiv->hsv', X, W_v)
    scores = Q @ K.transpose(0, 2, 1) / np.sqrt(d_k)
    weights = softmax_numpy(scores, axis=-1)
    output = weights @ V
    projected = np.einsum('hsv,hvd->hsd', output, W_o)
    return projected.transpose(1, 0, 2).reshape(
        seq_len, input_dim)

# JAX
def softmax_jax(x, axis=-1):
    e_x = jnp.exp(x - jnp.max(x, axis=axis, keepdims=True))
    return e_x / jnp.sum(e_x, axis=axis, keepdims=True)

def attention_jax(X, W_q, W_k, W_v):
    d_k = W_k.shape[-1]
    Q = X @ W_q
    K = X @ W_k
    V = X @ W_v
    scores = Q @ K.T / jnp.sqrt(d_k)
    attention_weights = softmax_jax(scores, axis=-1)
    output = attention_weights @ V
    return output

def multi_head_attention_jax(X, W_q, W_k, W_v, W_o):
    def myfun(X, w_q, w_k, w_v, w_o):
        return attention_jax(X, w_q, w_k, w_v) @ w_o

    projected = jax.vmap(myfun,
                            in_axes=[None, 0, 0, 0, 0]
                )(X, W_q, W_k, W_v, W_o)

    return jax.vmap(jnp.ravel, in_axes=1)(projected)

# DumPy
def softmax_dumpy(x):
    assert x.ndim == 1 # no need to think about dimensions!
    e_x = dp.exp(x - dp.max(x))
    return e_x / dp.sum(e_x)

@dp.wrap # needed to make functions with Slots auto-vectorizing
def attention_dumpy(X, W_q, W_k, W_v):
    Q = X @ W_q
    K = X @ W_k
    V = X @ W_v
    scores = Q @ K.T / np.sqrt(d_k)
    attention_weights = dp.Slot()
    with dp.Range(seq_len) as i:
        attention_weights[i, :] = softmax_dumpy(scores[i, :])
    output = attention_weights @ V
    return output

def multi_head_attention_dumpy(X, W_q, W_k, W_v, W_o):
    output = dp.Slot()
    projected = dp.Slot()
    final = dp.Slot()
    with dp.Range(n_head) as n:
        output[n, :, :] = attention_dumpy(X,
                                          W_q[n, :, :],
                                          W_k[n, :, :],
                                          W_v[n, :, :])
        projected[n, :, :] = output[n, :, :] @ W_o[n, :, :]
    with dp.Range(seq_len) as i:
        final[i, :] = dp.ravel(projected[:, i, :])
    return final

# DumPy (alternate)
def multi_head_attention(X, W_q, W_k, W_v, W_o):
    attn_weights = dp.Slot()
    projected = dp.Slot()
    final = dp.Slot()
    with dp.Range(n_head) as n:
        Q = X @ W_q[n, :, :]
        K = X @ W_k[n, :, :]
        V = X @ W_v[n, :, :]
        scores = Q @ K.T / np.sqrt(d_k)
        with dp.Range(seq_len) as i:
            attn_weights[n, i, :] = softmax_dumpy(scores[i, :])
        projected[n, :, :] = attn_weights[n, :, :] @ V @ W_o[n, :, :]
    with dp.Range(seq_len) as i:
        final[i, :] = dp.ravel(projected[:, i, :])
    return final

I gave each implementation a subjective “goodness” score on a 1-10 scale. I always gave the best implementation for each problem 10 points, and then took off points from the others based on how much thinking they required.

Problem Loops Numpy JAX (vmap) DumPy
Hilbert matrices 10 7 7 10
Covariance 9 4 10 9
Moving Ave. 10 6 6 8
Indexing 10 5 4 10
Gaussians 10 3 6 10
Attention 10 1 8 10
Mean 9.8 4.3 6.8 9.5

According to this dubious methodology and these made-up numbers, DumPy is 96.93877% as good as loops! Knowledge is power! But seriously, while subjective, I don’t think my scores should be too controversial. The most debatable one is probably JAX’s attention score.

What to remove?

The only thing DumPy adds to NumPy is some nice notation for indices. That’s it.

What I think makes DumPy good is it also removes a lot of stuff. Roughly speaking, I’ve tried to remove anything that is confusing and exists because NumPy doesn’t have loops. I’m not sure that I’ve drawn the line in exactly the right place, but I do feel confident that I’m on the right track with removing stuff.

1. Goodbye broadcasting

In NumPy, A*B works if A and B are both scalar. Or if A is 5×1×6 and B is 5×1×6×1. But not if A is 1×5×6 and B is 1×5×6×1. Huh?

In truth, the broadcasting rules aren’t that complicated for scalar operations like multiplication. But still, I don’t like it, because every time you see A*B, you have to worry about what shapes those have and what the computation might be doing.

So, I removed it. In DumPy you can only do A*B if one of A or B is scalar or A and B have exactly the same shape. That’s it, anything else raises an error. Instead, use indices, so it’s clear what you’re doing. Instead of this:

C = A[...,None] * B[None]

write this:

C['i','j','k'] = A['i','j'] * B['j','k']

2. Goodbye fancy indexing

Indexing in NumPy is absurdly complicated. When you write A[B,C,D] that could do many different things depending on what all the shapes are. I considered going cold-turkey and only allowing scalar indices in DumPy. That wouldn’t have been so bad, since you can still do advanced stuff using loops. But it’s quite annoying to not be able to write A[B] when A and B are just simple 1D arrays.

So I’ve tentatively decided to be more pragmatic. In DumPy, you can index with integers, or slices, or (possibly mapped) Arrays. But only one Array index can be non-scalar. I settled on this because it’s the most general syntax that doesn’t require thinking.

Let me show you what I mean. If you see this:

A[1, 1:6, C, 2:10] # legal in both numpy and dumpy

It’s “obvious” what the output shape will be. (First the shape of 1:6, then the shape of C, then the shape of 2:10). Simple enough. But as soon as you have two multidimensional array inputs like this:

A[B, 1:6, C, 2:10] # legal in numpy, verboten in dumpy

Suddenly all hell breaks loose. You need to think about broadcasting between B and C, orthogonal vs. pointwise indices, slices behaving differently than arrays, and quirks for where the output dimensions go. So DumPy forbids this. Instead, you need to write one of these:

D['i',:,:]     = A[B['i'],     1:6, C['i'],     2:10] # (1)
D[:,:,'i']     = A[B['i'],     1:6, C['i'],     2:10] # (2)
D['i','j',:,:] = A[B['i'],     1:6, C['j'],     2:10] # (3)
D['i','j',:,:] = A[B['i','j'], 1:6, C['i'],     2:10] # (4)
D['i','j',:,:] = A[B['i','j'], 1:6, C['i','j'], 2:10] # (5)

They all do exactly what they look like they do.

Oh, and one more thing! In DumPy, you must index all dimensions. In NumPy, if A has three dimensions, then A[2] is equivalent to A[2,:,:]. This is sometimes nice, but it means that every time you see A[2], you have to worry about how many dimensions A has. In DumPy, every time you index an array or assign to a dp.Slot, it checks that all indices have been included. So when you see option (4) above, you know that:

  • A has 4 dimensions
  • B has 2 dimensions
  • C has 1 dimension
  • D has 4 dimensions

Always, always, always. No cases, no thinking.

3. Goodbye complicated functions

Again, many NumPy functions have complex conventions for vectorization. np.linalg.solve sort of says, “If the inputs have ≤2 dimensions, do the obvious thing. Otherwise, do some extremely confusing broadcasting stuff.” DumPy removes the confusing broadcasting stuff. When you see dp.linalg.solve(A,B), you know that A and B have no more than two dimensions, so nothing tricky is happening.

Similarly, in NumPy, A @ B is equivalent to np.matmul(A,B). When both inputs have ≤2 or fewer dimensions, this does the “obvious thing”. (Either an inner-product or some kind of matrix/vector multiplication.) Otherwise, it broadcasts or vectorizes or something? I can never remember. In DumPy you don’t have that problem, because it restricts A @ B to arrays with one or two dimensions only.

If you need more dimensions, no problem: Use indices.

Why remove?

It might seem annoying to remove features, but I’m telling you: Just try it. If you program this way, a wonderful feeling of calmness comes over you, as class after class of possible errors disappear.

Put another way, why remove all the fancy stuff, instead of leaving it optional? Because optional implies thinking! I want to program in a simple way. I don’t want to worry that I’m accidentally triggering some confusing broadcasting insanity, because that would be a mistake. I want the computer to help me catch mistakes, not silently do something weird that I didn’t intend.

In principle, it would be OK if there was a evil_solve method that preserves all the confusing batching stuff. If you really want that, you can make it yourself:

evil_solve = dp.MappedFunction(jnp.linalg.solve) # not recommended

You can use that same wrapper to convert any JAX NumPy function to work with DumPy.

Discussion

Think about math: In two or fewer dimensions, coordinate-free linear algebra notation is wonderful. But for higher dimensional tensors, there are just too many cases, so most physicists just use coordinates.

So this solution seems pretty obvious to me. Honestly, I’m a little confused why it isn’t already standard. Am I missing something?

What about APL?

When I complain about NumPy, many people often suggest looking into APL-type languages, like A, J, K, or Q. (All single-letter languages are APL-like, except C, D, F, R, T, X, and many others. Convenient, right?) The obvious disadvantages of these are that:

  1. They’re unfamiliar.
  2. The code looks like gibberish.
  3. They don’t usually provide autodiff or GPU execution.

None of those bother me. If the languages are better, we should learn to use them and make them do autodiff on GPUs. But I’m not convinced they are better. When you actually learn these languages, what you figure out is that the symbol gibberish basically amounts to doing the same kind of dimension mashing that we saw earlier in NumPy:

AiX = np.linalg.solve(A.transpose(1,0,2,3),
                      X[None,...,None])[...,0]
Z = np.sum(AiX * Y[:,None], axis=-1).T

The reason is that, just like NumPy and vmap, these languages choose align dimensions by position, rather than by name. If I have to mash dimensions, I want to use the best tool. But I’d prefer not to mash dimensions at all.

What about named dimensions?

People also often suggest “NumPy with named dimensions” as in xarray. (PyTorch also has a half-hearted implementation.) Of course, DumPy also uses named dimensions, but there’s a critical difference. In xarray, they’re part of the arrays themselves, while in DumPy, they live outside the arrays.

In some cases, permanent named dimensions are very nice. But for linear algebra, they’re confusing. For example, suppose A is 2-D with named dimensions "cat" and "dog". Now, what dimensions should ATA have? ("dog" twice?) Or say you take a singular value decomposition like U, S, Vh = svd(A). What name should the inner dimensions have? Does the user have to specify it?

I haven’t seen a nice solution. xarray doesn’t focus on linear algebra, so it’s not much of an issue there. A theoretical “DumPy with permanent names” might be very nice, but I’m not sure how it should work. This is worth thinking about more.

What about Julia or [other language]

I like Julia! Loops are fast in Julia! But again, I don’t think fast loops matter that much, because I want to move all the loops to the GPU. So even if I was using Julia, I think I’d want to use a DumPy-type solution.

I think Julia might well be a better host language than Python, but it wouldn’t be because of fast loops, but because it offers much more powerful meta-programming capabilities. I built DumPy on top of JAX just because JAX is very mature and good at calling the GPU, but I’d love to see the same idea used in Julia (“Dulia”?) or other languages.

Prototype

OK, I promised a link to my prototype, so here it is: dumpy.py

It’s just a single file with around 700 lines. I’m leaving it as a single file because I want to stress that this is just something I hacked together in the service of this rant. I wanted to show that I’m not totally out of my mind, and that doing all this is actually pretty easy.

I stress that I don’t really intend to update or improve this. (Unless someone gives me a lot of money?) So please do not attempt to use it for “real work”, and do not make fun of my code.

PS. DumPy works out of the box with both jax.jit and jax.grad. For gradients, you need to either cast the output to a JAX scalar or use the dp.grad wrapper.

PPS. If you like this, you may also like einx or torchdim.


Update: Due to many requests, I have turned this into a “real” package, available on PyPi as dumpy-numpy. You can install it by typing:

pip install dumpy-numpy

Or, if you use uv (you should) you can play around with DumPy by just typing this one-liner in your terminal:

uv run --with dumpy-numpy --with ipython ipython

For example:

UV quickstart

I don’t like NumPy

By: dynomight
15 May 2025 at 00:00

They say you can’t truly hate someone unless you loved them first. I don’t know if that’s true as a general principle, but it certainly describes my relationship with NumPy.

NumPy, by the way, is some software that does computations on arrays in Python. It’s insanely popular and has had a huge influence on all the popular machine learning libraries like PyTorch. These libraries share most of the same issues I discuss below, but I’ll stick to NumPy for concreteness.

NumPy makes easy things easy. Say A is a 5×5 matrix, x is a length-5 vector, and you want to find the vector y such that Ay=x. In NumPy, that would be:

y = np.linalg.solve(A, x)

So elegant! So clear!

But say the situation is even a little more complicated. Say A is a stack of 100 5×5 matrices, given as a 100×5×5 array. And say x is a stack of 100 length-5 vectors, given as a 100×5 array. And say you want to solve Aᵢyᵢ=xᵢ for 1≤i≤100.

If you could use loops, this would be easy:

y = np.empty_like(x)
for i in range(100):
    y[i,:] = np.linalg.solve(A[i,:,:], x[i,:])

But you can’t use loops. To some degree, this is a limitation of loops being slow in Python. But nowadays, everything is GPU and if you’ve got big arrays, you probably don’t want to use loops in any language. To get all those transistors firing, you need to call special GPU functions that will sort of split up the arrays into lots of little pieces and process them in parallel.

The good news is that NumPy knows about those special routines (at least if you use JAX or CuPy), and if you call np.linalg.solve correctly, it will use them.

The bad news is that no one knows how do that.

Don’t believe me? OK, which of these is right?

y = linalg.solve(A,x)
y = linalg.solve(A,x,axis=0)
y = linalg.solve(A,x,axes=[[1,2],1])
y = linalg.solve(A.T, x.T)
y = linalg.solve(A.T, x).T
y = linalg.solve(A, x[None,:,:])
y = linalg.solve(A,x[:,:,None])
y = linalg.solve(A,x[:,:,None])[:,:,0]
y = linalg.solve(A[:,:,:,None],x[:,None,None,:])
y = linalg.solve(A.transpose([1,2,0]),x[:,:,None]).T

No one knows. And let me show you something else. Here’s the documentation:

np.linalg.solve

Read that. Meditate on it. Now, notice: You still don’t know how to solve Aᵢyᵢ=xᵢ for all i at once. Is it even possible? Did I lie when I said it was?

As far as I can tell, what people actually do is try random variations until one seems to work.

Why NumPy bad

NumPy is all about applying operations to arrays. When the arrays have 2 or fewer dimensions, everything is fine. But if you’re doing something even mildly complicated, you inevitably find yourself with some operation you want to apply to some dimensions of array A, some other dimensions of array B, and some other dimensions of array C. And NumPy has no theory for how to express that.

Let me show you what I mean. Suppose:

  • A is a K×L×M array
  • B is a L×N array
  • C is a K×M array

And say that for each k and n, you’d like to compute the mean over the L and M dimensions. That is, you want

   Dkn = 1/(LM) × ∑lm Aklm Bln Ckm.

To do that, you’ve got two options. The first is to use grotesque dimension alignment tricks:

D = np.mean(
        np.mean(
            A[:,:,:,None] *
            B[None,:,None,:] *
            C[:,None,:,None],
        axis=1),
    axis=1)

The hell, you ask? Why is None everywhere? Well, when indexing an array in NumPy, you can write None to insert a new dimension. A is K×L×M, but A[:,:,:,None] is K×L×M×1. Similarly, B[None,:,None,:] is 1×L×1×N and C[:,None,:,None] is K×1×M×1. When you multiply these together, NumPy “broadcasts” all the size-1 dimensions to give a K×L×M×N array. Then, the np.mean calls average over the L and M dimensions.

I think this is bad. I’ve been using NumPy for years and I still find it impossible to write code like that without always making mistakes.

It’s also borderline-impossible to read. To prove this, I just flipped a coin and introduced a bug above if and only if the coin was tails. Is there a bug? Are you sure? No one knows.

Your second option is to desperately try to be clever. Life is short and precious, but if you spend a lot of yours reading the NumPy documentation, you might eventually realize that there’s a function called np.tensordot, and that it’s possible to make it do much of the work:

D = (1/L) * np.mean(
                np.tensordot(A, B, axes=[1,0]) *
                C[:,:,None],
            axis=1)

That’s correct. (I promise.) But why does it work? What exactly is np.tensordot doing? If you saw that code in some other context, would you have the slightest idea what was happening?

Here’s how I’d do it, if only I could use loops:

D = np.zeros((K,N))  
for k in range(K):  
    for n in range(N):  
        a = A[k,:,:]  
        b = B[:,n]  
        c = C[k,:]  
        assert a.shape == (L,M)  
        assert b.shape == (L,)  
        assert c.shape == (M,)  
        D[k,n] = np.mean(a * b[:,None] * c[None,:])

People who’ve written too much NumPy may find that clunky. I suspect that’s a wee bit of Stockholm Syndrome. But surely we can agree that it’s clear.

In practice, things are often even worse. Say that A had shape M×K×L rather than K×L×M. With loops, no big deal. But NumPy requires you to write monstrosities like A.transpose([1,2,0]). Or should that be A.transpose([2,0,1])? What shapes do those produce? No one knows.

Loops were better.

OK I lied

There is a third option:

D = 1/(L*M) * np.einsum('klm,ln,km->kn', A, B, C)

If you’ve never seen Einstein summation before, that might look terrifying. But remember, our goal is to find

   Dkn = 1/(LM) × ∑lm Aklm Bln Ckm.

The string in the above code basically gives labels to the indices in each of the three inputs (klm,ln,km) and the target indices for the output (->kn). Then, np.einsum multiplies together the corresponding elements of the inputs and sums over all indices that aren’t in the output.

Personally, I think np.einsum is one of the rare parts of NumPy that’s actually good. The strings are a bit tedious, but they’re worth it, because the overall function is easy(ish) to understand, is completely explicit, and is quite general and powerful.

Except, how does np.einsum achieve all this? It uses indices. Or, more precisely, it introduces a tiny domain-specific language based on indices. It doesn’t suffer from NumPy’s design flaws because it refuses to play by NumPy’s normal rules.

But np.einsum only does a few things. (Einops does a few more.) What if you want to apply some other function over various dimensions of some arrays? There is no np.linalg.einsolve. And if you create your own function, there’s certainly no “Einstein” version of it.

I think np.einsum’s goodness shows that NumPy went somewhere.

Intermission

Here’s a painting which feels analogous to our subject.

Where did NumPy go wrong?

Here’s what I want from an array language. I ain’t particular about syntax, but it would be nice if:

  1. When you want to do something, it’s “obvious” how to do it.
  2. When you read some code, it’s “obvious” what it does.

Wouldn’t that be nice? I think NumPy doesn’t achieve these because of its original sin: It took away indices and replaced them with broadcasting. And broadcasting cannot fill indices’ shoes.

I don’t love NumPy broadcasting

NumPy’s core trick is broadcasting. Take this code:

A = np.array([[1,2],[3,4],[5,6]])
B = np.array([10,20])
C = A * B
print(C)

This outputs:

[[ 10  40]
 [ 30  80]
 [ 50 120]]

Here, A is a 3×2 array, and B is a length-2 array. When you multiply them together, B is “broadcast” to the shape of A, meaning the first column of A is multiplied with B[0]=10 and the second is multiplied with B[1]=20.

In simple cases, this seems good. But I don’t love it. One reason is that, as we saw above, you often have to do gross things to the dimensions to get them to line up.

Another reason is that it isn’t explicit or legible. Sometimes A*B multiplies element-by-element, and sometimes it does more complicated things. So every time you see A*B, you have to figure out which case in the broadcasting conventions is getting triggered.

But the real problem with broadcasting is how it infects everything else. I’ll explain below.

I don’t like NumPy indexing

Here’s a riddle. Take this code:

A = np.ones((10,20,30,40))
i = np.array([1,2,3])
j = np.array([[0],[1]])
B = A[:,i,j,:]

What shape does B have?

It turns out the answer is 10×2×3×40. That’s because the i and j indices get broadcast to a shape of 2×3 and then something something mumble mumble mumble. Try to convince yourself it makes sense.

Done? OK, now try these:

C = A[:,:,i,j]
D = A[:,i,:,j]
E = A[:,1:4,j,:]

What shapes do these have?

  • C is 10×20×2×3. This seems logical, given what happened with B above.

  • What about D? It is 2×3×10×30. Now, for some reason, the 2 and 3 go at the beginning?

  • And what about E? Well, “slices” in Python exclude the endpoint, so 1:4 is equivalent to [1,2,3] which is equivalent to i, and so E is the same as B. Hahaha, just kidding! E is 10×3×2×1×40.

Yes, that is what happens. Try it if you don’t believe me! I understand why NumPy does this, because I’ve absorbed this 5000 word document that explains how NumPy indexing works. But I want that time back.

For fun, I tried asking a bunch of AI models to figure out what shapes those arrays have. Here were the results:

I used this query:

Take this python code

A = np.ones((10,20,30,40))
i = np.array([1,2,3])
j = np.array([[0],[1]])
B = A[:,i,j,:]
C = A[:,:,i,j]
D = A[:,i,:,j]
E = A[:,1:4,j,:]

what shapes do B, C, D, and E have?

Claude 3.7 used “extended thinking”. Here are all the incorrect outputs:

AI B C D E
GPT 4.1     10×2×3×30  
Grok 3     10×3×30×2 10×3×2×40
Claude 3 Opus 10×3×2×30 10×20×3×2 10×3×30×2 10×3×2×40
Llama 4 Maverick     10×3×30×2 10×3×2×40
o3     10×2×3×30  
Claude 3.7     10×3×30×2 10×3×2×40
AI B C D E
GPT 4.1 ✔️ ✔️ X ✔️
Grok 3 ✔️ ✔️ X X
Claude 3 Opus X X X X
Llama 4 Maverick ✔️ ✔️ X X
o3 ✔️ ✔️ X ✔️
Claude 3.7 ✔️ ✔️ X X
Gemini 2.5 Pro ✔️ ✔️ ✔️ ✔️
DeepSeek R1 ✔️ ✔️ ✔️ ✔️

(DeepSeek’s chain of thought used “wait” 76 times. It got everything right the first time, but when I tried it again, it somehow got B, C, and D all wrong, but E right.)

This is insane. Using basic features should not require solving crazy logic puzzles.

You might think, “OK, I’ll just limit myself to indexing in simple ways.” Sounds good, except sometimes you need advanced indexing. And even if you’re doing something simple, you still need to be careful to avoid the crazy cases.

This again makes everything non-legible. Even if you’re just reading code that uses indexing in a simple way, how do you know it’s simple? If you see A[B,C], that could be doing almost anything. To understand it, you need to remember the shapes of A, B, and C and work through all the cases. And, of course, A, B, and C are often produced by other code, which you also need to think about…

I don’t like NumPy functions

Why did NumPy end up with a np.linalg.solve(A,B) function that’s so confusing? I imagine they first made it work when A is a 2D array and and b is a 1D or 2D array, just like the mathematical notation of A⁻¹b or A⁻¹B.

So far so good. But then someone probably came along with a 3D array. If you could use loops, the solution would be “use the old function with loops”. But you can’t use loops. So there were basically three options:

  1. They could add some extra axes argument, so the user can specify which dimensions to operate over. Maybe you could write solve(A,B,axes=[[1,2],1]).
  2. They could create different functions with different names for different situations. Maybe solve_matrix_vector would do one thing, solve_tensor_matrix would do another.
  3. They could add a Convention: Some arbitrary choice for how solve will internally try to line up the dimensions. Then it’s the user’s problem to figure out and conform to those Conventions.

All these options are bad, because none of them can really cope with the fact that there are a combinatorial number of different cases. NumPy chose: All of them. Some functions have axes arguments. Some have different versions with different names. Some have Conventions. Some have Conventions and axes arguments. And some don’t provide any vectorized version at all.

But the biggest flaw of NumPy is this: Say you create a function that solves some problem with arrays of some given shape. Now, how do you apply it to particular dimensions of some larger arrays? The answer is: You re-write your function from scratch in a much more complex way. The basic principle of programming is abstraction—solving simple problems and then using the solutions as building blocks for more complex problems. NumPy doesn’t let you do that.

Attention please

One last example to show you what I’m talking about. Whenever I whine about NumPy, people always want to see an example with self-attention, the core trick behind modern language models. So fine. Here’s an implementation, which I humbly suggest is better than all 227 versions I found when I searched for “self-attention numpy”:

# self attention by your friend dynomight

input_dim = 4  
seq_len = 4  
d_k = 5  
d_v = input_dim  
  
X = np.random.randn(seq_len, input_dim)  
W_q = np.random.randn(input_dim, d_k)  
W_k = np.random.randn(input_dim, d_k)  
W_v = np.random.randn(input_dim, d_v)  
  
def softmax(x, axis):  
    e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))  
    return e_x / np.sum(e_x, axis=axis, keepdims=True)  
  
def attention(X, W_q, W_k, W_v):  
    d_k = W_k.shape[1]  
    Q = X @ W_q  
    K = X @ W_k  
    V = X @ W_v  
    scores = Q @ K.T / np.sqrt(d_k)  
    attention_weights = softmax(scores, axis=-1)  
    return attention_weights @ V  

This is fine. Some of the axis stuff is a little obscure, but whatever.

But what language models really need is multi-head attention, where you sort of do attention several times in parallel and then merge the results. How do we do that?

First, let’s imagine we lived in a sane world where we were allowed to use abstractions. Then you could just call the previous function in a loop:

# multi-head self attention by your friend dynomight
# if only we could use loops

n_head = 2
  
X = np.random.randn(seq_len, input_dim)  
W_q = np.random.randn(n_head, input_dim, d_k)  
W_k = np.random.randn(n_head, input_dim, d_k)  
W_v = np.random.randn(n_head, input_dim, d_v)  
W_o = np.random.randn(n_head, d_v, input_dim // n_head)

def multi_head_attention(X, W_q, W_k, W_v, W_o):  
    projected = []  
    for n in range(n_head):  
        output = attention(X,
                           W_q[n,:,:],
                           W_k[n,:,:],
                           W_v[n,:,:])  
        my_proj = output @ W_o[n,:,:]  
        projected.append(my_proj)  
    projected = np.array(projected)  
  
    output = []  
    for i in range(seq_len):  
        my_output = np.ravel(projected[:,i,:])  
        output.append(my_output)  
    return np.array(output)  

Looks stupid, right? Yes—thank you! Cleverness is bad.

But we don’t live in a sane world. So instead you need to do this:

# multi-head self attention by your friend dynomight
# all vectorized and bewildering

def multi_head_attention(X, W_q, W_k, W_v, W_o):  
    d_k = W_k.shape[-1]  
    Q = np.einsum('si,hij->hsj', X, W_q)  
    K = np.einsum('si,hik->hsk', X, W_k)  
    V = np.einsum('si,hiv->hsv', X, W_v)  
    scores = Q @ K.transpose(0, 2, 1) / np.sqrt(d_k)  
    weights = softmax(scores, axis=-1)
    output = weights @ V  
    projected = np.einsum('hsv,hvd->hsd', output, W_o)  
    return projected.transpose(1, 0, 2).reshape(
        seq_len, input_dim)  

Ha! Hahahahahaha!

So what then?

To be clear, I’m only suggesting that NumPy is “the worst array language other than all the other array languages”. What’s the point of complaining if I don’t have something better to suggest?

Well, actually I do have something better to suggest. I’ve made a prototype of a “better” NumPy that I think retains all the power while eliminating all the sharp edges. I thought this would just be a short motivational introduction, but after I started writing, the evil took hold of me and here we are 3000 words later.

Also, it’s probably wise to keep some distance between one’s raving polemics and one’s constructive array language API proposals. So I’ll cover my new thing next time.

Mailbag #2

By: Tim Urban
14 April 2021 at 12:41

Last month, I emailed readers announcing an upcoming mailbag post—the WBW post version of an AMA. 1,500 questions poured in—remarkably interesting, creative questions on a wide range of topics. I picked some for this round, and we’re keeping the rest in a database that I’ll go back to for future mailbags (mailbag@waitbutwhy.com is always open, so send questions anytime and we’ll add them to the list).

There’s a lot to cover, so let’s get going.

Question from my 7-year-old: How many germs would you have to put together to actually be able to see them? What would it look like? – Kirsten, Zachary’s mum (Sydney, Australia)

You’re my kind of guy, Zachary. Let’s discuss.

There are a lot of germs out there. A smaller germ, like the virus that causes covid, is 120nm across. The smallest object we can see is about 0.1mm—about the width of a human hair. You’d have to line up about 800 coronaviruses to get to 0.1mm—but that would be a one-dimensional line way too thin to see. To actually see something, you’d have to give it some area and turn the 800-virus line into an 800 x 800 square. That clump of 640,000 viruses would be just big enough to see as a tiny speck.1 As for what it would look like, your guess is as good as mine Zachary.

I’m never good at stopping once I start on an exploration of size, and doing this answer got me googling all kinds of things.

Like how big all the bacteria in the human body would be if you clumped it all together. I had always heard the famous stat that there are 10 times as many bacteria in your body as human cells. It turns out that that’s been debunked. The real ratio is closer to 1:1, with both kinds of cells in the ballpark of 40 trillion in an adult human body.2

Anyway, the NIH estimates that all that bacteria adds up to only 0.3% of a human’s body mass. Zachary is 7, so let’s estimate his mass at 25 kg. That means the bacteria in his body adds up to 75 grams—about the weight of a plum. Assuming the density of Zachary’s bacteria is similar to the density of his body, that would make the bacteria ball about the size of a plum too.

This led me to the obvious next question: how big would all the bacteria on Earth be if you bunched them all together?3

Thankfully, University of Georgia microbiologist William Whitman has done the hard work of coming up with an estimate for the number of bacteria on Earth: 5 nonillion.

That’s 5,000,000,000,000,000,000,000,000,000,000.

To couple that estimate with a wild estimate of our own, we can use the volume of an E. coli bacteria (.7µm3) and assume that’s somewhere around the average size of Earth bacteria. 5 nonillion x .7µm3 comes out to a cube with a base of about 15km (~9.5mi).

This massive cube of bacteria covers a large part of Los Angeles and rises higher than the cruising altitude of commercial airplanes. If you were on the ground, it would take a full day to walk around the cube, and it would be a bad experience because the whole day you’d be right next to a vile bacteria cube.

If we took a huge butter knife and smeared the cube out evenly on the Earth’s surface, it would cover the entire Earth with a layer of pure bacteria 7mm (1/4 inch) thick, and everything would be gross.

K let’s move on.

What do you think of scientists putting mini human brains in mice? Where do you think this kind of science could lead in the future? – Leah N. (Quincy, IL)

I have no idea what’s going on with this question but it’s now all I care about. I’m picturing a world with tiny field mice with human intelligence. They’d form tiny societies run by tiny iron-fisted mouse tyrants. They could have little academies and do research on mice-related things. They might organize a mouse Olympics that we could televise and bet on. Some people would befriend mice and some might even choose to get married to mice. I hope this is what you’re talking about Leah, and I refuse to look it up because I’m sure whatever you’re actually talking about will be a huge letdown.

Drugs: do you do them? What do you think of them? – Tom C. (Nottingham, UK)

Kind of what I think of cars. Cars can be incredibly useful, fun, and life-changing if you understand how cars work and know how to drive safely. If you don’t know anything about driving or you tend to drive recklessly, you’re probably not ready to use cars.

Why do we prefer to watch a film we haven’t watched before but we want to listen to songs that we have heard hundreds of times? – Anastasia S. (Athens, Greece)

Many films are the most fun the first time because a part of our brains loses itself in the plot and actually experiences what the characters are experiencing, to an extent. Part of why that’s so fun is the uncertainty, which makes it like a real-life adventure. Once we know what’s going to happen, the experience is less exciting for the same reason an adventure would be less exciting if you already knew how everything was gonna play out.

A song works the opposite way. Not knowing where the song is going can be intellectually interesting, but what our brain really wants to do is mentally dance with the song, and it can’t do that if it doesn’t know the “steps.” That’s why a catchy song is only okay the first time but it can quickly become orgasmic after we hear it a few times and our brains get the hang of it. Once our brain memorizes a song, the song has a special pathway carved in our heads and it feels great to light that pathway up by hearing the song (and if we don’t give our brain the chance, it might just start lighting up the pathway itself, and the song gets “stuck in your head”).4

Back to films, the thing is, a film isn’t just a plot. It can also be a piece of art, which is why in some cases, we do like to watch films (or TV shows) repeatedly (I’ve seen The Shawshank Redemption, Back to the Future, Shutter Island, and The Office (UK) like 9 times each).

What age range do you currently think you’re most likely to die (disregarding cryonics)?

Multiple choice:

  1. 70–100
  2. 100–125
  3. 125–150
  4. 150–250
  5. 250+

– Sean M. (New Orleans, LA)

The pessimistic part of my brain says: A) 70–100. My grandparents lived to 84, 87, 91, and good old Nana is still going strong at 95.5 My lifestyle is not a big enough step above theirs, healthwise, to imagine that I’ll beat them out by very much. And sure, medicine is improving dramatically, but then I look at charts like this one (click for bigger view):1

As promising as that data is, we don’t seem to be getting too far with the 100+ thing. Even the most freak outliers don’t crack 120. The human body seems programmed to shut itself down somewhere around the century mark, if it hasn’t already. Clearly something other than the current type of healthcare advances will be needed to crack through the natural human lifespan ceiling—it’ll take some deeper recoding of the human genome.

But then the optimistic part of my brain imagines what it would be like to show George Washington around 2021 and watch him die of shock at how magical everything was, and it thinks about how the accelerating returns on progress mean that the world of 70- to 100-year-old me might be equally shocking to people today, and so it answers D) 150–250.

The pessimistic part of my brain, looking at reality, makes a sad face and pats the optimistic part of my brain on the head.

The optimistic part of my brain, remembering how bad humans are at intuitively understanding exponential growth, pats the pessimistic part of my brain on the head.

How do you fall asleep? – Anastasia S. (Athens, Greece)

When I can’t fall asleep, it’s either because my mind is caught in an anxious loop of some kind or my phone is keeping me up. The surefire fix to the anxious loop issue is to distract my mind with something else.

But what that something else is is important. If I start texting or scrolling around social media or brainstorming / researching for work or playing an addictive iPhone game, that’ll just make me even less likely to sleep. Other activities, for whatever reason, seem to have the opposite effect and put me right to sleep. For me, what’s effective is reading in the dark on a Kindle on dark mode, listening to a podcast or audiobook,6 doing a crossword puzzle on my phone, or as a last resort, YouTube videos, which work for the same reason it’s so easy to fall asleep in front of the TV.

The problem with this is that most of these involve my phone, which also has all of those things that are great at keeping me up. Worse, when I wake up next to my phone, I inevitably start my day by getting sucked into inane phone stuff. This isn’t just a time-waster—the grown-up-ness with which I start my day usually sets the tone for the whole day. And the morning is often when some of the best insights pop up for me—if they have space to pop up. The phone kills that space.

So I came up with a little scheme. Which brings me to:

What is the best permanent lifestyle change you made in your life? – Elmar S. (Munich, Germany)

I dug an old iPhone 6 out of my drawer (I had tried to trade it in but I botched something with the “activation lock” whatever the hell that is and Apple sent it back to me) and this became my bed phone. No SIM card, not signed into iCloud, no apps except Kindle, iBooks, Downcast (podcasts), NYT Crosswords, YouTube, and some practical ones (alarm, calendar, notes and voice recorder for those morning insights). Then I leave my real phone in a different room, as far away and inconveniently located as possible. This has been pretty groundbreaking for me. I find that I actually look forward to saying bye to my normal phone and transitioning to the bed phone stage of the night. Bed phone still has worlds of fun in it—just the healthy-for-night/morning kinds of fun.7

Do you have an inner monologue? – Malaika D. (Groningen, The Netherlands)

I actually wanna talk about this. I kind of think I don’t have an inner monologue. I have certain moments when I’m actually talking to myself in my head, but I find that 90% of the time my head is up in the clouds (which is most of the time) I have a lot of thoughts going on but no specific words in my head. I can’t tell if that’s what everyone is like or if I’m weird. The reason I wonder about it is that when people explain why you should meditate, they say something like, “it’s important to quiet the constant ‘chatter’ in your head.” And I always wonder whether “chatter” is just the word they’re using for the constant stream of thoughts or if there is actually a voice to those thoughts in most people’s head but not mine. Someone tell me what’s up.

Anyway I get your bigger point, Malaika, and I’m with you, but I don’t have a good answer beyond something obvious like “try meditation!” So hopefully a commenter can tell us both how to fix ourselves.

What really makes you angry? – Ingrid M.

  • When I’m late and a person in front of me is going really slowly at the cashier
  • When I’m late in an Uber and the driver makes a wrong turn
  • When I’m late at the airport with no margin for error and TSA decides it’s the perfect time to search my bag and there are two other bags that need to be searched in front of mine and no one is searching any of the bags because the woman who searches bags is doing something else
  • When I go to a Broadway show and look at the little slip of paper in the Playbill and the main character is replaced by an understudy
  • When anyone needs me to do something that involves printing something
  • Bluetooth connectivity difficulties
  • When I get a new bottle of liquid soap and I can’t get the nozzle to pop up and instead it just spins around infuriatingly
  • Any other kind of packaging that’s hard to open
  • When my packages get stolen from my shit-ass building lobby
  • When I’m sleeping somewhere either with no A/C or bad A/C and it’s like 78º all night and I sleep 2 hours
  • When some gem of a photo op presents itself with my dog or something on the street and my phone camera freezes on the blurred screen
  • When I finally select the flight I want on Kayak and I’m relieved I got a good price and I go to buy it and it takes me to the airline site where I fill everything out and click “purchase” and it comes back with a message that says “the flight you’ve selected is not available.”
  • When people disagree with me about politics and think they’re right
  • When I bought a tin of fancy salt and the fucking lid was impossible to get off and I finally got it to snap off and the salt went everywhere
  • When I go to a site and select a menu item and the site finishes loading and the page jumps up just as I click and I select the wrong item and then I press back and select again and the same thing happens
  • Obscure name clues in crossword puzzles
  • When my opponent opens up a huge spot in Words With Friends and my letters are awful
  • When I’m at a breakfast restaurant and ask for Tabasco Sauce and the waiter remembers to bring it 0 out of 100 times
  • When I’m watching something riveting on an airplane TV and the captain makes a long, drawn-out announcement
  • When my tortoise shits on the floor while I’m not in the room and then tramps through it and gets shit on every square inch of the apartment floor and 3 inches up every wall
  • How long it takes an iPhone to turn back on when you plug it back in after it dies 
  • When I’m watching a movie with someone and I’ve seen it and they haven’t and they talk or look at their phone during an important moment
  • Sandwiches or burgers with hard, thick bread
  • When I’m in the car with my sisters and they think it’s an okay thing to blast their inane songs
  • When an apartment building puts a sign up that says “Please keep noise down between 10pm and 7am” because no one cares about late-night people (early riser privilege!)
  • The 99-year-old technology at my mom’s house
  • Websites that won’t let you go back – you just keep hitting the “back” button and it keeps refreshing the current page
  • When I get delivery food that includes fries and instead of a little container of ketchup, they give me a bunch of tiny packets
  • When I ask someone for the address where I’m going and they tell me directions instead of just giving me the address to put in my phone
  • When I go on vacation with people who want to wake up early and fill the day with icky activities
  • Jeans with button flies

Are we bigger than we are small, or smaller than we are big? – Jarhead (Toronto, Canada)

Biologically, we’re definitely bigger than we are small. A typical blue whale is about 15 times the length of a human and around 2,500 times the mass. A human compared to microbes is many orders of magnitude larger. We’re less big amongst mammals, but even there we’re bigger than we are small. A human is about 35 times the length of the smallest mammal—the adorable bumblebee bat—and 40,000 times more massive.

Moving beyond the realm of biology—if we consider ourselves in relation to well-understood objects, we might make our endpoints a proton (10-16m) and the sun (108m), in which case we’re much closer to the size of the sun than the size of a proton. Big again. But if we use the Milky Way (1020m) as our upper endpoint, then we become smaller than we are big.

Then there’s the whole range of size. According to my Universe in a Nutshell app (plug!), the smallest known unit of space, the Planck length (maybe our universe’s pixel size), is about 10-35m. The largest space we can see is the observable universe, which is around 1026m range. If those are our endpoints, the midpoint between them would be a bit smaller than the width of a human hair—making us bigger than we are small.

But the observable universe is probably only a tiny piece of the full universe, and then there’s the multiverse if that’s a thing. So the real upper endpoint is probably high enough, if it exists at all, to make us much smaller than we are big.

My takeaway here is that we’re extremely middling when it comes to size, which I guess isn’t the worst thing to be.

How do I get over a breakup if I’m still in love? – Anon

There’s no way to make an excruciating loss not excruciating except time, and I’m sorry to hear about the excruciatingness you’re going through.

Maybe it’ll help to remember that everyone you know, everyone you see on the street, everyone who’s ever lived in every place and time, has probably been there too or will be there in the future—because excruciating loss is an unfortunate staple of the human experience.

It might also help to remember that the part of you that feels like this wound will never heal is almost definitely wrong, and being wrong about that is also a staple of the human experience. Make that part of you write a letter to yourself, expressing how it feels. It’ll be fun to read that letter later, when you’re totally in love with someone new.

When parking, if both options are available, do you pull into the spot or back in? – Brighid D. (Manchester, NH)

Pulling in forward is Present You living their best life and treating Future You like their assistant. Backing in is Present You acting like Future You’s assistant. Pulling forward out of a spot you previously backed into is a highly luxurious experience, but as a procrastinator type whose jam it is to fuck over Future Tim, a rare one for me.

What is the coolest magic trick that you’ve seen? – Abhinav P. (Jaipur, Rajasthan, India)

Professional magic is by far the most underrated form of entertainment. Anytime I hear about a new magic show in NYC getting buzz, I buy tickets (magic over video can also be surprisingly great—check out this incredible show).

The most inexplicable trick I’ve seen was actually one where I happened to be the chosen subject from the audience. This guy (a “mentalist” not a magician whatever that means) had me go into my phone contacts and scroll and stop randomly somewhere and then remember where my finger was. It was on a friend named Lucy. He told me to put the phone away and concentrate on the name. He then told some fictional story and towards the end revealed that the main character’s name was Lucy. There was no way he could see the phone at any point, and having held it close to me, I’m almost sure no one behind me could see it either. Later in the night he correctly guessed my friend’s ATM PIN number. Nothing he did was okay. I’m upset just talking about this.

Why do you live in NYC? – Yertle T. (New York, NY)

NYC is an extreme place. It’s great at being exceptional and it’s great at being awful. It’s a high-priced filth bucket with great weather 25% of the time. It’s supposedly this world-leading city and yet it’s often behind the innovation curve and feels less futuristic than many other cities. The restaurants are world-class but it can also sometimes be hard to find a fresh salad for lunch. I could go on.

But it’s also a place where within a 20-minute walk of my apartment I can find every cuisine in the world, every type of art, every kind of shop, and a million different cultural experiences (many in secret underground nooks). It feels like the entire world is crammed into this city.

Among US cities, it’s not #1 for tech (SF), entertainment (LA), politics (DC), medicine (Boston), or a handful of other major industries. But it’s at least top 5 in every industry. NYC has breadth like nowhere else I know (at least in the US). This is especially appealing to me since I like to dig into a lot of totally different areas with my writing.

On a me-specific level, it’s a place where a large portion of my friends and acquaintances live, and of those who don’t live here, a large portion come through the city at least once a year. Nowhere other than NYC would I be able to get this level of in-person contact with such a large percentage of people I care about. (It’s also a quick trip to Boston, where much of my family lives.)

Finally, I really like the historical aspect. 300 years of human ingenuity gives a place a lot of character (I know that sounds paltry to non-Americans but it’s a big deal here!). Every street has an epic story to it. It makes the city like a big history museum to explore, and I like knowing that I’ll never even scratch the surface.

The awful things about NYC suck. A lot. But for me at least, the good things make living here a clear net positive.

How are you? Just checking in. – Jake E. (Kristiansand, Norway)

Besides the giant dark “I gotta finish this book” cloud hovering over my entire existence and preventing me from being present or content in any moment ever, great!

What is the silliest thing you have placed on top of your tortoise? – Juan

For his 14th birthday, we threw a party for Winston and tied a string around his shell with a helium balloon attached. This was a great way to make sure no one stepped on him because the balloon was at eye level, like another guest at the party. Anyway he doesn’t like parties so he moseyed his way into some little private cove, thinking no one would know where he was, not realizing there was a very obvious balloon giving him away.

What should one do if a piece of media (book or movie or song) that holds some nostalgic value to you turns out to have been created by someone who’s a ‘bad’ human being? – Anoushka D. (Delhi, India)

I think in most cases, we should feel fine loving the art we love, regardless of who the artist is, what they did, or what they believed. John Lennon was physically abusive in his early relationships. Picasso and Jackson Pollock were abusive husbands. Walt Disney wasn’t a fan of the Jews, nor is Cat Stevens. Ellen DeGeneres was a dick to her staff. Norman Mailer stabbed his wife. I could have listed 1,000 of these because artists are often messed up, unstable people who do or say bad things (and because most people secretly hold deep-down beliefs you would disapprove of if you knew about them). That a lot of great art emerges from troubled minds shouldn’t be a surprise or invalidate the art as great art to be enjoyed—at least not to me.

In many cases, we’re also talking about artists who lived in different times—times when your own ancestors who lived then almost certainly were doing or saying some things that would make them “bad people” by today’s standards.

For those who generally agree with me about this, there might still be exceptions. Maybe an important part of why you liked The Cosby Show was the belief that Bill Cosby really was a lot like Cliff Huxtable, and knowing what you know now, the show loses its charm. I just find those to be the rare exceptions, not the rule.

Finally, I would say there are two okay views to have here: 1) my view, that you can enjoy art by people you disapprove of, and 2) the opposite view, that not liking the artist ruins the art. What’s not an okay view is 3) “Art by a person I disapprove of should be banned so that no one is allowed to consume it, and any platform that doesn’t ban it should be punished.” There’s way too much #3 going on right now.

What countries were your absolute favorite travel experiences? – Samantha K. (Singapore)

When I was 20, I took a solo backpacking trip to Thailand and China and became infatuated with traveling. Since then (covid notwithstanding), I’ve tried to visit at least two new places every year. Here are some of my favorite travel experiences and best recommendations (most can be done on a backpacker’s budget):

Taking a commuter boat for three days down the Amazon from Belem to Manaus (Northern Brazil), sleeping in a mess of hammocks and looking out at the little river communities along the way.

Riding the Trans-Siberian railway from Moscow to Krasnoyarsk, looking out the window, making friends with passengers, and seeing how the political views changed the further east we got (more on that trip here).

Driving around desolate parts of Utah national parks, and sleeping every night outside the car on the edge of a canyon and waking up to the best view ever.

Staying with a family who lives in a yurt on the edge of the alpine Lake Song-Kul in Kyrgyzstan, spending the day in silence tagging along with the non-English-speaking shepherd as he did his thing with the sheep up on the mountain, and watching their weekly horse race.

Driving around Southern Italy and stopping for lunch in small walled villages on the tops of hills and waving to window nonnas.

The Yangon New Year’s Water Festival in Myanmar. Good god. We got invited onto the back of a truck with a bunch of locals, and the truck proceeded to drive around the city for five hours while we got absolutely wrecked by power hoses.

Driving from Dubai to Southern UAE and camping out on the softest dunes ever.

Going to Gen Yamamoto’s bar in Tokyo for six of the fanciest little cocktails imaginable (he’s kind of the Jiro of cocktails) over two amazing hours.8 Also just the entire country of Japan in general.

Taking a boat two hours from a small Greenlandic town to a 46-person village and doing a stay with the town grandmother. More on that trip here.

How can I become a higher-rung thinker? – Avery H. (Zeeland, MI)

(If you don’t know what “high-rung” means, check out chapters 7 and 8 of The Story of Us.)

To borrow a Paul Graham suggestion, keep your identity small. High-rung thinking isn’t some fancy thing. It’s just being self-aware about what you know and what you don’t know—about staying close to the humility sweet spot.

One of the biggest impediments to this is when you start to identify with certain beliefs, stances, or ideologies. Once that happens, your Primitive Mind enters the equation and will do whatever possible to keep you from changing your mind, which cripples your ability to learn. Keeping your identity small is hugely liberating, and a lot less stressful. When your identity is small, you have nothing to cling onto and you can just relax and explore. So start by doing a self-audit and figuring out which ideas you’ve come to hold sacred—we all do this, it’s human—and remind yourself that they’re just ideas, they’re not you.

What are your favorite podcasts? – Laura B. (Amsterdam, The Netherlands)

Currently hooked on Dan Carlin’s Hardcore History. 100 people recommended it to me and I kept putting it off and finally listened to the 6-part WWI series and that was that for me. Went back to episode 1 and went through chronologically, currently up to episode 42 and loving every minute. History is so incredibly riveting if told by the right person, and Dan is the right person.

What is your favorite form of transportation? – Sidhi (San Francisco, CA)

Without question the hoverboard. A few years ago, I was walking in NY and some guy passed by with one and I accosted him and asked him what the fuck that incredibly fun-looking thing was. He told me he actually sells them and gave me his info. A few days later I was the weird guy rolling around NY like a futuristic nerd, having the time of my life. I got really good at it and impressed lots of strangers and everything was great. Then, after a few weeks, the battery, though appearing to be half full, suddenly died and I wiped out in the middle of the street mortifyingly. I carried the board home and charged it up and pretended like nothing had happened and kept riding around. Then a few days later the battery died again out of nowhere and I wiped out mortifyingly in the street again. Turns out I had bought a cheap Chinese-made knockoff without realizing it. Bummer.

I was gonna get a new one, this time online with some research and not from some shady fuck I met on the street—but right around that time, NY decided it didn’t know what a hoverboard was and didn’t know how to categorize it or whether it should need a license or whether they should go in the bike lane or sidewalk and just shrugged and banned them. Disappointing all around.

If I time machined to 2040, one of the things I’d hope to see would be lots of people traveling around on fun hands-free toys like that.

How do you feel about advertisements? Do they annoy you, or are you enticed about what they’re trying to sell to you? – Fred M. (Chicago, IL)

I love targeted ads, when the AI is good enough. The problem with targeted ads is normally they’re not really good at targeting, so they’re useless and annoying. But Instagram has gotten pretty good at it and now I own a gravity blanket, a moon pod, stretchy jeans, a great hoodie, and like 19 fidget toys and physics toys and other stupid things that make me happy. Buying dumb shit you like is a good thing to do because you’re supporting small businesses (this is the story I tell myself), and effective targeted ads connect supply and demand in an optimal, nuanced way. I hope down the road the targeting gets so good that ads will pop up and be like “your dad’s birthday is coming up, here are five great options for him that we know he wants—click your favorite and it’ll arrive to him, gift-wrapped, on his birthday!”

I know I’m supposed to feel the opposite way about all of this because it’s creepy and non-privacy-y, but I’m gonna be dead at some point for trillions of years so who cares if big companies are manipulating my simple psychology for profit, if it brings me joy?

If you were forced to live in a country outside of the top 50 for 5 years, which country would it be and why? – Dom W. (New York, NY)

If you ask me when I’m feeling energetic and ambitious, I’d say Nigeria. After visiting there in 2014, I came away feeling excited. With over 200 million people, it’s the 6th-most populous nation, and Lagos is a massive metropolis bustling with energy—somewhere I wouldn’t get bored and where I could find lots of outlets for my own projects.

But Nigeria would also be a hectic choice, and a hard place to live at times, so if you ask me when I’m tired, it would probably seem a bit “much” for a five year stint. In that mood, I’d choose Laos. Never been, but it looks insanely beautiful and the food would make me happy.

What is the best piece of advice you have ever received? – Laura H. (New York, NY)

I met Chris Anderson, the head of TED, in 2015. He had read a few WBW posts and offered me the opportunity to give a TED Talk at the 2016 conference (which was six months away). Immediately full of both gratitude/excitement and dread/anxiety, I asked him if it might be better to wait a couple years until I had some more speaking experience. He paused thoughtfully for a few seconds before saying, “There’s no time like the present.” I took his advice. Since then, his voice saying those words has popped into my head again and again during hard decisions, and I’m yet to regret following them.

Great advice is sometimes great because it’s totally original or framed in an original way. But, as in my story, a well-known platitude, at the perfect moment, can also make a huge impact. What makes Chris’s advice so valuable to me wasn’t that it was something new—it was that the lesson I learned from taking the advice in that particular moment turned a cliché into a mantra.

What are your thoughts on TikTok? – Annie (Ontario, California)

For a long time, TikTok was this impossibly annoying-seeming thing that wouldn’t go away. Then my friend Isabelle started posting pro-nuclear-energy videos on TikTok, so I finally got the app to watch them. Then I got sucked in.

The thing is, the TikTok algorithm is really good. It knows me better than I know myself. I don’t follow any accounts on TikTok, I just go to the “For You” tab and start scrolling down the video stream TikTok has prepared for me, and the curation quality is strong.

Sometimes there are people being funny. Sometimes there are people doing pranks. There’s George and Hector and this squirrel and this drunk chicken and the bees lady. And then there are the trolls. People trolling their boyfriends and girlfriends and husbands and wives and cats and dogs and babies and teachers and customers. And a lot of people trolling their parents. TikTok isn’t for everyone. But it is for me.

I’ve now created a fun activity, where I “favorite” the best videos during a scroll and at some point beam my phone up to the TV to show my wife the favorites. Highly recommend this activity.

Are you a tabletop gamer? – Ryan M. (Odessa, TX)

Of the staples, I love Scrabble and have played like 1,000 games of Words With Friends (which has in turn fucked up my Scrabble game). I have periodic chess phases, though they never last long enough for me to become not bad (my chess.com rating peaked in the 900s during my last bout).

My sister is always upsettingly pushing new games on me, which I first resist and then become addicted to. Recently, she’s gotten me into Coup and Codenames (both delightful). I’ve dipped my toe into the Polytopia wormhole and am scared about the possibilities. Not a tabletop game, but a consistent winner with groups is what my friends and I call the Name Game (variations are called Celebrity or Fishbowl)—rules in this footnote.9

What supplements or multivitamins do you take? – Stacey W. (Perth, Australia)

I usually take some kind of one-a-day multivitamin that I deep down believe does nothing but I take it just in case. And a spoonful of Metamucil every morning so I can be happy.

And then I like to experiment with work/productivity supplements. I’ve recently been trying a combination of gingko biloba and lion’s mane supplements. I’ve also been trying out Qualia Mind, made by Flow Genome Project, but just started so no verdict yet. Always open to suggestions, so let me know if you use anything that works well for you!

What’s a recent embarrassing and/or interesting rabbit hole you’ve found yourself going down semi-unintentionally? – Jace L. (Tulare, CA)

I have a love-hate relationship with YouTube. Love the videos, hate myself. The problem is, YouTube spirals happen for me almost exclusively when I’m in the Dark Playground—that place you’re in when you should be working but you’re not working. Either someone sends me a YouTube link or I go there for genuine research purposes—and then the algorithm pulls me into the pit.

Some of these pits are at least productive, when I’m learning something about science or history. Others at least are high quality art or comedy or something else interesting.

But then there are the dark moments. When nothing good at all is happening. In order to answer Jace’s question, I decided to play a dangerous game: lifting off the hood of my YouTube history and seeing, staring me in the face, all the places I’ve been. Some of the more disturbing finds from the past few years (each spiral goes up chronologically, with the first video I watched at the bottom):

Here’s a typical run:

Also typical:

One day, I spent a lot of time with this man while he hurt himself:

This day started with string quartet covers before things got Jewy:

Sorry not sorry:

Who needs to work when you have Hitler!

Needed a drink at the end of this:

The Bushmen and the Pygmies are my friends and not yours:

Not sure what happened here:

No regrets:

Not proud of this:

Can’t explain this:

Not useful:

Possibly the strangest sequence I found:

Would you rather be 11 feet tall or nine inches tall? – Alexander R. (Windhoek, Namibia)

Surprisingly hard one. 11 feet seems like the obvious answer. You’re a full “thing” everywhere you go and everyone would flip out, but then they’d get used to it and you could still have an almost normal life. You could still have an almost normal house. You could for sure play in the NBA. You could still have sex.

No one is fucking a nine-inch tall person. And you couldn’t go anywhere without everyone you’re about to see being fully prepped ahead of time for the situation. Your house would be a joke. That said, there would be benefits. You could sneak into anything and hide very easily. Like you could probably smuggle yourself on a plane to Russia without anyone noticing, sneak into the Kremlin, and listen in on the most secretive Putin meetings while hiding behind the curtains or somewhere. But then what?

So I guess my answer is that the nine-inch-tall option would be very fun for a while but then when you got bored of the perks, you’d be confronted with your life as a pencil person and would probably regret the choice. So I’d go with 11 feet tall.

Can you suggest methods for dealing with the crushing realization that people you used to have a lot of respect for don’t hold the principles/values you thought they did and therefore might no longer be worthy of your respect? Especially close people like parents, teachers, mentors, and religious leaders? – JD I. (Charleston, SC)

I find this to be a pretty reliable graph:

The better I get to know people I’ve previously demonized, the more pleasantly surprised I am. The better I get to know people I’ve previously lionized, the more disappointed I become. This makes sense, because people are neither demons nor angels—they’re humans. The crushing realization you’re experiencing is probably less some awful truths you’re learning about people you admire and more a truth you’re seeing about humans in general. Even the people you look up to are fallible and flawed—but that doesn’t mean they can’t also be worthy of your love and respect.

With all that in mind, I’d consider three things:

First, take a hard look at ways you feel these people are disappointing you, and try asking whether your own judgments might be off. On more than one occasion, I’ve scoffed at those I once considered wise only to later realize that they weren’t all that wrong after all.

Second, think about all the ways a human can be good and wise, and you’ll be reminded of certain ways these people are still very much worthy of your admiration. Focusing on those areas can restore the respect you want to feel towards them.

Finally, where you do firmly believe you have something right that they don’t, remember that wisdom is a multi-generational collaboration. When mentorship works, it raises mentees to be wiser than the mentor—mentees who can then be even better mentors to the next generation. This is how the wisdom trend slopes continually upwards. If your mentors are forever wiser than you, they failed as mentors. So when you do feel wiser, it’s not a bad thing—it means it’s your turn to take the wisdom torch and try to bring it to the next level.

What are your most left-wing, and most right-wing, political positions? – Hannah J. (London, UK)

At the moment I’m feeling progressive about the legalization of drugs and sex work (in both cases, the black markets that result from illegality seem to cause more harm than the thing itself), and conservative about wanting the government to be smaller (bigger government seems to correlate with both less competence and more corruption).

Do you plan to write updates for the AI and SpaceX posts? – Matt J. (Sartell, MN)

Yes and yes. SpaceX first.

What conspiracy theory do you think is most probably true? – Maja (Ottawa, Canada)

One that I’ve been considering (only a hunch, no actual info) is that big alcohol companies (Anheuser-Busch, etc.) are the new big tobacco companies (Philip Morris, etc.). And instead of trying to suppress evidence that cigarettes are disastrous for your health, the big alcohol companies need to keep all the other fun drugs stigmatized and illegal—and with tens of billions of dollars at stake, they’ll spend billions of dollars to keep the status quo the way it is.

When I grew up, the rules were clear: if a party is a cool party, it has alcohol at it, and if you’re a cool person, you drink alcohol. If your party doesn’t have alcohol, it’s not a cool party, and if you don’t drink alcohol, you’re not a cool person. If you take a step back and think about it, that’s kinda weird. This one particular drug, of so many, has been deemed A) the socially acceptable, non-stigmatized one, and B) the definition of cool. Meanwhile, alcohol is just an alright drug—probably a below average one—while also being one of the most dangerous and harmful ones. It makes no sense—until you remember that cigarettes made no sense either…

What’s your best advice for someone moving to NYC? – Tessa C. (Kansas City, MO; living in Dublin, Ireland; moving to New York, NY)

Get rid of your stuff (or store it somewhere) unless you really love it. Small apartments can be cozy and great if you don’t have too much stuff.

Do all the touristy things (Statue of Liberty, etc.) while you’re still in the honeymoon phase, because if you don’t, you’ll never do them.

Look up a list of the best pizza places in the city and go to one a month until you’ve tried them all. Lucali and Paulie Gee’s are my top recs.

Look up lists like this and actually do the things. Nowhere has more amazing secret spots than NYC.

Why the King of Spades? And are his eyes always looking up that way? – Hector F. (San Antonio, TX)

When I was tossing around potential names for the site, “Miniature King” was in the running. At least to me it was, before it was promptly vetoed by my wife / then girlfriend. But in the process, I had become attached to the visual I had in mind—something with a very small, very tyrannical king who was outraged about being the non-consensual mascot of a random blog. So when it was time to design the eventual site, I stuck with the king as the mascot. He’s looking up because he’s rolling his eyes at the concept of being the WBW mascot and because he’s upset about the little men that are marching up onto his head, which is not a very dignified thing to be happening to a great tyrant king.

What’s a view/opinion you were holding for a long time but changed recently (let’s say in the last few years)? – Mike L. (Singapore)

Religion being a generally bad thing overall. Obviously religion can be, and often is, a bad thing that does more harm than good. But what’s become more clear to me is that we’re a religious species, period, and while today’s prominent religions are certainly flawed, the political religions of history or today are often even worse.

It just doesn’t seem like we’re a species that’s ready for widespread atheism quite yet. Until things change, traditional religions are probably more helpful than they are harmful. Still thinking about this one.

How do you balance reading the news/staying up to date on current events with avoiding burnout and the feeling that everything is terrible all the time and nothing you do matters so why bother? – Fenway D. (Greensboro, NC)

A lot of what used to be news has morphed into propaganda, and there’s no shame in ignoring propaganda. The really big stuff will filter its way to you no matter what you do.

If you do want to keep up with the news, it also doesn’t have to happen directly, via the media. Listening regularly to a few good interview-style podcasts, for example, can be a great way to get a feel for the big issues going on in a more indirect way.

As far as feeling like nothing you do matters, I actually think that’s an intuitive but incorrect assumption. In a moment when so many people are afraid to express their real views in public, doing so can make a real difference.

You often mention being inspired and challenged by interesting conversations. How have you increased the proportion and/or frequency of these conversations in your life? – Andrew N. (London, UK)

One thing that can go a long way: see the people you really like talking to more frequently. This is about more than the obvious point that doing so literally increases the cumulative time you’ll talk to them: When you don’t see someone frequently enough, every time you do hang out ends up being a big catch-up, where both of you update the other on your lives. This doesn’t seem like a problem until you consider that the whole relationship now mostly consists of catching each other up (and reminiscing about old times because you’re not making any new times together). This isn’t actually hanging out. Hanging out means shooting the shit about whatever’s on your mind and exploring new topics together. If I haven’t seen someone in forever and then they’re in town for a week and I see them twice, it’s the second hangout, with the catch-up out of the way, where the more interesting conversation usually happens.10

Two friends and I started zoom-chatting every Friday during covid and it just turned into a thing that now happens once a week, and the conversations are great because they’re free of big-picture catch-up—they’re either about ongoing keeping up, or more often, just fun divergent conversation about third-party topics. If you have anyone you like talking to enough to do it a few times every month, try to make that happen.

Do you have some tips of where and/or how to meet people who tend to think more high-rung, both online and in real life? Or is it more about recognizing them scattered throughout our lives? – Steven D. (Zavantem, Brussels, Belgium)

Where’s the entrance to the idea labs? I’m a teenager and right now I’m trying to accumulate knowledge and skills as much as I can. There are quite a few people who inspire me and they all seem to be in a network of other smart/amazing people. The question is – how do you find other people who want to grow and exchange knowledge? – Mikhail I. (Moscow, Russia)

Three ideas:

1) Mine people you already know for hidden high-rung-ness. Sometimes two people get stuck in a dynamic where both feel like they have to agree with each other, which is kind of boring and stressful and makes it hard to dig deep and explore anything—but neither one actually prefers it to be this way. They’re just stuck in a social rut. In this case, all it might take is one person actually starting to say what they really think and being willing to break the “disagreement” ice, and the whole relationship can transform into something much more interesting and high rung. Of course, there are plenty of other times when an Echo Chamber dynamic is in place because one or both people very much want it to be that way, and an attempt by one party to snap out of that will go over badly (btw, this isn’t the end of the world—it just means that high-rung conversation won’t be a part of that relationship, you can still enjoy the rest of what it has to offer). If you do find that most of your existing relationships are stuck on the intellectual low rungs, you’ll want to supplement things with some new friendships that can better indulge your intellect. This is where the other two points come in.

2) Collect high-rungers you happen upon in the wild. Post-college-age people tend to be really bad at making new friends. But it doesn’t have to be this way. There will be moments—at work, at a dinner party, at some event—when you discover a super interesting-seeming person you really like talking to, and you wish you were friends with them. If you feel this way, there’s a decent chance they feel the same way about you. And if you can fight off your mammoth (who is irrationally horrified by the possibility you may put yourself out there and get rejected) and get in the habit of getting their phone number and sending a “great meeting you! would love to chat again sometime” text—and then if the response is positive, actually scheduling a coffee or lunch or drink (mammoth: “but what if they were just being polite!”), over time you’ll accumulate a whole new basket of high-rung brains to play with.11 It’s really easy to just not put yourself out there for new friendships as an adult, but those who do are usually rewarded.

3) Actively go out looking for high-rungers. This is the ickiest, most amorphous item, and I don’t have specific suggestions beyond: put yourself in high-rung environments whenever possible. That can mean the place you live, the place you work, the places you spend your free time, the forums you hang out in online, the events you go to. Speaking of which—

Do you plan to have another Wait But Hi?
– Fausto L. (Madrid, Spain)
– Mini (Perth, Australia)
– Sam W. (Raleigh, NC)
– Barbara P. (Budapest, Hungary)
– Kelvin J. (Turku, Finland)
– Chelsea I. (Perth, Australia)

Yes, definitely. Like everything Wait But Why, this has been on hold because of the Story of Us albatross, but the answer is yes. We consider “Wait But Hi” to be not a single event, or even a single type of event, but the “in-person” category of Wait But Why. The first Wait But Hi centered around a hypothesis that people who read WBW would probably like meeting other people who read WBW, and the event confirmed it. Alicia, Andrew, and I have had many brainstorms about other creative WBH formats, so stay tuned.

What animal would you shrink down to pocket size (or the size of a coke can) to keep as a pet? – Tandice U. (New York, NY)

Elephant.

Most other animals already have a mini version (a small tiger or lion is basically a kitten, small alligator is a gecko, small buffalo is like half the weird little dogs I see on the street). But there’s nothing like a small elephant. Plus they’re usually so big so it’s extra funny. Plus they’re smart and have great personalities. And they live like 80 years. The ideal miniature pet.

Who is your favorite Tim? – Tim (Auckland, New Zealand)

Tim Berners-Lee

Runner-up: Tim Duncan

If you had power to dictate what should be taught in schools—things that everyone had to master before they get to enter adulthood—what would you add to the curriculum? – Audrey T. (Toronto, Canada)

General (should be part of many classes):

  • Independent, first-principles thinking
  • How to argue and debate productively
  • How to be an effective learner
  • Intellectual humility
  • Independent problem-solving skills, through practice

Subject-specific:

  • Basic math
  • Basic science, and more importantly, a lot of practice with the scientific method
  • Basic civics
  • Basic money management
  • Enough writing and speaking skills to be an effective communicator
  • Enough history to understand why fundamental liberal rights were invented and how precious they are
  • Older students should be able to go deeper in the areas they feel most drawn to or, if resources allow, take electives where they can learn a more specific skill
  • Indoctrination in the teacher’s particular political ideology
  • Oh wait definitely not that last one

I just started a new relationship. How do I not fuck it up and/or how do I make love stay? – Robert B. (Denver, CO)

A general suggestion: Have your mantra be “I’m gonna show them exactly who I really am and if it turns out they don’t like it, they’re not the right person for me anyway.” Takes all the pressure off.

A specific suggestion: Plan fun, creative, exciting “mystery dates” and refuse to tell them what the plan is until they get there. A mystery date can be any kind of experience or activity—just something outside the box.

Do you think, overall, covid accelerated progress of the human colossus or slowed down progress of the human colossus? – Sean M. (New Orleans, LA)

Desperate times call for desperate measures and desperate measures have a knack for wildly accelerating progress and innovation. At least in the modern era, this phenomenon usually outweighs the destruction and lost time caused by desperate times. The classic example would be World War II, which is why we have the computer, the nuclear weapon, and the moon landing—or at least why we got those things as quickly as we did.

Covid certainly slowed down progress in some ways—but a ton of learning and innovation happened in the past year that may have set us way forward in some very important areas. The average person in 1945 knew that the war had advanced certain technologies, but few could have foreseen the amazing ways those advances would be applied in the decades to come. If something like mRNA technology has just taken a major leap forward, the people of 2050 might have covid to thank for some of their most groundbreaking biotech.

Are you optimistic about the future of the United States? – Stephanie W. (Washington, D.C.)

All signs point to “no”. But American history is full of moments when all signs pointed to no. So yes?

What do you hope Gen Z does with the world? – Anne Z. (Winnipeg, Manitoba, Canada)

Makes political tribalism a lame thing for lame old people.

If you were an omnipotent god for 24 hours, what would you do? After the 24 hours, you’d turn back into your regular self but still remember that day. – Lena S. (Munich, Germany)

“Omnipotent god” is a very legit amount of power. Here’s what I’d do:

Restore the level of carbon in the carbon cycle to 1850 levels and invent some insanely powerful and efficient, cheap-to-make, totally safe nuclear fusion reactor that you could fuel with garbage like the Delorean in Back to the Future 2, and put instructions for how to make it online.

Randomly select a color from the Pantone deck and make all human skin that color.

Invent an easy, cheap way to make succulent lab-grown meat and put the instructions online.

Make the U.S. government a parliamentary system.

Learn the deal with the Fermi Paradox.

Learn the deal with dark matter.

Tribalism out. Teleportation in.

Tweak the human body so that Oreos and Chinese food are both perfect health food.

All couches are forever incredibly comfortable and like an L-couch except the long part of the L extends across the whole couch, so it’s like a king bed.

All internet is free and fast forever.

All human brains are on a mild dose of MDMA at all times.

Terraform Mars to Earth-like perfection with a snap of my omnipotent fingers. Make a few backup copies of Elon that activate one by one if he dies or gets bored. Incept the idea in Elon’s head that Tim Urban absolutely must be on the first voyage to Mars but not to do any icky work, just to blog about the experience.

Perfect anti-aging technology so everyone can revert their body to any age they prefer and stay there until they’re bored enough of living they’re ready to die. Put instructions for the technology I used to make Elon backups online so everyone can make backups of themselves in case they die in an accident. Use my omnipotence to ensure that all backups are in fact still “you.” All backups are constantly updated to mirror the main version, and only one can be activated one at a time.

All sentences in articles or tweets that are untrue automatically appear in red block letters.

My book is done and everyone agrees it’s just “a really good point.”

My dog is perfectly trained, no longer needs to eat or drink or pee or shit, bathes herself once a week, lives forever, and can be switched off when we go on vacation.

All my shrunk t-shirts are unshrunk and permanently unshrinkable.

There’s a Dan Carlin episode for everything that’s ever happened and Hamilton goes all the way through to today, instead of stopping in like 1802.

Rainbows are climbable and slidable. Clouds are jumpable.

All ocean and pool water is 95 degrees F. All hot tubs 104.

Paper straws are banned, plastic straws are back.

Tweak the human body so exercise makes you fat and depressed and stagnation makes you fit and happy.

Create a benevolent superintelligent AI that “just gets it.” This AI ensures that all future AI is friendly and that no one is too big of a dick. The AI is also in charge of making sure that none of my new rules or inventions backfires in unexpected ways.

Who is your dog? – Tim Urban (New York, NY)

Glad you asked!

What are your thoughts on Elon’s controversial side? – Ekin K. (Izmir, Turkey)

It emerges from the same quality that invents new things, challenges conventions, and changes the world: extreme originality, whether people like it or not. So I’m all for it.

How was Burning Man? – Zenon S. (Columbus, MI)

Roughly 100X cooler and more fun than I had expected.

Why hasn’t anyone created something between a cat and a dog? What would it be called? – Leo L. (London, UK)

How has writing Story of Us changed you personally? – Ankit L. (Pune, India)

Writing the Story of Us—first as a series and now as a book—has totally reshaped the way I think. The core mental model in the book has become a lens I use constantly. I’ve also learned a lot about politics, history, psychology, and sociology that I didn’t know before. My goal with the book is to take the most important parts of both the framework and the things I’ve learned and package it in a way that can resonate with many different kinds of minds and actually stick. I want people who read the book to feel 20 years from now like it still impacts the way they think.

There were lots of mailbag questions about both the series and the book, but given that every month, as I get closer to the finish line, what the best plan is seems to shift, I’ll leave it at that big-picture point for now.

What’s your favorite thing that has come about because of something you’ve written? – Laura D. (Hampshire, UK)

Several people have told me that they moved closer to their family after reading The Tail End. That’s a big deal!

I’ve been a Patreon for like 4 years now. But now that you will be releasing a book, will you still need my support? What are your plans for the future? – Kyros J.(Barcelona, Spain)

Here’s how I think about Patreon, from our side:

Patreon is a means of creative freedom. Without Patreon, money would be a sizable part of our decision-making process. We’d have to take on sponsorships, sell products, and use paywalls, whether we felt good about it or not. Patreon allows us to do whatever we think actually makes sense for us and what we’re trying to do. It allows us to make a big list of potential projects and to prioritize them based on what we’re most excited about and what we think is the most important, not based on what will make the most money. Patreon means that we’ll do some projects that make money and others that won’t, and we don’t have to worry too much about which ones are which. It allows us to start a podcast without a sponsor and have the “we’ll get a sponsor if and when it feels right” attitude. It allows us to turn the Story of Us into a book, because it makes sense in this case, and to have turned down opportunities to make other post series into books, because it wasn’t something we felt made sense. It allows us to email readers about a product in our store only when we’re actually excited to tell them about that product. Having money as a secondary concern is an unbelievable creative luxury and one that makes Wait But Why a better place.

Here’s how I think about Patreon, from your side:

The independent internet creator is a pretty new kind of thing, and I think we’re all still figuring out the best ways for that to work, financially. Patreon gives regular readers three options:

1) I like this person’s work, and I’d like to pay something for it

2) I like this person’s work, and I’d like to pay something for it, but right now I can’t afford to

3) I like this person’s work, but it’s not something I want to pay for

The great thing about having a sizable audience is that if a small portion of our readers end up in Category 1, we’re covered. So far, that’s exactly what’s happened. Category 1 readers have combined to be an incredibly important support for us and provide that treasured creative freedom I talked about.

But that also means that we’re totally cool with Categories 2 and 3.

People in Category 2, who would like to support us but don’t have the budget to spare: not only is that okay, but please know that the last thing I want is for WBW support to be an uncomfortable or stressful expense in your life. I love the idea of WBW being free for people who are only able to consume free content at a given time in their lives.

People in Category 3, who simply prefer to put their dollars towards other causes, please continue doing so. This is the cool thing about the voluntary donation model—only some people have to do it for it to work perfectly. Category 1 has everyone else covered.

If you’ve been a Category 1 supporter of ours, huge hug, we love you, and we’re endlessly grateful. You should also never feel weird or guilty about stopping. Supporting us doesn’t have to mean supporting us forever. Once a Category 1er, always a Category 1er, no matter when you stop.

Finally, Patreon is only one way to support WBW. There’s also buying stuff we sell and, probably most important, sharing WBW with others.

As for the future of Wait But Why:

At the moment, it’s hard to imagine a time when I will want to be doing anything other than creating new things on this platform. My “future topics list” is longer than I could ever get through in one life, and as long as I’m curious, I’ll be working my way through it.

Within that broader picture, there are a ton of fun possibilities. We plan to expand the platform from just writing to writing, audio, and video. We want to keep trying new things and new collaborations (like the app we made with Kurzgesagt). We want to dig in deeper with the community with more in-person events. We’ve sketched out plans for a new, much better site.

It’s been a slow few years as I’ve worked on a single, big project, but my excitement and energy have never been higher. I hope I’m still making good things when I’m 80 and I hope you’re still here with me.

—————

If you like Wait But Why, sign up for our unannoying-I-promise email list and we’ll send you new posts when they come out.

To support Wait But Why, visit our Patreon page.

For my daily musings, follow WBW on Twitter and Instagram.

—————

Other places to go:

Mailbag #1

The Travel Series

The Procrastination Matrix


  1. Depending on how transparent viruses are, we might need to stack a bunch of these virus squares to make it opaque enough to see, which would raise the total number of viruses needed into the millions or even billions.↩

  2. The NIH does note that most of the human cells are red blood cells (a further rabbit hole reminded me that red blood cells look like delicious gummy candy), and they admit that red blood cells are kind of just “bags full of hemoglobin.” If you only count human cells with a nucleus, the ratio is 10 bacteria for every human cell.↩

  3. Zachary got more than he bargained for here.↩

  4. As a one-time budding film composer, it used to be stressful playing the score I had written for the director the first time. They had inevitably become attached to their stupid temp score (the existing film scores directors put into the film before handing it over to the composer to write the original music) and were hearing mine for the first time, so they’d predictably (infuriatingly!) not like it on the first listen. I’d play it for them a few times and the next day, they’d listen again and usually would suddenly like it. Film scoring is annoying.↩

  5. Every time I call her, Nana asks me when my book will be done, and her new thing is telling me now that I’ll be very regretful if she dies before she gets to put it on her coffee table—thanks Nana!↩

  6. If you sleep in a bed with someone else, you probably can’t just listen out loud to stuff without bothering them. A single AirPod solves the problem for me—and you can just take it out when you’re drifting off and that pauses the podcast (or video).↩

  7. A more austere version of this is just having books or a Kindle in the bedroom and phone elsewhere. But I like the variety the bed phone gives me.↩

  8. Email office@genyamamoto.jp to make a reservation, at least a month before you’ll be in Tokyo.↩

  9. Minimum four people, max 15 or even a little higher. Two teams, everyone writes down names on little slips of paper, folds them and puts them into a big communal bowl (one name on each slip), and each team also has their own empty bowl. You want about 40 names total, so if it’s a 2-on-2 game, everyone should put 10 names in the bowl each. If it’s 5-on-5, everyone should do four names. Names can be celebrities, friends of the people playing, or famous fictional characters—you can go a little obscure but you want at least half of the group to know each name. There are three rounds. In the first round, Person 1 on Team A starts – they have one minute – they pick a name out of the bowl and say whatever they want (other than stupid cheats like saying ‘the name rhymes with Ill Pinton!’ and no doing impressions or pointing at anything – just verbal description) – when one of their teammates successfully calls out the name, they put the piece of paper into their team’s bowl and pick the next name. After a minute their turn is up and the name in their hand goes back into the big bowl. Teams alternate and who the name-picker is on each team alternates with each turn. When all the names are out, Round 1 is over. Names in each team’s bowl are tallied up and the score is written down. Then all the names go back into the big bowl. Round 2 is the same as round 1 except the name-picker can only say one word now. No facial expressions or anything. One word and then they have to go blank and hope it’s enough for their team to get it. Round 3 is like the other 2 rounds except no words now – just silent acting. Usually that’s the end but if you wanna get weird, bonus round 4 is the same as round 3 except the name-picker now has a bed sheet over their entire head and body, like a kid in a ghost costume. The team with the highest cumulative score in the 3 (or 4) rounds wins. Only other rule is that if in round 1, the name-picker doesn’t know the name, they can’t pass, so they can try to get it using other tricks “first name is the same name as the first U.S. president, etc.” or they can challenge. When a player challenges, the timer stops, they say the name, and whoever knows who that person is raises their hand. If fewer than half the other players know the name, the name-picker counts the name as a point and it goes into their team’s bowl, the timer is started where it left off, and the turn continues. If half or more of the people know the name, the other team gets that name in their bowl and the player’s turn then continues. Those who don’t know the name need to learn it then because in future rounds there are no challenges.↩

  10. It’s always an option to just skip the big catch-up with friends you don’t see very often, but that’s not great either because it’s hard to be close when you don’t know what’s going on with someone.↩

  11. A less risky option if your mammoth is flipping out is to invite the person to a gathering you’re organizing.↩

The post Mailbag #2 appeared first on Wait But Why.

The Big and the Small

By: Tim Urban
22 September 2020 at 16:28

I have a surprise for you.

I’ll tell you about it in a minute. First, let’s have a little fun. Come with me.

I haven’t told anyone this before, but I actually live in the fun room. I just gave you a tour of my house, where I spend a large portion of my life thinking about the size of things.

I have visitors in the fun room from time to time, but after a few minutes, they’re usually pretty funned out and leave me to my crises. But one day, something unexpected happened.

It was 2013. Wait But Why was a few months old. And I got an email from someone named Philipp Dettmer. I was slightly unsettled by the pp and tt, but I decided to read what he had to say.

He explained that he lives in Germany where he makes animated educational videos about a lot of different things, kind of like Wait But Why but a different medium. I took a look at his YouTube page. It was named a random string of letters:

kurzgesagt

Apparently it means “In a Nutshell” in German, but I didn’t know that at the time and was very close to being done with Philipp Dettmer for good when I decided to watch one of the videos.

It was delightful.

I watched another. And another. And then it hit me.

Philipp—this random man in Germany—also lives in the fun room.

The next day, we were on the phone. There was a lot to talk about. We decided we had to do something together, and we settled on adapting one of my early posts into a kurzgesagt video.1

In the seven years since then, Philipp and I have become great friends, and I have not missed a kurzgesagt video since. Whenever Philipp and I get dinner, we head straight to the fun room to talk about the universe. And a few months ago, we decided to collaborate again. It was time to go public with the fun room.

At some point in our pasts, we had both become enamored with two fun room icons, Cary and Michael Huang, known on the internet as the Huang Twins. The twins do a lot of cool things, but it was their Scale of the Universe toy that we loved most.

Inspired by their work, we decided to go for it. We wanted to make the best size explorer we could imagine. We called it Universe in a Nutshell.

Of course, it took roughly 18 times longer than we thought it would. We brainstormed the interface for quite some time, and then Philipp, along with the incredible kurzgesagt team, dug into working on illustrations and animations, while I worked with the team on the written explainers.

Thousands of human hours later, the app is done. 250 objects, 30,000+ words of explanations and fun facts, just the right mood music, and what we think is a pretty great interface.

Here’s how it works:

The app is a giant wall. The wall is impossibly large—large enough to fit full-sized galaxies on it. The wall is also impossibly high-resolution—hi-res enough to contain sharp images of subatomic particles.

I know it seems weird that I’m talking about it as if it’s a real object, but I have a good reason: to get the full mind-blow effect, you have to realize what you’re looking at. You’re not moving forward and backward through a tunnel of objects—it just seems that way because your brain will refuse to accept the insanity of the actual sizes you’re seeing. By reminding yourself again and again that all of these objects are “painted” on the same 2D wall, you’ll hopefully experience some fun mind-bending moments.

The app looks like this:

And this:

And this:

And there are three ways to zoom:

You can tap any object for an explainer and fun facts. All of the facts you just read in the fun room are also in the app, along with a million other things. (The surprisingly-good-for-Wait-But-Why-illustrations from the fun room are the work of Kurzgesagt. Don’t think this is some new normal.)

Three more things:

1) The app is $3 in the US store, which, considering the fact that a magical infinite wall should really go for like $2.5 million, is a huge steal.

2) If you like the app, please give it a rating or review. For example, here’s my review:

3) This is V1 of the app. As we made the app, we had a constant side conversation going on about possibilities for V2, V3, and beyond. We’re launching today, but this is just the beginning. We want this app to be a brain playground that just keeps getting better and better. And here’s where you come in.

As you use the app, whenever you have an “I wish the app did X” or “It would be so cool if the app could do Y” thought, we want to hear it. As this is V1, we of course also want to know whenever you have a “This isn’t working right” or an “I think this fact is off” thought. In the app’s menu, you’ll see a “feedback” tab, where you can tap the feedback email address and let us know what you’re thinking.

Okay, here you go. Click the left app for iPhone/iPad, the right one for Android.

___________

Also: To commemorate the app launch, Kurzgesagt did their own fun room creation today—a delightful video about the sizes of stars.

___________

End things:

If you like Wait But Why, sign up for our unannoying-I-promise email list and we’ll send you new posts when they come out.

To support Wait But Why, visit our Patreon page.

___________

More trips to the fun room:

The Fermi Paradox

The SpaceX Post

The Quadrillion Sour Patch Kids Post

And a trip to the not-so-fun room


  1. Kurzgesagt did an updated version of the video in 2018.↩

The post The Big and the Small appeared first on Wait But Why.

You Won’t Believe My Morning

By: Tim Urban
25 March 2020 at 21:50

You won’t believe my morning.

I went out on my daily excursion to sit on the front step of my building for ten minutes holding my breath when people walked by. Normally, I spend the time diddling around my phone, but I forgot to bring my phone this morning, so I just looked around.

As I was taking in the emptiness of the street, a little glint caught my eye in a patch of dirt on the sidewalk. I bent over to look closer, and there was the glint again. It wasn’t a normal glint like from a shiny rock or a piece of metal—it was a little pinprick of flashing light.

Intrigued, I was now on all fours looking closer. And I saw the most surreal thing.

Tiny houses.

Like tiny houses. Each about a millimeter high, like ornately carved grains of sand.

I was either dreaming or looking at the coolest, cutest little art project ever.

As I examined the microscopic village, I noticed what looked like a scrawl of teeny letters on the dirt next to the houses. It said:

PUT YOUR THUMB ON THE OVAL

Now fully having the time of my life, I looked around for an oval. I searched for a few minutes with no luck until I saw, a bit outside the area where the houses were, a little strip of silver, maybe two millimeters long and a millimeter wide. Careful not to damage the houses, I put my thumb on it.

I won’t be able to accurately describe what happened next, but I’ll try my best. Imagine if the ground underneath you suddenly felt like a furiously spinning liquid whirlpool, combined with the feeling of freefalling, combined with your entire visual field turning into a blurry gray, combined with the worst nausea of your life.

And then, just as fast as it started, it all stopped. I cowered for a few seconds trying to catch my breath, and when I opened my eyes, I wasn’t in New York anymore.

I was on a pastoral ranch, surrounded by big log cabins and a bunch of people staring down at me as I looked up at them, still on all fours. One of the people said to me, “Are you okay?”

“I feel okay, but I’m having severe hallucinations.”

They all started cheering and hugging and high-fiving each other.

“Are you doctors?” I asked.

“He thinks he has the virus,” one of them said, and they all roared with laughter.

A woman shushed the crowd and said, “Okay, back to work everyone. I’ll give him the briefing.”

The others left, and the woman smiled at me. “I’m Layla.”

“Hi Layla. Do I have coronavirus?” I asked.

“You don’t have coronavirus,” she said. “You’re just tiny. We shrunk you to 1/10,000th of your normal size. You haven’t moved anywhere, you’ve just gotten much smaller.”

“Fuck,” I explained.

“Yeah, I imagine it’s a lot to take in,” she said. “Let me try to clear things up. There are different tiers of human life, not just the one you’re used to. Our tier is exactly 1/10,000th the size of yours. In yours, people are about 180 centimeters tall. Here, we’re about 0.18 millimeters tall. We’d say 180 micrometers tall, but I know in your world, micrometers don’t mean very much.”

I stared at her. “You’re telling me I’m 0.18 millimeters tall right now?”

“Correct,” she said. “About half the size of a dust mite, or a little taller than the width of a human hair up in your world. A person in your world with really good eyesight could barely see you, if they looked closely. And see that house over there?” She pointed to a large, three-story house. “In the scale you’re used to, that would be about 10 meters high. Here, it’s about the size of a cubic millimeter—the size of a grain of sand on the sidewalk. Some of our houses were actually carved from grains of sand.”

“Hold on.” I stopped her. “I’m very scared of bugs. I wrote a whole post about it once. Are there giant bugs here?”

“Yes and no. There are no bugs in our village because we lined the perimeter with a poison that kills any insects that come too close. But you wouldn’t want to walk too far away from the houses—about three centimeters from here, you’ll cross that barrier and run into things you won’t like.”

“How about flying bugs?” I asked diligently.

“Oh, I haven’t mentioned time yet. Okay so time here moves 100 times faster than it does up on your tier. Time scales up inversely with the square root of the size difference. So 1/10,000th of the size means 100X faster time. So when a flying insect starts to descend into the village, our defense crew has over a minute here to handle the situation. They fire a jet of air at the insect that deflects it away from us. Same deal for dogs. Every year or two, a dog will pee on us. The defense crew keeps track of every dog walking by. At 1/100th the speed of our world, we first see an approaching dog about ten minutes before it gets near us, and by the time a peeing dog is lifting its leg over us, we’ve had plenty of time to draw the tarps, which roll over the entire village and cover everything—the same tarps we use every time it rains or snows.”

“Good to know. But why am I here?”

“Right, I was about to get to that. After a heated debate in the village, we voted to bring someone from your world here, because we wanted to show you something. We’ve been trying to catch someone’s eye in your world for three of your weeks. That’s almost six years here. That’s why everyone was so excited to see you.”

“How did you catch my attention?”

“With that.” She pointed at a tube on the roof of one of the houses that looked like large telescope. “That’s a super-powered laser that we’ve been trying to shine in people’s eyes as they walked by on the sidewalk. But no one noticed. Until today. Way out on the horizon, we piled boulders into the shapes of letters that spelled out the instructions, and you were dicking around just hard enough that you actually bent down to read them. We’re thankful.”

“What was that oval in the instructions?”

“Look down.”

I was standing on a metallic oval about the size of a swimming pool.

“This is our trans-tier station. When you touched this with your thumb, it shrunk you down to our tier.

“Wait.” I looked up at her. “Do I live here now?”

She laughed. “Don’t worry. We’ll send you back to your world in a little bit. Now come with me.”

I walked with Layla towards the other side of the village. I looked around. Everyone was staring at me. Some people waved.

“What’s that?” I asked Layla, pointing up at what looked like a sheet of color stretching into the sky.

“That’s your apartment building. All you can see from here is the first brick. That band of light way up in the sky is the mortar between the first and second brick in the wall.”

We walked into a small building and entered a room with a long white table in the middle. Layla turned towards a wall on the side of the room, and suddenly the outline of a square formed in the wall. The square moved outwards into the room and rotated downwards until it was parallel to the floor. Layla pulled a tweezer-like tool from her pocket and carefully pinched what looked like nothing a few centimeters above the panel.

She walked over to me. “Open your palm.”

She put her tweezers into my hand and dropped something the size of a grain of sand. I raised up my palm to look at it. It was reddish and fuzzy.

“What is it?” I asked.

“SARS-CoV-2. What you’ve been calling ‘coronavirus.’”

I flung it across the room. “What the fuck.”

Layla laughed, touched the square, and the little object flew back across the room into its spot above the square.

“It can’t hurt you here. The viruses of your world are way too big to do any damage in our bodies.”

I stared at her, trying to process the situation. “How do you know about coronavirus? And how did you…get one?”

“Oh we know everything about your world. Your tier moves so slowly compared to ours that your technology is eons behind ours. Our tools have allowed us to watch your world since your prehistoric days. As for how we got a coronavirus particle, we didn’t get it—we made it.”

“You made the coronavirus?”

“Well, with a little help. Follow me.”

Again I followed Layla, again dumbfounded, this time out of the building toward a fenced-in area outside. When we got close, a door in the fence opened and inside, we stood together on the edge of what looked like a barren circle of land.

Layla opened her hand. The little virus was sitting neatly on her palm. “SARS-CoV-2 is a standard sized Coronavirus—about 120 nanometers in diameter. 120 nanometers is ridiculously small in your world, but in ours, you can roll it around in your fingers.

“Cool…let me think about that for a minute.”

“Wait, I can explain this better. In your world, this is a perfect size analogy:

SARS-CoV-2 : grain of sand :: grain of sand : house

In both cases, the relationship is 1-to-10,000, which is also the relationship of our world to yours. In your world, your apartment building is huge, a grain of sand is tiny, and this coronavirus is unfathomably microscopic. In ours, the virus is tiny, a grain of your sand is huge enough to live in, and your apartment building is unfathomably large.”

“Makes sense.”

“Now, what’s a virus made of? Atoms. And atoms are about 0.1 nanometers in diameter—about 1/1,000th of the diameter of a SARS-CoV-2 particle. That’s small even for us. An atom is almost as small for us as a virus is for you. Constructing a virus requires incredibly complex engineering and tools that can interact with the quantum field. We can’t do it ourselves.”

“So how—”

“Look down.”

I looked at the ground.

She pulled me toward the center of the circle of land and then pulled me to an abrupt stop.

“Look closer.”

I bent over as far as I could and strained my eyes. No. Fucking. Way.

Another microscopic world.

“Is that…”

“Yup. That’s the tier below us. Give me your thumb.”

She carefully placed the virus particle onto the ground. Then she guided my hand to the ground, touching both of our thumbs to a little metallic spot.

Whirlpool. Falling. Gray. Nausea. Misery.

I eventually got a hold of my trembling and drooling and opened my eyes. In every direction, as far as I could see, stretched a hazy blue-ish / purple-ish plane. It also began to dawn on me that I wasn’t standing or sitting on anything—I was floating.

After about a minute of wondering what the hell was going on in my life, a patch of the sky darkened. The dark region became smaller and more defined until it condensed into Layla floating next to me.

“Please don’t leave me again,” I said.

“Sorry, my thumb hit the pad a split second later than yours did. Time moves 100 times faster here than up there, so you got to spend some reflective time here by yourself.”

“Okay where are we?”

“We’re outside your apartment building. Remember?”

“Righttt. So what size are we?”

“We went down the same jump you did when you transitioned from your world to ours—we shrunk to 1/10,000th of our previous size. So you’re now 18 nanometers tall. If you stood on the edge of a cross-section of a human hair, it would take you about two hours to walk across it.”

“Jesus.”

“Time here is now going by at 100X the speed of time in my world, which means it’s moving 10,000 times faster than the speed of time in yours. You could spend a year here and less than an hour would pass in your world.”

“Kind of like Inception?”

“Not really. Anyway, nothing down here works the way it does in our worlds. Like see how the ground is all eternal and purply?”

“Yeah.”

“That’s not really ground, and it’s not really purple. When you’re this small, there aren’t any solid objects in the sense you understand. And your eyes are too small now to perceive the visible light spectrum.”

“So what the hell?”

“I don’t really get it either. But the people who live here have incredibly advanced ways of manipulating the quantum field so we can feel like we’re intact humans, floating in place, seeing purple. They set it up this way because it’s something we can make sense of.”

“That’s nice of them. Where are they all anyway?”

“The thing is, they don’t like our world, and they really don’t like your world. They interact with us occasionally, when it’s necessary, but they’ll never allow you or anyone from your world to see them or know anything about how they live. This is actually the first time anyone from your world has been allowed down here, other than Andy Kaufman, who’s lived here since 1984.”

“Then why am I allowed to be here?”

“So I can show you this.”

Layla straightened her arm in front of her, with her palm facing outwards. Her palm lit up and when it did, a bizarre-looking giant object was revealed in front of us.

“This,” Layla said, “is SARS-CoV-2. Down here, it’s the size of a house.”

I looked up at the vast virus in front of me. It looked nothing like it had when it was a fuzzy grain of sand in my hand. It was transparent, like a massive, intricately structured, sphere-shaped jellyfish. There was a kind of furious movement within the transparency, but I couldn’t see anything specific moving. It was confusing.

Layla motioned for me to come close to the virus. She took my hand and placed my palm onto the virus’s almost invisible surface. It felt kind of like palming a grape bunch except instead of grapes it was tapioca balls like the ones in those bubble teas—if the tapioca balls were vibrating so vigorously that it felt a little like being mildly electroshocked, like when you put your finger in an electrical outlet. It was an unpleasant sensation, though not quite painful, and super weird and cool-feeling.

“Pull one off,” Layla said.

It took me a few tries to latch onto one of the atoms, because they’re “slippery” (quotes because there is no word for what it felt like, but “slippery” gets the general idea across), and when I finally got one and pulled, there was a lot of resistance. When I pulled it, it dragged the adjacent atoms along with it, and the harder I pulled, the more ferociously and unpleasantly it vibrated. Finally it snapped free. I looked at my fist—I had an atom.

Layla smiled. “Cool right?”

“So cool. Can I keep it?”

“Sure good luck with that.”

I was so awe-struck by so many things, I had forgotten how intensely confused I was.

“Wait, so why did you make this?”

She turned towards me. “As I said, because of the way time moves—”

“Your world has been around a lot longer than ours.”

“Yes. And this world we’re in now has been around a lot longer than mine. They know much more about everything than we do, and they can do things we can’t even begin to understand. The stuff they can do is so over our heads they can’t even explain it to us. And we’re that far ahead of your world.

For a long time, as advanced as this world was, it relied on us to preserve our world for its own survival. It exists on a patch of dirt in our village. If our village were destroyed, they would be destroyed with it. But a while back, they worked out the technology for how to be location independent, which means they can move from wherever they are to any other place in the universe instantly.

We don’t have that technology yet. We tried to learn from them, but we couldn’t grasp the fundamental ideas well enough to develop it ourselves. So we’re stuck in our location.”

“In New York City?”

“Yup. We migrated here in the 1800s when we determined it would be a good place to interface with your world, should the need ever arise. We’re also not the only people in our world. There are lots of villages like ours in different parts of the Earth. Once we connected with you, the others stopped broadcasting their location. There’s no reason for more than one of us to be revealed to your world.

For most of time, Earth was a safe and stable home for our world. But over the last century, your world has been advancing exponentially in technology but remaining stagnant in wisdom. You’re rapidly gaining tremendous powers but still behaving like short-sighted primates. The voice of wisdom is there, but it’s being trampled over by political parties, religions, and nations too mired in blind conflict to lift their heads up and see the bigger picture.”

“It’s funny you say that Layla. I’m actually writing a whole thing about—”

“Oh I know. We did our research on everyone who lives in your vicinity so we’d know how to communicate with the person we brought to us if we caught someone’s attention. That’s why I’m speaking English and speaking in the odd way you do. Your little series is cute, even if it took us forever to read—but it will have limited effect. Your world is stubborn about growing up. And in the process of destroying yourselves, we believe you’ll destroy us as well.”

“So you’re trying to kill us off with a pandemic.”

“If we wanted to kill you off, you’d all be dead right now. It’s an option we hope we don’t have to use. We were once like you and we empathize with your struggle.

We created this coronavirus to fall into a certain sweet spot—not damaging enough to destroy your world, but bad enough to cause a long and scary global crisis. Short of an alien attack, it is the one thing that could make all humans in your world feel like they’re on the same team against a common enemy. The first and most crucial step on the road to a long-lasting species is the epiphany that you truly are a single team, alone in a dark and dangerous universe. We’re hoping the virus can help push you in that direction.”

“I feel like there was a better way to do this.”

“We probably could have thought a little harder about it.”

“Yeah cause it’s going pretty badly up there is the thing.”

“Totes. Anyway, we couldn’t make the virus on our own. It’s hard enough to make something that small and complex that involves atomic and subatomic construction, but we wanted the virus to be precisely as harmful as it is. We needed help. The tier below us is less vulnerable to your world than we are, but for reasons I’m not entirely sure about, they also believe living in a multi-tiered ecosystem may be important in the future—so they share our interests. They agreed to build the virus for us.”

“How did you manage to get the virus into our world?” I asked.

“It’s funny. You have to imagine it from our perspective. If you’re us, the world you’re planning to transform is a planet with nearly an AU diameter, full of 18-kilometer tall people—people so tall, your world’s airplanes could accidentally fly into their belly buttons. Now imagine you’re standing on that planet, smaller than one of their dust mites, pinching between your fingers something the size of a grain of sand on your scale. You find your way onto one of these giants’ football-field-size teeth, and you flick the grain of sand into his kilometer-wide chasm of a mouth. And that’s supposed to change the trajectory of their future. It seems impossible.

“And yet.”

“And yet. With some very clever maneuvers, we flicked our little particle into the mouth of an unlucky giant, and it did the trick. By the way, we were dying when you blamed it on a pangolin of all things.”

“He seemed guilty. I still don’t understand why you brought me down here.”

“We weren’t originally planning to reveal any of this backstory to your world. But after watching things unfold for the first few weeks, we don’t see enough of the effect we were hoping for. Maybe if your world learns that there are other worlds out there—worlds that did manage to triumph in the wisdom game—it’ll empower the wise voices to stand up with a bit more courage in this struggle and in even greater challenges that lie ahead. It’s a long shot, but these are desperate times.”

“I guess it’s worth a try. I’ll write a post explaining what I learned from you.”

“And tweet out the post and send it out to your email list and stuff?”

“Eh. Fine.”

“Ready to go back up to your life? Only 23 seconds have passed there since you shrunk down.”

“Yeah let’s do it.”

Layla and I moved ourselves on top of the big metal oval.

“I’m going to increase my size 10,000-fold and yours 100,000,000-fold, so you can go back home in one shot…which is a good thing because transporting up is even worse than transporting down.”

“Great…”

“Ready?”

“I have one more question.”

“Yes?”

“How many tiers are there?” I asked.

“No one knows for sure. The people on this atomic-level tier tell us they know of at least one tier below them, though they won’t tell us more than that. And no one seems to know about tiers above yours. Your world is thinking about that, with all your multiverse talk. We’re still working on that one too.”

“If you ever figure it out, will you let me know?”

“One thing at a time.”

We touched our thumbs to the metal.

___________

If you like Wait But Why, sign up for the email list and we’ll send you new posts right when they come out.

To support Wait But Why, visit our Patreon page.

More Posts:

As discussed, why bugs ruin everything.

Another time we transported ourselves.

And a puzzle for your quarantine.

The post You Won’t Believe My Morning appeared first on Wait But Why.

❌
❌