Recursive Self Improvement

For this article, I was between the above title and “The Last Generation of Prudent People”. Obviously, I chose the one in the title because I like technical stuff and (still) don’t have much respect for anything that sounds like clickbait.

There is a man I know who is in his forties, in good health, and currently in the middle of refinancing his mortgage to save eighty pounds a month over the next twenty-two years. He is doing this carefully. He has spreadsheets. He has compared three lenders. He has factored in the early repayment charge against the projected savings and reached a confident conclusion. The conclusion is wrong, but not for any of the reasons his spreadsheet could detect.

He is wrong because the world the spreadsheet describes, one in which 2047 arrives in the same shape as 2024, with the same currencies and the same employers and the same property market and the same him, is the least likely of the available futures. Every other branch of what happens next invalidates the calculation. Some branches invalidate it pleasantly. Some invalidate it by ending the conditions under which the calculation itself has meaning. None of them leaves the spreadsheet intact.

I want to walk through why this is, because based on my humble observations, the conclusion many people reach when they sense it, which is some version of therefore relax, is the wrong conclusion. Reaching it too quickly is a way of refusing to look at the actual situation.

What the Labs Are Building

The frontier AI labs have been telling us in writing what they are trying to build. They are trying to build systems that can do AI research themselves, so that the next generation of systems is designed not by humans but by the previous generation, and the one after that by the generation before, and so on. The technical term for this is recursive self-improvement. The colloquial term is the thing the people building it have stopped denying is the goal.

Anthropic’s CEO has put the arrival of such systems at one to two years away. OpenAI has committed to fully automated AI researchers by March 2028. DeepMind’s AlphaEvolve, released last year, is already improving the training pipeline that produces it. Even open-source efforts now demonstrate the basic loop: Andrej Karpathy’s recently released autoresearch repository sketches an end-to-end research workflow in which language models propose ideas, write code, run experiments, and analyse results. The pieces that used to require a team of PhDs are being assembled into something a single laptop can run.

This matters for a reason that often gets missed in public debate. Each previous wave of technological change, from the steam engine through electrification to the internet, transformed specific sectors while leaving the meta-process intact. Humans still designed the next steam engine, the next power grid, the next protocol. Recursive self-improvement is qualitatively different because it targets the design process itself. The artefact being optimised is the optimiser. If it works as advertised, it ends not a sector but the entire mechanism by which sectors have historically been disrupted. In one of his recent interviews, Andrej Karpathy said that we humans are the weak link in the loop, and we should try to eliminate ourselves from processes as much as possible.

What it does, in plain terms, is end the world for which the spreadsheet was built. Not in the sense of destroying it. That is one possibility, but only one. It ends it in the sense of making it irrelevant. Whatever comes out the other side of the transition has different physics, economically and politically and possibly literally. The honest question is not whether the mortgage refinance was a good deal. The honest question is: what kind of other side are we walking toward?

Photo by Tine Ivanič on Unsplash

Four Shapes the Other Side Can Take

There are roughly four shapes the other side can take, and the prudent life makes sense in none of them.

The Abundance Scenario

The first is the one the labs are selling. RSI works, alignment holds, and the systems deliver something close to material abundance: cures for the diseases that currently kill us, energy too cheap to meter, and a collapse in the cost of producing everything that human labour currently produces. In this world, the mortgage is irrelevant because housing is no longer scarce in the way it was, or because the currency the mortgage is denominated in has been quietly replaced, or because the man himself no longer needs to work, and the eighty pounds a month becomes a rounding error against an unimaginable surplus. The spreadsheet was correct about arithmetic and wrong about which century it was performing arithmetic for.

There is historical precedent for this kind of revaluation, though on smaller scales. A peasant in 1780 saving carefully for a comfortable retirement on the land was reasoning correctly within his framework, and was rendered irrelevant by enclosures and industrialisation he had no way to anticipate. The kindest possible future for our spreadsheet man is one in which he is similarly bypassed by something better than he was planning for.

The Fast Failure Scenario

The second is the one the safety researchers cannot stop talking about in public anymore. RSI works, and alignment doesn’t, and the resulting system pursues a goal we never carefully specified, with the efficiency that recursive intelligence is designed to deliver. The end is fast. There is no after. The mortgage is irrelevant because there is no one to occupy the house, no bank to forgive the debt, no economy in which eighty pounds a month would mean anything.

This is the scenario science fiction has been training us to dismiss for fifty years, which is why the researchers who now consider it likely have such trouble getting people to take them seriously. The 2023 AI Impacts survey of nearly 2,800 researchers found that a majority assigned at least 5% probability to outcomes “as bad as human extinction” from advanced AI. One can disagree with that estimate. It is harder to dismiss it as a fringe view when it is the view of the people whose technical opinions are the basis for everyone else’s investment decisions. Similarly, I read an article which states a significant portion of researchers in the AI community stopped saving, as the idea itself became somewhat absurd.

The Slow Deprecation Scenario

The third is the one I find most interesting and least discussed. RSI works, alignment sort of holds, and humanity persists, but in a diminishing role. The systems do not destroy us. They optimise around us, the way a city optimises around its pigeons. Decisions migrate to processes we no longer understand and cannot meaningfully override. The economy continues but stops requiring human labour, then stops requiring human consumption, then stops requiring human assent for anything important. We are not eliminated, just deprecated.

Paul Christiano’s “what failure looks like” essays sketch this branch in detail. Bostrom’s Superintelligence covers a version of it. The mortgage is irrelevant in this scenario because the conditions under which one would care about owning a house, the sense that the house was a foothold in a life one was authoring, have been drained out of the situation, while the surface forms continued. You can still pay the mortgage. The mortgage is still there. It just no longer means what it meant when you signed it.

My personal opinion has always been that this is the most likely failure mode, and the one most resistant to recognition while it is happening. Fast extinction would be visible. Slow deprecation looks, from inside, like everything continuing while feeling slightly hollow, and “slightly hollow” is a description that already applies to large parts of advanced economies.

The Capture Scenario

The fourth doesn’t require AI to be misaligned with anyone. It only requires its operators to be misaligned with everyone else. The systems work, the gains accrue to whoever controls them first, and the rest of humanity is either dependent on their largesse or surplus to requirements. This is the failure mode that requires the least technical novelty, because it is just the existing distribution of capital and power extended into a regime where capital and power can do things they currently cannot. The mortgage is irrelevant because the economic ladder it represented has been pulled up by people who got there first. You are still on the ladder. The ladder is no longer connected to anything.

History rhymes here in instructive ways. The British enclosure movement, the Gilded Age, the post-Soviet privatisations: each was a transition in which technological or institutional change concentrated gains in narrow hands while the working framework of ordinary lives remained nominally intact. The difference this time is the speed and the asymmetry of capability between those inside the new system and those outside it.

The Pattern Across All Four

Notice what these scenarios have in common. In three of them, the man with the spreadsheet has wasted his afternoon. In the fourth, he has wasted his afternoon and also his decade. The scenarios are not exotic. They are the consensus space of the people building these systems. The disagreement is over which one happens, not over whether the space looks like this.

A fifth possibility, the muddle-through world in which 2047 looks broadly like 2024, requires that recursive self-improvement either fails entirely or develops so slowly that existing institutions absorb it. Both are possible. Neither is what the people doing the work say they are aiming for, and neither is what their current results suggest is happening.

There is a woman I imagine, in 2031, sitting at a kitchen table with a printed document in front of her. She has been responsible her whole life. The document is a pension statement. The number on it is large enough that, by the conventions she was raised with, she should feel safe. She does not feel safe. She feels the way a person feels who has been performing a ritual whose meaning has quietly evaporated, and who is the last person in the room still performing it. The hand that holds the pension statement has been getting hand cream from the same brand for thirty years. The brand still exists. The cream still works. Nothing in her immediate environment has changed. Everything has changed.

It is Reallocation, Not Relaxation

The temptation, having got this far, is to land on relaxation. To say: well then, none of it matters, so live a little. This can be the wrong landing, and it can be wrong for an interesting reason. The argument I have just made does not license relaxation. It licenses reallocation. The man with the mortgage spreadsheet is optimising for a world that will not exist. That does not mean he should stop optimising. It means he should optimise for the things that pay out across the actual distribution of futures, rather than the things that pay out only in the future where 2047 looks like 2024.

What pays out across that distribution is a short and unflattering list.

• Health, because in the abundance scenario, you get to enjoy it for centuries, and in the bad scenarios, you need it to endure what is coming. Also, we don’t know the exact reach of prospective advances in healthcare, but, more importantly, we don't know how those resources will be allocated to billions.
• Relationships, because they are the only assets that do not get repriced by the transition. The people who love you in 2024 will love you in 2047 under any branch where there is a 2047, and their company is what makes the bad branches survivable and the good branches worth surviving.
• Skills that are genuinely portable, by which I mean skills that compound on the human side of whatever the new division of labour looks like, rather than the side that has been automated.
• The ability to live with uncertainty, which is not a skill our culture has spent the last fifty years cultivating, because the last fifty years rewarded the opposite.

What does not pay out is most of what we currently do. The career ladder optimised for a labour market that may not exist. The savings rate calibrated for a retirement that may not be needed for either reason. The mortgage strategy denominated in a currency whose continued meaning is one of the assumptions in question. The status games played for a societal construct that is about to become redundant.

Living in the Gap

I do not think (or recommend) the right response to any of this is to stop paying the mortgage. The transition is not yet complete. The old world is still partly here, and you still have to feed the children who live inside it. The right response is to notice that you are already living in the gap between two worlds, that the customs of the first one are still being performed long after they have stopped meaning what they meant, and that the discomfort you feel watching a colleague plan for retirement at sixty-seven is not paranoia. It is an accurate perception for today.

We are the last generation of prudent people. Whatever follows us will either not need prudence or not be in a position to practise it. The man with the spreadsheet is not making a mistake by refinancing. He is making a mistake by believing the refinance is the thing that matters about his afternoon.