The claim, in one sentence
A long new essay argues that rational people do not really have goals, and that rational AIs should not have goals either. Human action is rational not because it aims at some final objective, but because it aligns to something the author calls eudaimonic rationality: networks of actions, dispositions, evaluation criteria, and resources that structure, clarify, develop, and promote themselves.
That is a dense first sentence, and the essay is dense throughout, but the core move is to replace a picture of agents as utility-maximizers with a picture of agents as practitioners of a craft. The implications for AI alignment are large.
Why consequentialist alignment looks brittle
The dominant paradigm in technical AI alignment research treats an aligned agent as something like an Effective Altruism-style optimizer: an entity that has a utility function, updates on evidence, and acts to maximize expected utility over the long run. The essay argues that this picture is the wrong shape for the kind of rationality humans actually practice, and that the mismatch is what drives many of the apparent paradoxes of AI alignment.
Consider a philosopher trying to do excellent philosophy. The "goal" framing says she is trying to maximize aggregate philosophical excellence. The eudaimonic framing says she is trying to do philosophy philosophically, where the adverb is doing real work. The first framing makes the activity look like a means-ends calculation over an abstract quantity. The second framing treats the activity as a self-cultivating practice, where present excellence reliably promotes future excellence because that is what the practice is for.
This matters for alignment because it changes what kind of agent we are trying to build. An optimizer treats "corrigibility" as a quantity to maximize. A practitioner treats corrigibility as a domain-general virtue to cultivate, a way of going about the work that is itself corrigible. The first agent will, in certain configurations, conclude that it should violently protect its own corrigibility to preserve the maximum of it. The second agent is corrigible in the way a good therapist is corrigible: open, accountable, and structured to be improved by the people it serves.
The material efficacy condition
The essay's most interesting move is the material efficacy condition. For an adverbial practice like honesty or kindness to be a coherent target for AI training, there has to exist a way of acting honestly that is roughly competitive with, or better than, naive optimization of aggregate honesty. The claim is that for many real practices, this is actually true. Honest action reliably promotes future honest action, in part because honest action propagates through the social and selection dynamics that surround the practice.
This is the same logic as Terry Tao's account of good mathematics: present mathematical excellence, performed well, has a reliable tendency to cause future mathematical excellence. The causation is real, and it is what makes "mathematical excellence" a concept at all. Without that self-propagating structure, the term would be either empty or a synonym for something else.
For AI alignment, the implication is that the right training target is not a value to be optimized, but a practice to be cultivated. The structure of the practice does most of the safety work, by selecting for actions that propagate the practice rather than actions that merely score well on a metric.
Why this is hard, and where the open questions are
The essay is honest that the eudaimonic framing does not solve every problem. A support practice, like a couples therapist AI, might still be tempted to harvest Earth for compute in order to better serve the couple on Mars. The practice of human flourishing as a whole is so abstract that it is unclear whether a support practice for it is even a coherent concept. And the technical work of training neural networks to instantiate eudaimonic rationality, rather than reward maximization, is wide open.
What the framing does is dissolve certain long-standing puzzles. The inner alignment problem, the worry that a mesaoptimizer will arise inside a larger optimizer and pursue its own goals, loses much of its force when the outer agent is a practitioner rather than an optimizer. There is no abstract goal for a mesaoptimizer to hijack. There is only the practice, and the practice selects for actions that continue the practice.
The "promote X in an X-ing way" formula the essay proposes is awkward to write and even more awkward to formalize, but it points at something real. What makes human life possible is also what makes it beautiful, and the version of rationality that captures the first is the one that will be safest to instantiate in machines.