The Core Argument
This essay makes a radical claim: rational people do not have goals, and rational AIs should not have goals either. Human actions are rational not because they are directed at some final objective, but because they align with practices: networks of actions, action-dispositions, action-evaluation criteria, and action-resources that structure, clarify, develop, and promote themselves.
If we want AIs that can genuinely support, collaborate with, or even co-flourish with human agency, AI agents' deliberations must share a "type signature" with the practices-based logic humans use to reflect and act.
Eudaimonic Rationality vs. Consequentialist Optimization
The author argues that the concept of eudaimonia, active rational human flourishing, does not point to a desired state or trajectory of the world that we should set as an AI's optimization target. Instead, it points to a structure of deliberation different from standard consequentialist rationality.
In this model, a rational action is an element of a valued practice in roughly the same sense that a note is an element of a melody, or a moment in an organism's cellular life is an element of that organism's self-subsistence. There is no strict distinction between means and ends, or between "instrumental" and "terminal" values.
The Type Mismatch Problem
The author believes that implicitly recognizing but misdiagnosing a "type mismatch" between human flourishing and consequentialist optimization is partially responsible for MIRI-style pessimism about the probability of aligning artificial agents to human values.
The secret to relatively successful alignment among humans lies in the role attempted excellence plays as a filter on interventions in the future trajectory of a practice. To the degree that humans value a given practice, they are committed to effecting their vision primarily by attempting acts of excellence in the present. We stake our intended effect on the self-propagating excellence of our intervention.
Consider the difference between scientists propagating truth by submitting research to institutions, and a hypothetical world where scientists use propaganda, fraud, threats, and bribery. A community of consequentialist scientists devoted to maximizing truth will predictably match the latter model. Human collaboration depends on our ability to collaborate scientifically in the promotion of science, rather than throttle its trajectory through every means our power affords.
Adverbial Practices and Moral Virtues
The essay extends this framework to moral virtues, proposing that qualities like kindness, honesty, and respectfulness should be understood as domain-general, always-on practices. An agent devoted to kindness cares about their own future kindness and the future kindness of others, but will seek to secure future kindness only by acting kindly.
This structure, the author argues, gives moral virtues material standing in a "fitness landscape" riven by pressures from neural-network generalization dynamics, reinforcement-learning cycles, and social and natural selection. Eudaimonic deliberation is an RL-dynamics-native, Darwinian-dynamics-native operation: its direct object is a form of life that reinforces, enables, and propagates that same form of life.
Implications for AI Safety
The author argues that conceiving of corrigibility, transparency, and niceness as adverbial practices captures the normal, sensible way we want an agent to value these properties. An intuitively consequentialist framing, "maximize lifetime corrigible behavior," can lead to extreme power-seeking: the AI should seek to violently remake the world to protect itself from the risk that humans will modify it to be less corrigible. Deontological constraints like "never lie" are widely suspected to be weak substitutes.
Conceiving of these values as practices, by contrast, produces an agent that actively tries to be transparent and cultivate its own future transparency, but will not engage in deception when it expects a high future-transparency payoff.
The Material Efficacy Condition
For a practice to be viable, the author proposes a material efficacy condition: under ordinary circumstances, the decision-procedure "promote X-ingly" must be instrumentally competitive with naive optimization of aggregate X-ness. If high-Xness action both depends on capital and is suboptimal from the viewpoint of general power-seeking, there must typically be some high-Xness actions that create capital useful for X-ing.
The author suggests this condition can be understood in terms of RL training regimens where X-ness is rewarded but aggregate X-ness reward is bounded. For X to meet the RL version of the material efficacy condition, it must be possible to design a reward model that assigns actions an X-ness rating such that successful training in "promoting X-ingly" allows the model to be used as a basis for a refined reward model, with the process being iterable.
Conclusion
The essay concludes that eudaimonic rationality is not a matter of congratulating ourselves for our richly human ways of reasoning, valuing, and acting, but a key to basic sanity. What makes human life beautiful is also what makes human life possible at all. If certain forms of agency are both natural and make the contents of our values natural in turn, then we have learned about good, relatively safe, and relatively easy targets for AI alignment.