Evolutionary Game Theory in Machine Learning and AI: Naturality, Regret, and Recurrence
Learners as open systems, games as wirings, and what survives the duplication of an action
The replicator equation of evolutionary game theory and the standard learning rules of machine learning are the same objects under different names. Multiplicative weights is the replicator flow sampled in time; natural policy gradient on a softmax policy is the replicator field; Bayesian updating is the replicator map with log-likelihood as payoff. This paper organizes those identifications around one structure and attaches a constant to each qualitative statement. It is the application paper to the companion volume Evolutionary Game Theory in Higher Categories, which treats J. McKenzie Alexander's Evolutionary Game Theory (Cambridge Elements, 2023).
Format. Double-column formal paper, 9 pages, 46 references; status tags on every numbered statement (classical, literature, reformulation, observation, programme); a companion script with 21 computational checks.
The structure
A learner is an open dynamical system: payoffs $\pi\in\mathbb R^S$ in, a state $z$, a mixed strategy $x=r(z)\in\Delta(S)$ out. A game is a wiring of learners: self-play feeds back $\pi=A\,r(z)$; two players are wired crosswise; a league or a population on a graph is a larger diagram. A family of learners is natural if its closed-loop dynamics commutes with payoff-preserving maps $f\colon(S,A)\to(T,B)$ of games:
$$\begin{CD} \Delta(S) @>{\text{learn in }A}>> \Delta(S) \\ @V{f_*}VV @VV{f_*}V \\ \Delta(T) @>>{\text{learn in }B}> \Delta(T) \end{CD}$$
For clone maps this says that listing an action twice, or splitting it into indistinguishable variants, changes nothing about the total weight of the variants. Large action spaces contain near-duplicates (synonymous tokens, equivalent moves, redundant members of a league), so the condition has content.
Results: qualitative statement, then its quantitative form
| Topic | Qualitative | Quantitative |
|---|---|---|
| Hedge and replicator | one is the other, sampled | exact at every step size |
| Softmax policy gradient | replicator with one extra factor $x_i$ | time to escape a bad start $\sim 1/(2D\epsilon)$ against $\log(1/\epsilon)/D$: ratios $3.0$, $11.8$, $73.3$ at $\epsilon=0.1,0.01,0.001$ |
| Duplicated actions | replicator and Hedge natural; policy gradient and Boltzmann $Q$-learning not | splitting an action into $k$ copies multiplies policy-gradient learning time by exactly $2k/(k+1)$; replicator time unchanged |
| Regret | no regret | at most $\ln n$ in continuous time for every horizon, by an identity; at most $\sqrt{T\ln n/2}$ in discrete time |
| Zero-sum games, flow | relative entropy to equilibrium conserved; recurrence | — |
| Zero-sum games, Hedge | iterate spirals out, time average converges | entropy rises by exactly $\log\mathbb E\,e^{\eta f}\approx\tfrac12\eta^2\mathrm{Var}(f)$ per step; boundary reached after $1451$ steps at $\eta=0.1$ |
| Optimism | last iterate converges | $0.070\to1.5\times10^{-10}$ in $6000$ steps |
| Entropy regularization | converges; biased; not natural | $\frac{d}{dt}D(u\,\Vert\,x)=-\alpha\,(D(u\,\Vert\,x)+D(x\,\Vert\,u))$, so rate at least $\alpha$; with Rock duplicated the regularized equilibrium puts $(0.40,0.36,0.24)$ on the three types at $\tau=0.5$ |
| Evaluating agents | ratings plus cycles plus a harmonic part | planted example: ratings recovered, transitivity index $0.885$, curl $1.5$ |
| Redundant agents | score averaging and a name-based mutation chain not natural; replicator time average natural | mean scores $(0,0,0.25,-0.25)$; chain $(0.4,0.4,0.2)$ on types; replicator $(\tfrac13,\tfrac13,\tfrac13)$ |
| New conventions in a finite population | can invade by drift | more likely than neutral iff the barrier is below $\tfrac13$: $N\rho=1.021,1.009,0.991,0.958$ |
| Copying topology | amplifiers and suppressors | line $0.167$ < well-mixed $0.251$ < star $0.287$ ($N=6$, $r=1.2$) |
| Inference | Bayes is Hedge at $\eta=1$ | exact |
Three points the organization makes visible
One term, two roles. The excess $\Lambda_\eta=\log\mathbb E_x e^{\eta(\pi-x\cdot\pi)}$ from holding the payoff fixed for a step is the whole difference between regret $\ln n$ and regret $\sqrt T$, and in a zero-sum game it is the energy pumped into the cycle at every step.
Three repairs act in three places. Averaging changes the readout; optimism changes the box; entropy regularization changes the wiring, adding a loop from each learner's output to its own input. Averaging keeps naturality and loses the iterate. Regularization gains a rate and loses both naturality and the equilibrium.
Naturality sorts methods. The same test, commuting with clone maps, separates replicator from policy gradient, Nash-based evaluation from score averaging, and entropy on types from entropy on names. In a strictly feed-forward copying topology, by contrast, the game does not matter at all: fixation is $1/N$ whatever the fitness.
A remark relates the framework to recent work on populations of agents built on large language models (replicator dynamics over agent types, evolutionarily stable trust mechanisms, group selection on prompts). The titles and identifiers of five such items were checked online on 17 November 2026; their results were not evaluated. The paper's contribution there is a set of constraints: prompts that induce the same behaviour are clones; stability should be asked of sets and not of single prompts; and conclusions about invasion by one new agent need the finite-population correction.
What is and is not claimed
No new theorem of learning theory. The identifications, regret bounds, recurrence, Hodge decomposition, one-third law and isothermal theorem are from the literature and are cited. The observations are: the naturality theorem for learners with its exact slowdown factor; the entropy-regularization identity with its rate; the naturality table for evaluation methods; and the placing of the three repairs. They are short, and parts are likely known in other forms; invariance to redundant agents, for one, is a stated aim of Nash averaging (Balduzzi et al.). The statement about the mutation chain concerns the kernel of the paper's own small implementation and makes no claim about $\alpha$-Rank as published or as used.
Scope: everything is tabular, with at most six actions and populations of a few hundred, chosen to be exactly computable. Function approximation, sampling noise and partial observation are outside it. No experiment on a trained model was run; the numbers illustrate the statements and are not benchmarks.
Files
evolutionary-game-theory-in-machine-learning-and-ai.pdf— the paper (9 pages, two columns).evolutionary-game-theory-in-machine-learning-and-ai.tex— complete LaTeX source as a single file (pdflatex, three passes).verify-evolutionary-learning.py— 21 checks, Python standard library only, under half a minute; exits 0 if and only if all pass.
Status and authorship
Contribution. Organization of known results around a naturality criterion, with exact constants, a small number of observations, and reproducible computations.
Limitations. As listed above; in addition, the bibliography was assembled from memory except for seven entries whose details were checked online, proofs marked as sketches are sketches, and nothing has been independently refereed. The script's checks are evidence that the numbers are stated correctly and are not proofs. Corrections are welcome.
AI assistance. The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall, mathematical consistency checks, computation and typesetting.