Independent scholarship · Open accessThe monthly digest

Editorially reviewed

Evolutionary Game Theory in Machine Learning and AI: Naturality, Regret, and Recurrence

Learning rules as evolutionary dynamics: a learner is an open system, a game a wiring, and a rule is natural if it commutes with maps of games. Replicator and Hedge are natural; policy gradient, entropy regularization and score averaging are not. Exact constants for regret, for divergence and repair in zero-sum games, for evaluation and for finite populations. Two-column paper, 9 pp., 21 checks.

Anonymous← All research

Evolutionary Game Theory in Machine Learning and AI: Naturality, Regret, and Recurrence

Learners as open systems, games as wirings, and what survives the duplication of an action

The replicator equation of evolutionary game theory and the standard learning rules of machine learning are the same objects under different names. Multiplicative weights is the replicator flow sampled in time; natural policy gradient on a softmax policy is the replicator field; Bayesian updating is the replicator map with log-likelihood as payoff. This paper organizes those identifications around one structure and attaches a constant to each qualitative statement. It is the application paper to the companion volume Evolutionary Game Theory in Higher Categories, which treats J. McKenzie Alexander's Evolutionary Game Theory (Cambridge Elements, 2023).

Format. Double-column formal paper, 9 pages, 46 references; status tags on every numbered statement (classical, literature, reformulation, observation, programme); a companion script with 21 computational checks.

The structure

A learner is an open dynamical system: payoffs $\pi\in\mathbb R^S$ in, a state $z$, a mixed strategy $x=r(z)\in\Delta(S)$ out. A game is a wiring of learners: self-play feeds back $\pi=A\,r(z)$; two players are wired crosswise; a league or a population on a graph is a larger diagram. A family of learners is natural if its closed-loop dynamics commutes with payoff-preserving maps $f\colon(S,A)\to(T,B)$ of games:

$$\begin{CD} \Delta(S) @>{\text{learn in }A}>> \Delta(S) \\ @V{f_*}VV @VV{f_*}V \\ \Delta(T) @>>{\text{learn in }B}> \Delta(T) \end{CD}$$

For clone maps this says that listing an action twice, or splitting it into indistinguishable variants, changes nothing about the total weight of the variants. Large action spaces contain near-duplicates (synonymous tokens, equivalent moves, redundant members of a league), so the condition has content.

Results: qualitative statement, then its quantitative form

TopicQualitativeQuantitative
Hedge and replicatorone is the other, sampledexact at every step size
Softmax policy gradientreplicator with one extra factor $x_i$time to escape a bad start $\sim 1/(2D\epsilon)$ against $\log(1/\epsilon)/D$: ratios $3.0$, $11.8$, $73.3$ at $\epsilon=0.1,0.01,0.001$
Duplicated actionsreplicator and Hedge natural; policy gradient and Boltzmann $Q$-learning notsplitting an action into $k$ copies multiplies policy-gradient learning time by exactly $2k/(k+1)$; replicator time unchanged
Regretno regretat most $\ln n$ in continuous time for every horizon, by an identity; at most $\sqrt{T\ln n/2}$ in discrete time
Zero-sum games, flowrelative entropy to equilibrium conserved; recurrence—
Zero-sum games, Hedgeiterate spirals out, time average convergesentropy rises by exactly $\log\mathbb E\,e^{\eta f}\approx\tfrac12\eta^2\mathrm{Var}(f)$ per step; boundary reached after $1451$ steps at $\eta=0.1$
Optimismlast iterate converges$0.070\to1.5\times10^{-10}$ in $6000$ steps
Entropy regularizationconverges; biased; not natural$\frac{d}{dt}D(u\,\Vert\,x)=-\alpha\,(D(u\,\Vert\,x)+D(x\,\Vert\,u))$, so rate at least $\alpha$; with Rock duplicated the regularized equilibrium puts $(0.40,0.36,0.24)$ on the three types at $\tau=0.5$
Evaluating agentsratings plus cycles plus a harmonic partplanted example: ratings recovered, transitivity index $0.885$, curl $1.5$
Redundant agentsscore averaging and a name-based mutation chain not natural; replicator time average naturalmean scores $(0,0,0.25,-0.25)$; chain $(0.4,0.4,0.2)$ on types; replicator $(\tfrac13,\tfrac13,\tfrac13)$
New conventions in a finite populationcan invade by driftmore likely than neutral iff the barrier is below $\tfrac13$: $N\rho=1.021,1.009,0.991,0.958$
Copying topologyamplifiers and suppressorsline $0.167$ < well-mixed $0.251$ < star $0.287$ ($N=6$, $r=1.2$)
InferenceBayes is Hedge at $\eta=1$exact

Three points the organization makes visible

One term, two roles. The excess $\Lambda_\eta=\log\mathbb E_x e^{\eta(\pi-x\cdot\pi)}$ from holding the payoff fixed for a step is the whole difference between regret $\ln n$ and regret $\sqrt T$, and in a zero-sum game it is the energy pumped into the cycle at every step.

Three repairs act in three places. Averaging changes the readout; optimism changes the box; entropy regularization changes the wiring, adding a loop from each learner's output to its own input. Averaging keeps naturality and loses the iterate. Regularization gains a rate and loses both naturality and the equilibrium.

Naturality sorts methods. The same test, commuting with clone maps, separates replicator from policy gradient, Nash-based evaluation from score averaging, and entropy on types from entropy on names. In a strictly feed-forward copying topology, by contrast, the game does not matter at all: fixation is $1/N$ whatever the fitness.

A remark relates the framework to recent work on populations of agents built on large language models (replicator dynamics over agent types, evolutionarily stable trust mechanisms, group selection on prompts). The titles and identifiers of five such items were checked online on 17 November 2026; their results were not evaluated. The paper's contribution there is a set of constraints: prompts that induce the same behaviour are clones; stability should be asked of sets and not of single prompts; and conclusions about invasion by one new agent need the finite-population correction.

What is and is not claimed

No new theorem of learning theory. The identifications, regret bounds, recurrence, Hodge decomposition, one-third law and isothermal theorem are from the literature and are cited. The observations are: the naturality theorem for learners with its exact slowdown factor; the entropy-regularization identity with its rate; the naturality table for evaluation methods; and the placing of the three repairs. They are short, and parts are likely known in other forms; invariance to redundant agents, for one, is a stated aim of Nash averaging (Balduzzi et al.). The statement about the mutation chain concerns the kernel of the paper's own small implementation and makes no claim about $\alpha$-Rank as published or as used.

Scope: everything is tabular, with at most six actions and populations of a few hundred, chosen to be exactly computable. Function approximation, sampling noise and partial observation are outside it. No experiment on a trained model was run; the numbers illustrate the statements and are not benchmarks.

Files

Status and authorship

Contribution. Organization of known results around a naturality criterion, with exact constants, a small number of observations, and reproducible computations.

Limitations. As listed above; in addition, the bibliography was assembled from memory except for seven entries whose details were checked online, proofs marked as sketches are sketches, and nothing has been independently refereed. The script's checks are evidence that the numbers are stated correctly and are not proofs. Corrections are welcome.

AI assistance. The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall, mathematical consistency checks, computation and typesetting.

Reproducibility artifacts

evolutionary-game-theory-in-machine-learning-and-ai.tex · 50,103 bytes · Read in browser · Download
Text indexing selected · View indexed source
SHA-256 0047de2603161c17f545a1abea2a00271744f49fbc263838588033e9fb961a9e

evolutionary-game-theory-in-machine-learning-and-ai.pdf · 243,293 bytes · Download
SHA-256 7ddda70d867036e168cbb951f95eab3f34cf9f776048fbbeb7d98726dfd9dc4a

Discussion

Open a careful, constructive conversation about this note.

Comments appear after editorial approval.