Evolutionary Game Theory in Machine Learning and AI: Naturality, Regret, and Recurrence
Continuous, single-column reading. Equations, tables, proofs, references and appendices stay in document order.
Contents · 19 sections
Evolutionary Game Theory in Machine Learning and AI:
Naturality, Regret, and Recurrence
Anonymous
Higher Categorical Semantics ⋅Version 1.0, 17 November 2026
Abstract. The replicator equation of evolutionary game theory and the standard learning rules of machine learning are the same objects under different names: multiplicative weights is the replicator flow sampled in time, natural policy gradient is the replicator field, and Bayesian updating is the replicator map with log-likelihood as payoff. We organize these identifications around one structure. A learner is an open dynamical system with payoffs as input and a mixed strategy as output; a game is a wiring of learners; and a learning rule is natural if its closed-loop dynamics commutes with payoff-preserving maps of games, in particular with the duplication of actions. Each qualitative statement is paired with a quantitative one. Naturality: the replicator learner is natural, while softmax policy gradient, entropy regularization, score averaging and a mutation-chain ranking are not, with computed defects; duplicating an action k times multiplies the policy-gradient learning time by exactly 2k/(k+1). Regret: at most lnn in continuous time for every horizon, against Tlnn/2 in discrete time. Recurrence: in zero-sum games the relative entropy to equilibrium is conserved by the flow, increases by exactly logEeηf≈η2Var(f)/2 per step of multiplicative weights, and decays at rate at least α under entropy-regularized dynamics. Finite populations of learners obey the one-third law and the amplifier and suppressor phenomena of evolutionary graph theory. All numerical statements are recomputed by a companion script with 21 checks. No new theorem of learning theory is claimed; the contribution is the organization, the exact constants, and the naturality criterion.
Introduction
Evolutionary game theory studies populations in which strategies that earn more become more common. Its central object, the replicator equation x˙i=xi((Ax)i−xTAx), was written down for animal conflicts [43, 28]. Machine learning arrived at the same equation several times: as the continuous-time limit of reinforcement learning [4] and of multiplicative weights [13, 41], as natural policy gradient on a softmax policy [22], and implicitly in population-based training, where agents copy their betters [20]. A compact account of the evolutionary side, written for philosophers and candid about the fragility of the models, is Alexander’s Element [1]; a companion volume [11] retells it in categorical language. This paper carries the results of that volume to machine learning.
The organizing idea is to separate what the evolutionary literature fuses. In (1) a learning rule and a game are multiplied together. We take them apart (): a learner is an open dynamical system, a box with payoffs flowing in and a mixed strategy flowing out, and a game is a wiring of such boxes. Self-play, two-player training, leagues and populations on graphs are different diagrams filled with the same boxes. The question that the categorical setting makes precise is which learners are natural: whose behaviour does not depend on how the actions happen to be named, and in particular on whether one action has been listed twice.
Results.
Each is stated with its status. Tags follow the conventions of the series: [classical; ]a classical result; [literature; source] a result of the cited literature; [reformulation; cf. source] a known result restated; [observation]an observation of this paper, proved here. Tags such as [B6] name checks in the companion script verify-evolutionary-learning.py.
Naturality (). The replicator learner is natural for all payoff-preserving maps. Softmax policy gradient is not natural for duplication, and the slowdown is the exact factor 2k/(k+1) (). Entropy regularization and two common ways of ranking agents fail the same test ().
Regret (). An identity gives regret at most lnn in continuous time; the term that discretization adds is the one that reappears as divergence in games.
Recurrence (). In zero-sum games continuous-time learning cycles, discrete-time learning spirals out at a computable rate, and three repairs act at three different places of the wiring diagram.
Populations (). Fixation probabilities, the one-third law, and the effect of the copying topology.
lists the pairs of qualitative and quantitative statements. states what is not claimed.
Related work.
The link between reinforcement learning and replicator dynamics goes back to [4] and is surveyed in [6]; the behaviour of regularized learning in zero-sum games is the subject of [36, 27, 3]; evolutionary methods for evaluating agents are developed in [5, 33]. On the categorical side, learners as open systems composed along wiring diagrams follow [42, 30], and compositional accounts of gradient-based learning are given in [14, 8]. What we add to both is the question of naturality with respect to maps of games, which neither literature asks in this form.
Learners, games, and naturality
Games and their maps
A symmetric game(S,A) is a finite set S of actions and a matrix A∈RS×S; Aij is the payoff of i against j. Mixed strategies form the simplex Δ(S). A payoff-preserving mapf:(S,A)→(T,B) is a function with Bf(i)f(j)=Aij. These are the morphisms of a category Game [11]. They are generated by three kinds: inclusions of a subset of actions, clone maps, which are surjections identifying actions with equal rows and columns, and symmetries. A map f induces the pushforward f∗:Δ(S)→Δ(T), which adds the weights of identified actions. The basic fact is that f∗ preserves payoffs: (Bf∗x)f(i)=(Ax)i and (f∗x)TB(f∗x)=xTAx.
Learners
Definition 1. A learner on S is an open dynamical system with input a payoff vector π∈RS, a state z, a readout x=r(z)∈Δ(S), and an update: z˙=V(z,π) in continuous time or z+=U(z,π) in discrete time.
Five learners recur below. All read out by softmax, applied to Q/τ in the last case.
learner
state
update
replicator (R)
scores z
z˙=π
Hedge, MWU (H)
scores z
z+=z+ηπ
optimistic (O)
scores z
z+=z+η(2πt−πt−1)
policy grad. (PG)
logits θ
θ˙i=xi(πi−x⋅π)
Boltzmann Q (Q)
values Q
Q˙=α(π−Q)
(PG) is gradient ascent on expected payoff x⋅π in the logits; (Q) is the expected, all-actions form of Q-learning with Boltzmann exploration [44, 38].
Proposition 2 (Hedge is the replicator, sampled). [classical; ]The replicator learner fed a payoff stream π(⋅) has x(t)=softmax(logx(0)+∫0tπ), and in the variable x it obeys x˙i=xi(πi−x⋅π). One step of Hedge with step size η is the time-η map of this flow under a payoff held constant over the step. [B1]
Proposition 3 (Policy gradient and its natural form). [classical; [22, 2]] For x=softmax(θ), ∂(x⋅π)/∂θi=xi(πi−x⋅π). Multiplying this gradient by the inverse of the Fisher matrix diag(x)−xxT, on the tangent space, gives π−(x⋅π)1: natural policy gradient on a softmax policy is the replicator learner. [B2]
So (PG) differs from (R) by one factor of xi in the update of the logits. Neural Replicator Dynamics makes exactly this change in a deep policy-gradient method, bypassing the gradient through the softmax, and reduces to Hedge in the single-state tabular case [18]. In the variable x, (PG)x˙i=xi(xi(πi−πˉ)−j∑xj2(πj−πˉ)), with πˉ=x⋅π. The extra factor has a cost that is easy to state.
Proposition 4 (Escape from a bad start). [classical; cf. [29]] Two actions, payoff gap D>0, the better action at initial probability ϵ. The time for it to reach 21 is tRtPG=D1logϵ1−ϵ,=2D1(ϵ1−1−ϵ1+2logϵ1−ϵ).
Proof. With x the probability of the better action, (R) is x˙=Dx(1−x) and, by (2), (PG) is x˙=2Dx2(1−x)2. Integrate. ◻
For D=1 and ϵ=0.1, 0.01, 0.001 the two times are 2.20 and 6.64, 4.60 and 54.1, 6.91 and 506: ratios 3.0, 11.8 and 73.3. [B3] The replicator learner recovers from a bad initialization in logarithmic time and policy gradient in time of order 1/ϵ. In evolutionary terms (PG) treats a rare good action as the drift game treats a neutral mutant [11].
Closing the loop
A learner becomes a dynamical system when its input is supplied. Self-play in a symmetric game feeds back π=Ar(z); for (R) this is (1). Two learners in a bimatrix game (A,B) are wired crosswise, π1=Ar(z2) and π2=BTr(z1). A league or a population on a graph is a larger diagram. Formally, open dynamical systems are algebras for an operad of wiring diagrams [42, 30], and a multi-agent learning problem is a diagram together with a choice of box for each node ().
A game is a wiring of learners. The boxes can be changed without changing the diagram, and the diagram without changing the boxes.
Naturality
Definition 5. A family of learners, one for each finite set S, is natural for a class of payoff-preserving maps if for every f:(S,A)→(T,B) in the class, f∗ carries self-play trajectories in Δ(S) to self-play trajectories in Δ(T).
Naturality for inclusions says that an action never tried is never missed, and the rest behaves as if it did not exist. Naturality for clone maps says that listing an action twice, or splitting it into indistinguishable variants, changes nothing about the total weight of the variants. Naturality for symmetries is equivariance.
Theorem 6 (Which learners are natural). [observation]
(R) and (H) are natural for all payoff-preserving maps.
(PG) is natural for inclusions and symmetries and not for clone maps. If an action of total probability y is split into copies with probabilities x1,…,xk, the leading term of its drift is ∑cxc2 where the unsplit learner has y2.
In the two-action problem of , with either action split into k equal copies, the probability y of the better class obeys y˙=(1+k1)Dy2(1−y)2. Every learning time is multiplied by exactly 2k/(k+1).
(Q) is natural for symmetries and not for clone maps. Its rest points are the logit equilibria x=softmax(Ax/τ), which depend on the number of copies of each action, and every face of the simplex is repelling: no action stays unused.
Proof. (a) For (R), with y=f∗x, the pushforward identity gives y˙t=∑f(i)=txi((Ax)i−xTAx)=yt((By)t−yTBy) for t in the image and yt≡0 otherwise. (H) is a time-η map of the same flow with frozen payoffs, and frozen payoffs are also preserved. (b) In (2) the factor xi gives invariance of faces. Summing (2) over the copies of one action gives (∑cxc2)(π−πˉ)−y∑jxj2(πj−πˉ), and ∑cxc2<y2 unless all but one copy is empty. (c) By symmetry the copies stay equal. Substituting xc=y/k for the split action in (2) and summing gives the stated equation in both cases; compare with 2Dy2(1−y)2. (d) In the variable x, (Q) in self-play is x˙i=xi(τα(πi−πˉ)−α(logxi−∑jxjlogxj)), whose rest points satisfy logxi=πi/τ+const. The entropy term makes x˙i/xi→+∞ as xi→0, and it distinguishes one action of weight y from two of weight y/2, since 2⋅2ylog2y=ylogy. ◻
Computed defects. For Rock–Paper–Scissors with Rock duplicated, the largest discrepancy over fifty random states between the pushed-forward field and the field of the three-action game was 4×10−17 for (R) and 0.028 for (PG). [B19] From ϵ=0.01 the policy-gradient time to 21 was 54.1, 72.1, 86.5 and 96.2 for k=1,2,4,8, and the replicator time 4.60 for every k. [B20] For (Q) the logit equilibrium of Rock–Paper–Scissors is uniform at every temperature; with Rock duplicated its mass on the three types is (0.49,0.29,0.23) at τ=2, (0.40,0.36,0.24) at τ=0.5 and (0.34,0.35,0.31) at τ=0.1. [B21]
Remark 7. Large action spaces in practice contain near-duplicates: synonymous tokens, equivalent moves, redundant members of a league. says which update rules are indifferent to them. It is the finite-dimensional shadow of the invariance of natural gradient under reparametrization [2], stated for the maps that matter in games. Entropy bonuses are the clearest case of a method that counts names: at high temperature the regularized equilibrium gives duplicated Rock half the mass.
Regret
A learner has no regret if, against any stream of payoffs, its average payoff is asymptotically at least that of the best fixed action. For the replicator learner this is an identity.
Theorem 8 (Continuous time). [classical; [41, 23]] For the replicator learner and any bounded measurable payoff stream, ∫0T(πi−x⋅π)dt=logxi(0)xi(T)≤logxi(0)1 for every action i and every horizon T. From the uniform initial state the regret is at most lnn.
Proof.dtdlogxi=πi−x⋅π by , and xi(T)≤1. ◻
The bound does not grow with T. Against a stream of six payoffs that alternates between random vectors and vectors chosen to punish the current favourite, the regret at T=80 was 0.24, against ln6=1.79, and the identity held to 5×10−8. [B4]
Theorem 9 (Discrete time). [classical; [13, 7]] For Hedge with payoffs in [0,1] and η=8lnn/T, the regret after T steps is at most Tlnn/2.
Sketch. One step multiplies the normalizing constant by Exeηπ, and Hoeffding’s lemma gives logExeη(π−x⋅π)≤η2/8. Summing, the discrete learner satisfies the identity of up to an excess of at most η/8 per step: the regret is at most lnn/η+ηT/8. ◻
Over forty runs with n∈{2,5,10}, T∈{200,1000}, half against random payoffs and half against an adversary, the largest ratio of regret to bound was 0.50. [B5] The whole difference between lnn and T is the term Λη(x,π)=logExeη(π−x⋅π)=21η2Varx(π)+O(η3), the price of holding the payoff fixed for a step. The same term decides the fate of discrete learning in games.
Corollary 10. [classical; ]If both players of a zero-sum game with payoffs in an interval of length 1 use Hedge with the step of , their time-averaged strategies form a 2lnn/(2T)-equilibrium. In a general game the time-averaged joint play approaches the set of coarse correlated equilibria.
In self-play of Rock–Paper–Scissors, whose payoffs have range 2, the time average over T=20000 steps could be exploited for 0.0015, against the corresponding bound 0.0105. [B8]
Zero-sum games: recurrence, divergence, repair
Regret bounds speak of averages. What the strategies themselves do is another matter, and it is the matter that concerns adversarial training [15], where the iterate and not the average is the model.
The flow cycles
Theorem 11 (Conservation and recurrence). [literature; [19, 36, 27]] Let A be antisymmetric with an interior equilibrium x∗. Along replicator self-play D(x∗∥x(t)) is constant. For two replicator learners in a zero-sum game with interior equilibrium (x∗,y∗) the sum D(x∗∥x)+D(y∗∥y) is constant, and the flow is Poincaré recurrent: almost every trajectory returns arbitrarily close to its starting point infinitely often.
Proof of conservation.dtdD(x∗∥x)=−∑ixi∗(πi−πˉ)=−(x∗)TAx+xTAx. Antisymmetry gives xTAx=0, and (x∗)TAx=−xTAx∗=0 because Ax∗ is constant on the support of an interior equilibrium and that constant is the value 0. ◻
Conservation says that such a game does no work on the learner. An antisymmetric matrix has no symmetric part, so the gradient component of the replicator field vanishes [11]; and when the equilibrium is uniform the game is purely cyclic in the sense of the Hodge decomposition of .
The discretization spirals out
Proposition 12 (Exact drift of Hedge). [reformulation; cf. [3]] In Hedge self-play of an antisymmetric game with interior equilibrium x∗, each step changes the relative entropy by exactly D(x∗∥x+)−D(x∗∥x)=Λη(x,Ax)≥0, with equality only if all actions in use earn the same. Hence D(x∗∥xt) is non-decreasing, and it increases by about 21η2Varx(Ax) per step.
Proof.xi+=xieηπi/Z with Z=Exeηπ, so the difference is −ηx∗⋅π+logZ. As in , x∗⋅π=0=x⋅π, so the difference is logZ=Λη, and Jensen’s inequality gives the sign. ◻
At the state (0.5,0.3,0.2) of Rock–Paper–Scissors the increase per step was 5.01×10−5, 2.01×10−4 and 8.08×10−4 for η=0.05, 0.1, 0.2, against 21η2Var=5.00×10−5, 2.00×10−4, 8.00×10−4. [B6] With η=0.1 the relative entropy increased at every step and some action fell below probability 0.01 after 1451 steps. [B7] The quantity (4) that costs a learner T in regret is, in a game, energy pumped into the cycle at every step. The time average still converges (); the iterate leaves.
Three repairs, in three places
Proposition 13 (Optimism). [literature; [37, 10, 9]] The optimistic learner (O), which uses 2πt−πt−1 as its prediction of the next payoff, converges in the last iterate in zero-sum games with a unique equilibrium, for small enough η.
From the same initial state and η=0.1 the relative entropy to equilibrium fell from 0.070 to 1.5×10−10 in 6000 steps. [B9]
Theorem 14 (Entropy regularization, with its rate). [observation]Let A be antisymmetric with the uniform state u as equilibrium. Under the dynamics (3), for every τ>0, dtdD(u∥x)=−α(D(u∥x)+D(x∥u)), and therefore D(u∥x(t))≤e−αtD(u∥x(0)).
Proof.dtdD(u∥x)=−∑iuix˙i/xi. The payoff terms vanish as in . The entropy terms give α(∑iuilogxi−∑jxjlogxj), and D(u∥x)+D(x∥u)=∑jxjlogxj−∑iuilogxi. ◻
Convergence of Q-learning dynamics in competitive games is known in much greater generality [25]; the point of is the exact identity, in which the rate is the learning rate α and does not depend on the game or the temperature. With α=0.7 the relative entropy fell from 0.070 to 6.1×10−8 at t=10, against the bound 6.4×10−5. [B10] In general the limit is a logit equilibrium and not a Nash equilibrium [B11], and by (d) it is not natural.
Remark 15 (Where each repair acts). Averaging changes the readout: the box and the wiring are untouched and one reports T1∑xt. Optimism changes the box: the learner carries one more step of memory. Regularization changes the wiring: (3) is the replicator learner in the game with the additional self-payoff −τlogxi, a loop from each learner’s output to its own input, up to a constant change of time scale. In evolutionary terms these are time-averaging of frequencies, anticipation, and mutation [44]. The trade is visible in the table of : averaging keeps naturality and loses the iterate; regularization gains a rate and loses both naturality and the equilibrium.
Evaluating agents
Training by self-play, leagues and population-based methods all need to say which of a set of agents is better [45, 26]. The data is an evaluation matrix: an antisymmetric K with Kij the advantage of agent i over agent j, for instance the log-odds of winning. This is a symmetric zero-sum game on the set of agents, and an evaluation method assigns to it a score vector or a distribution on agents.
Theorem 16 (Ratings and cycles). [literature; [21, 5]] Every antisymmetric K decomposes uniquely and orthogonally as Kij=(ri−rj)+Cij,ri=n1∑jKij,∑jCij=0. The first part satisfies Gij+Gjk+Gki=0 on every triple, and every matrix satisfying this is of that form. The second part carries the same curl Kij+Kjk+Kki as K.
The ratings r are the least-squares fit of an Elo-like scale; the transitivity index∥G∥2/∥K∥2 measures how much of the data a scale can explain. For five agents with planted ratings and one planted cycle of strength 0.5 among three of them, the decomposition returned the ratings (0.7,0.3,0,−0.3,−0.7), index 0.885 and curl 1.5 on the planted triple. [B12] That closed implies exact is the vanishing of the first cohomology of a simplex.
Proposition 17 (Incomplete comparisons). [literature; [21]] If only some pairs of agents have been compared, the decomposition acquires a third, harmonic component: circulations around loops of the comparison graph that bound no triple of mutually compared agents. It is locally consistent and globally not a rating, and its dimension is the first Betti number of the comparison complex.
Four agents compared only around a square, each beating the next by one unit, give the smallest example: zero divergence, no triangles, nothing to fit. [B15] Sparse tournament schedules create such classes by design.
The naturality criterion of applies to evaluation unchanged. A league that contains two copies of an agent, or ten near-copies obtained from one checkpoint, should not be ranked differently for that reason.
Proposition 18 (Naturality of evaluation methods). [observation]Let f:(S,K)→(T,K′) be a clone map between evaluation games.
Mean score against the field, si=n1∑jKij, is not natural: duplicating an agent raises the score of everything that beats it.
The set of Nash equilibria is natural: x is an equilibrium of (S,K) if and only if f∗x is one of (T,K′). When the equilibrium of (T,K′) is unique, every rule that selects an equilibrium is therefore natural on types. Invariance to redundant agents is the stated aim of Nash averaging [5].
The time average of replicator self-play is natural, and when (T,K′) has a unique interior equilibrium it assigns to each type its equilibrium mass.
The stationary distribution of the small-mutation chain in which a monomorphic population is invaded by a mutant drawn uniformly from the other names, with transition probability given by the fixation probability, is not natural.
Proof. (a) and (d) by example, below. (b) Payoffs against f∗x at the types are the payoffs against x at the agents, and a clone map omits no type [11, Prop. 1.10]. (c) By (a), f∗ carries trajectories to trajectories and hence time averages to time averages; the time-average theorem for the replicator equation [19] identifies the limit. ◻
The example. Rock, a second copy of Rock, Paper, Scissors. Mean scores are (0,0,0.25,−0.25): Paper looks best and Scissors worst, although the game is symmetric among the three types. The time-averaged replicator mass on the types is (0.3334,0.3333,0.3333). [B13] The small-mutation chain, uniform on plain Rock–Paper–Scissors, gives (0.2,0.2,0.4,0.2) on the four names: Rock’s type has mass 0.4, Paper 0.4 and Scissors 0.2. [B14] The reason is visible in the chain: a population of Scissors is now invaded twice as often.
Remark 19 (Chain components). The chain in (d) is in the manner of α-Rank [33], which ranks agents by the stationary distribution of a Markov chain on strategy profiles built from fixation probabilities, and which grounds the construction in Conley’s theory: the long-run behaviour of a dynamical system is carried by its chain recurrent set, and Markov–Conley chains are a finite model of the sink components. This is the right generalization of equilibrium for games that cycle [35], and the dynamical content is natural: chain components of a natural flow map into chain components. Statement (d) concerns only the mutation kernel of our small implementation, which counts names; we make no claim about α-Rank as published or as used. A kernel that draws mutants by type would restore invariance, at the price of knowing the types.
Finite populations of learners
Population-based training keeps N agents, lets better ones overwrite worse ones, and perturbs the copies [20]. With copying in proportion to fitness and one replacement at a time this is the Moran process of population genetics, and its classical theory applies [31, 11].
Theorem 20 (Fixation and the one-third law). [literature; [32]] Let two variants play a coordination game: each does best against its own kind, and the newcomer must exceed a fraction x∗ of the population before it is favoured. Under weak selection in a large population a single newcomer takes over with probability greater than the neutral value 1/N if and only if x∗<31.
With N=400 and selection intensity 10−3, Nρ was 1.021, 1.009, 0.991 and 0.958 at x∗=0.25, 0.30, 0.36 and 0.45. [B16] The deterministic dynamics says that a new convention, protocol or shared representation cannot invade from rarity at all when it needs partners; the finite population says it can, by drift, and when it is more likely than chance to do so.
Theorem 21 (The copying topology matters). [literature; [24]] Let offspring of agent i replace agent j with probability wij. The fixation probability of a variant of relative fitness r equals the well-mixed value (1−r−1)/(1−r−N) for all r if and only if W is doubly stochastic. A star amplifies selection. On a directed line, in which each agent copies only its predecessor, fixation has probability 1/N whatever the fitness.
For N=6 and r=1.2 the exact values are 0.167 on the line, 0.251 well-mixed and 0.287 on the star. [B17] The design reading is direct. Symmetric exchange between agents, of any shape, neither helps nor hurts selection. A hub that many agents copy from and that copies from them sharpens it. A strictly feed-forward pipeline of copying removes selection altogether: which variant ends up everywhere is decided by where it started.
Remark 22 (Populations of language-model agents). Recent work studies populations of agents built on large language models with exactly these tools: replicator dynamics over agent types in open-source games [34], evolutionarily stable strategies as a design target for trust mechanisms [12], group selection and replicator–mutator models for the evolution of cooperative prompts [16], evolutionary game theory as a probe of strategic behaviour [40], and a survey of the area [46]. The titles and identifiers of these five items were checked on 17 November 2026; we have not evaluated their results. Our contribution to that programme is a set of constraints. Two prompts that induce the same behaviour are clones, so an analysis of a prompt population should be natural in the sense of ; stability should be asked of sets of prompts and not of single ones, because stability of a point is destroyed by cloning and stability of a set is not [11, Thm. 2.8]; and conclusions drawn from the replicator equation about invasion by a single new agent are subject to the correction of .
Inference
Proposition 23 (Bayes is the replicator). [classical; [39, 17]] Let S be a set of hypotheses with likelihoods ℓh. Bayesian updating of a prior x on an observation o is one step of Hedge with η=1 and payoff πh=logℓh(o). [B18]
Bayesian inference is the replicator learner fed log-likelihoods, and the dictionary is exact. Naturality for clone maps is the statement that splitting a hypothesis into two indistinguishable ones, with the prior mass divided, changes no posterior probability of any event. The regret identity of is the statement that the log-loss of the Bayesian mixture exceeds that of the best hypothesis by at most log(1/xh(0)). And the elimination of dominated strategies with its rate [11] is posterior concentration: if observations come from h∗, the log-odds of any other h against h∗ fall at the expected rate D(ℓh∗∥ℓh) per observation.
Summary of results
Qualitative statements and their quantitative forms. All numbers are produced by the companion script.
Topic
Qualitative statement
Quantitative statement
Checks
Hedge and replicator
one is the other, sampled
exact at every step size
B1
Policy gradient
replicator with an extra factor xi
escape time ∼1/(2Dϵ) against log(1/ϵ)/D; ratios 3.0, 11.8, 73.3
B2, B3
Duplicated actions
(R), (H) natural; (PG), (Q) not
slowdown 2k/(k+1); defect 0.028
B19–B21
Regret
no regret
≤lnn continuous; ≤Tlnn/2 discrete; observed ratio ≤0.50
+21η2Var per step; boundary after 1451 steps; exploitability 0.0015
B6–B8
Optimism
last iterate converges
0.070→1.5×10−10 in 6000 steps
B9
Entropy regularization
converges; not natural; biased
rate at least α, by an identity; type masses (0.40,0.36,0.24) at τ=0.5
B10, B11, B21
Evaluation
ratings plus cycles plus harmonic
index 0.885; curl 1.5
B12, B15
Redundant agents
averaging and name-mutation chain not natural; replicator average natural
(0,0,0.25,−0.25); (0.4,0.4,0.2); (31,31,31)
B13, B14
New conventions
can invade by drift
iff barrier <31; Nρ=1.021,…,0.958
B16
Copying topology
amplifiers and suppressors
0.167<0.251<0.287
B17
Inference
Bayes is Hedge at η=1
exact
B18
collects the results. Read down the middle column it is a list of facts most of which are individually known; read across, each has a constant attached; and the recurring entry is naturality, which sorts the methods into those that depend on how actions and agents are named and those that do not.
Limits
Scope. Everything here is tabular: a finite set of actions and exact payoffs. Function approximation, sampling noise, and partial observation are outside it; they are where methods such as [18] do their work. The numbers come from games with at most six actions and populations of at most a few hundred, chosen to be exactly computable, and are illustrations of the statements and not benchmarks. No experiment on a trained model was run.
Novelty. The identifications of , the regret bounds, recurrence, the Hodge decomposition, the one-third law and the isothermal theorem are from the literature. The observations are , the identity of , , and the placing of the three repairs in . They are short, and parts of them are likely known in other forms; the invariance of Nash averaging under redundancy, for one, is a stated motivation of [5].
How categorical. Modestly. The category Game, the functor to dynamical systems, and the operad of wiring diagrams are ordinary category theory; the only cohomology is in degree one; the homotopy theory is in the Conley index behind , which we invoke and do not compute. The language earns its place by one question, whether a construction commutes with the maps, which turned out to separate methods that the literature usually compares on other grounds.
References. The bibliography was assembled from memory, except for [33, 18] and the five items of , whose bibliographic details were checked online on 17 November 2026. It should be verified before it is relied on.
Authorship. This paper was drafted with the assistance of an AI system (Anthropic’s Claude), which also wrote and ran the companion script.
Open problems.
[programme](1) Characterize the learners that are natural for clone maps among those derived from a regularizer; we expect entropy on types and not on names to be forced. (2) Give the rate in for a non-uniform equilibrium, where the limit moves with τ. (3) Extend (c) from time averages to chain components, so that a ranking by recurrent classes is natural by construction. (4) Find the analogue of the isothermal theorem for populations of learners whose fitness depends on the population, where the graph and the game interact.
The companion script
verify-evolutionary-learning.py uses only the Python standard library, fixes its random seed, runs in under half a minute, and exits with status zero if and only if its 21 checks pass. Flows are integrated by a fourth-order Runge–Kutta scheme; fixation probabilities are exact. The checks test instances and recompute the numbers quoted in the text. They do not prove the theorems.
B1
Hedge equals the time-η replicator flow, 30 random cases
B2
policy gradient in logits; Fisher preconditioning
B3
escape times, , against numerical integration
B4
continuous-time regret identity and bound
B5
discrete regret bound, 40 runs, random and adversarial
logit equilibrium with a cloned action, three temperatures
References
J. M. Alexander, Evolutionary Game Theory, Elements in Decision Theory and Philosophy, Cambridge University Press, 2023.
S. Amari, Natural gradient works efficiently in learning, Neural Comput. 10 (1998), 251–276.
J. P. Bailey, G. Piliouras, Multiplicative weights update in zero-sum games, in Proc. ACM Conf. Economics and Computation (EC), 2018, 321–338.
T. Börgers, R. Sarin, Learning through reinforcement and replicator dynamics, J. Econom. Theory 77 (1997), 1–14.
D. Balduzzi, K. Tuyls, J. Pérolat, T. Graepel, Re-evaluating evaluation, in Advances in Neural Information Processing Systems 31, 2018.
D. Bloembergen, K. Tuyls, D. Hennes, M. Kaisers, Evolutionary dynamics of multi-agent learning: a survey, J. Artificial Intelligence Res. 53 (2015), 659–697.
N. Cesa-Bianchi, G. Lugosi, Prediction, Learning, and Games, Cambridge University Press, 2006.
G. S. H. Cruttwell, B. Gavranović, N. Ghani, P. Wilson, F. Zanasi, Categorical foundations of gradient-based learning, in European Symposium on Programming (ESOP), 2022.
C. Daskalakis, I. Panageas, Last-iterate convergence: zero-sum games and constrained min-max optimization, in Innovations in Theoretical Computer Science (ITCS), 2019.
C. Daskalakis, A. Ilyas, V. Syrgkanis, H. Zeng, Training GANs with optimism, in Int. Conf. Learning Representations (ICLR), 2018.
Anonymous, Evolutionary Game Theory in Higher Categories, Higher Categorical Semantics, 2026. Companion volume.
Wang et al., Ev-Trust, a trust mechanism for multi-agent systems of language models built on evolutionary stability, preprint, arXiv:2512.16167, 2025. Full title and author list not recorded when the listing was checked.
Y. Freund, R. E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, J. Comput. System Sci. 55 (1997), 119–139.
B. Fong, D. I. Spivak, R. Tuyéras, Backprop as functor: a compositional perspective on supervised learning, in LICS 2019, IEEE, 2019.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in Advances in Neural Information Processing Systems 27, 2014.
Group selection to evolve cooperative prompts in populations of LLM agents, listed for NeurIPS 2026; authors not recorded when the listing was checked.
M. Harper, The replicator equation as an inference dynamic, preprint, arXiv:0911.1763.
D. Hennes, D. Morrill, S. Omidshafiei, R. Munos, J. Perolat, M. Lanctot, A. Gruslys, J.-B. Lespiau, P. Parmas, E. Duéñez-Guzmán, K. Tuyls, Neural replicator dynamics: multiagent learning via hedging policy gradients, in Proc. AAMAS, 2020; arXiv:1906.00190.
J. Hofbauer, K. Sigmund, Evolutionary Games and Population Dynamics, Cambridge University Press, 1998.
M. Jaderberg, V. Dalibard, S. Osindero, W. M. Czarnecki, J. Donahue, A. Razavi, O. Vinyals, T. Green, I. Dunning, K. Simonyan, C. Fernando, K. Kavukcuoglu, Population based training of neural networks, preprint, arXiv:1711.09846.
X. Jiang, L.-H. Lim, Y. Yao, Y. Ye, Statistical ranking and combinatorial Hodge theory, Math. Program. 127 (2011), 203–244.
S. Kakade, A natural policy gradient, in Advances in Neural Information Processing Systems 14, 2001.
J. Kwon, P. Mertikopoulos, A continuous-time approach to online optimization, J. Dyn. Games 4 (2017), 125–148.
E. Lieberman, C. Hauert, M. A. Nowak, Evolutionary dynamics on graphs, Nature 433 (2005), 312–316.
S. Leonardos, G. Piliouras, K. Spendlove, Exploration-exploitation in multi-agent competition: convergence with bounded rationality, in Advances in Neural Information Processing Systems 34, 2021.
M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, T. Graepel, A unified game-theoretic approach to multiagent reinforcement learning, in Advances in Neural Information Processing Systems 30, 2017.
P. Mertikopoulos, C. Papadimitriou, G. Piliouras, Cycles in adversarial regularized learning, in Proc. ACM-SIAM Symp. Discrete Algorithms (SODA), 2018, 2703–2717.
J. Maynard Smith, G. R. Price, The logic of animal conflict, Nature 246 (1973), 15–18.
J. Mei, C. Xiao, C. Szepesvári, D. Schuurmans, On the global convergence rates of softmax policy gradient methods, in Proc. Int. Conf. Machine Learning (ICML), 2020.
D. J. Myers, Double categories of open dynamical systems (extended abstract), in Applied Category Theory 2020, EPTCS 333, 2021, 154–167.
M. A. Nowak, Evolutionary Dynamics: Exploring the Equations of Life, Harvard University Press, 2006.
M. A. Nowak, A. Sasaki, C. Taylor, D. Fudenberg, Emergence of cooperation and evolutionary stability in finite populations, Nature 428 (2004), 646–650.
S. Omidshafiei, C. Papadimitriou, G. Piliouras, K. Tuyls, M. Rowland, J.-B. Lespiau, W. M. Czarnecki, M. Lanctot, J. Perolat, R. Munos, α-Rank: multi-agent evaluation by evolution, Sci. Rep. 9 (2019); arXiv:1903.01373.
Evaluating LLMs in open-source games, preprint, arXiv:2512.00371, 2025; authors not recorded when the listing was checked.
C. Papadimitriou, G. Piliouras, Game dynamics as the meaning of a game, ACM SIGecom Exchanges 16 (2019), no. 2, 53–63.
G. Piliouras, J. S. Shamma, Optimization despite chaos: convex relaxations to complex limit sets via Poincaré recurrence, in Proc. ACM-SIAM Symp. Discrete Algorithms (SODA), 2014, 861–873.
A. Rakhlin, K. Sridharan, Optimization, learning, and games with predictable sequences, in Advances in Neural Information Processing Systems 26, 2013.
Y. Sato, J. P. Crutchfield, Coupled replicator equations for the dynamics of learning in multiagent systems, Phys. Rev. E 67 (2003), 015206.
C. R. Shalizi, Dynamics of Bayesian updating with dependent data and misspecified models, Electron. J. Stat. 3 (2009), 1039–1074.
Strategic intelligence in large language models: evidence from evolutionary game theory, preprint, arXiv:2507.02618, 2025; authors not recorded when the listing was checked.
S. Sorin, Exponential weight algorithm in continuous time, Math. Program. 116 (2009), 513–528.
P. Schultz, D. I. Spivak, C. Vasilakopoulou, Dynamical systems and sheaves, Appl. Categ. Structures 28 (2020), 1–57.
P. D. Taylor, L. B. Jonker, Evolutionarily stable strategies and game dynamics, Math. Biosci. 40 (1978), 145–156.
K. Tuyls, K. Verbeeck, T. Lenaerts, A selection-mutation model for Q-learning in multi-agent systems, in Proc. AAMAS, 2003, 693–700.
O. Vinyals et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575 (2019), 350–354.
Xie, Liu, Lan, Szolnoki, Evolutionary games in LLM-agent societies: early milestones and perspectives, Chaos Solitons Fractals 213 (2026), 119158. Initials not recorded when the listing was checked.
Complete flattened LaTeX source
\documentclass[10pt,twocolumn]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[sc]{mathpazo}
\usepackage[scaled=0.9]{helvet}
\usepackage[letterpaper,margin=0.78in,columnsep=0.3in]{geometry}
\usepackage{amsmath,amssymb,amsthm,mathtools}
\usepackage{microtype}
\usepackage{xcolor}
\definecolor{accent}{HTML}{7A1F2B}
\definecolor{ink2}{HTML}{55595F}
\definecolor{fwd}{HTML}{1F4E79}
\definecolor{bwd}{HTML}{5B3A7A}
\usepackage{tikz}
\usetikzlibrary{arrows.meta,positioning}
\usepackage{booktabs,array}
\usepackage{enumitem}
\usepackage[font=small,labelfont={bf,sf}]{caption}
\usepackage[colorlinks=true,linkcolor=accent,citecolor=accent,urlcolor=accent,
pdftitle={Evolutionary Game Theory in Machine Learning and AI: Naturality, Regret, and Recurrence},
pdfauthor={Anonymous}]{hyperref}
\usepackage[nameinlink,capitalise,noabbrev]{cleveref}
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newcommand{\stag}[2]{{\footnotesize\sffamily\upshape\color{#1}[#2]}\enspace\ignorespaces}
\newcommand{\Scl}[1][]{\stag{ink2}{classical\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Sli}[1][]{\stag{fwd}{literature\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Srf}[1][]{\stag{accent}{reformulation\if\relax\detokenize{#1}\relax\else; cf.\ #1\fi}}
\newcommand{\Sob}{\stag{accent}{observation}}
\newcommand{\Spg}{\stag{bwd}{programme}}
\newcommand{\chk}[1]{{\footnotesize\sffamily\upshape\color{accent}[#1]}}
\newcommand{\cat}[1]{\mathsf{#1}}
\renewcommand{\Game}{\cat{Game}}
\newcommand{\Dyn}{\cat{Dyn}}
\newcommand{\Dist}{\Delta}
\newcommand{\R}{\mathbb{R}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\one}{\mathbf{1}}
\newcommand{\T}{\mathsf{T}}
\newcommand{\KL}[2]{D(#1\,\|\,#2)}
\newcommand{\sm}{\operatorname{softmax}}
\newcommand{\Var}{\operatorname{Var}}
\newcommand{\term}[1]{\textbf{\itshape #1}}
\setlength{\parskip}{0pt}
\frenchspacing
\raggedbottom
\title{\textbf{Evolutionary Game Theory in Machine Learning and AI:\\ Naturality, Regret, and Recurrence}}
\author{Anonymous}
\date{Higher Categorical Semantics \quad$\cdot$\quad Version 1.0, 17 November 2026}
\begin{document}
\twocolumn[
\maketitle
\begin{center}\begin{minipage}{0.86\textwidth}\small
\textbf{Abstract.} The replicator equation of evolutionary game theory and the standard learning rules of
machine learning are the same objects under different names: multiplicative weights is the replicator flow
sampled in time, natural policy gradient is the replicator field, and Bayesian updating is the replicator map
with log-likelihood as payoff. We organize these identifications around one structure. A learner is an open
dynamical system with payoffs as input and a mixed strategy as output; a game is a wiring of learners; and a
learning rule is \emph{natural} if its closed-loop dynamics commutes with payoff-preserving maps of games, in
particular with the duplication of actions. Each qualitative statement is paired with a quantitative one.
Naturality: the replicator learner is natural, while softmax policy gradient, entropy regularization, score
averaging and a mutation-chain ranking are not, with computed defects; duplicating an action $k$ times multiplies
the policy-gradient learning time by exactly $2k/(k+1)$. Regret: at most $\ln n$ in continuous time for every
horizon, against $\sqrt{T\ln n/2}$ in discrete time. Recurrence: in zero-sum games the relative entropy to
equilibrium is conserved by the flow, increases by exactly $\log\E e^{\eta f}\approx\eta^2\Var(f)/2$ per step
of multiplicative weights, and decays at rate at least $\alpha$ under entropy-regularized dynamics. Finite
populations of learners obey the one-third law and the amplifier and suppressor phenomena of evolutionary graph
theory. All numerical statements are recomputed by a companion script with 21 checks. No new theorem of
learning theory is claimed; the contribution is the organization, the exact constants, and the naturality
criterion.
\end{minipage}\end{center}
\vspace{1.2em}
]
\section{Introduction}
Evolutionary game theory studies populations in which strategies that earn more become more common. Its
central object, the replicator equation
\begin{equation}\label{eq:rep}
\dot x_i=x_i\bigl((Ax)_i-x^{\T}Ax\bigr),
\end{equation}
was written down for animal conflicts \cite{TaJo78,MSP73}. Machine learning arrived at the same equation
several times: as the continuous-time limit of reinforcement learning \cite{BoSa97} and of multiplicative
weights \cite{FrSc97,Sor09}, as natural policy gradient on a softmax policy \cite{Kak01}, and implicitly in
population-based training, where agents copy their betters \cite{Jad17}. A compact account of the evolutionary side, written
for philosophers and candid about the fragility of the models, is Alexander's Element \cite{Alex23}; a
companion volume \cite{EGHC} retells it in categorical language. This paper carries the results of that volume
to machine learning.
The organizing idea is to separate what the evolutionary literature fuses. In \eqref{eq:rep} a learning rule
and a game are multiplied together. We take them apart (\Cref{sec:setting}): a \emph{learner} is an open
dynamical system, a box with payoffs flowing in and a mixed strategy flowing out, and a \emph{game} is a wiring
of such boxes. Self-play, two-player training, leagues and populations on graphs are different diagrams
filled with the same boxes. The question that the categorical setting makes precise is which learners are
\emph{natural}: whose behaviour does not depend on how the actions happen to be named, and in particular on
whether one action has been listed twice.
\paragraph{Results.} Each is stated with its status. Tags follow the conventions of the series:
\Scl a classical result; \Sli[source] a result of the cited literature; \Srf[source] a known result restated;
\Sob an observation of this paper, proved here. Tags such as \chk{B6} name checks in the companion script
\texttt{verify-evolutionary-learning.py}.
\begin{itemize}[leftmargin=1.2em,itemsep=1pt,topsep=2pt]
\item \emph{Naturality} (\Cref{sec:setting,sec:eval}). The replicator learner is natural for all
payoff-preserving maps. Softmax policy gradient is not natural for duplication, and the slowdown is the
exact factor $2k/(k+1)$ (\Cref{thm:natural}). Entropy regularization and two common ways of ranking agents fail
the same test (\Cref{prop:evalnat}).
\item \emph{Regret} (\Cref{sec:regret}). An identity gives regret at most $\ln n$ in continuous time; the
term that discretization adds is the one that reappears as divergence in games.
\item \emph{Recurrence} (\Cref{sec:zerosum}). In zero-sum games continuous-time learning cycles, discrete-time
learning spirals out at a computable rate, and three repairs act at three different places of the wiring
diagram.
\item \emph{Populations} (\Cref{sec:pop}). Fixation probabilities, the one-third law, and the effect of the
copying topology.
\end{itemize}
\Cref{tab:summary} lists the pairs of qualitative and quantitative statements. \Cref{sec:limits} states what
is not claimed.
\paragraph{Related work.} The link between reinforcement learning and replicator dynamics goes back to
\cite{BoSa97} and is surveyed in \cite{BTHK15}; the behaviour of regularized learning in zero-sum games is the
subject of \cite{PiSh14,MPP18,BaPi18}; evolutionary methods for evaluating agents are developed in
\cite{BTPG18,OPP19}. On the categorical side, learners as open systems composed along wiring diagrams follow
\cite{SSV20,Mye21}, and compositional accounts of gradient-based learning are given in \cite{FST19,CGG22}. What we
add to both is the question of naturality with respect to maps of games, which neither literature asks in this
form.
\section{Learners, games, and naturality}\label{sec:setting}
\subsection{Games and their maps}
A \term{symmetric game} $(S,A)$ is a finite set $S$ of actions and a matrix $A\in\R^{S\times S}$; $A_{ij}$ is
the payoff of $i$ against $j$. Mixed strategies form the simplex $\Dist(S)$. A \term{payoff-preserving map}
$f\colon(S,A)\to(T,B)$ is a function with $B_{f(i)f(j)}=A_{ij}$. These are the morphisms of a category $\Game$
\cite{EGHC}. They are generated by three kinds: \emph{inclusions} of a subset of actions, \emph{clone maps},
which are surjections identifying actions with equal rows and columns, and \emph{symmetries}. A map $f$
induces the pushforward $f_*\colon\Dist(S)\to\Dist(T)$, which adds the weights of identified actions. The basic
fact is that $f_*$ preserves payoffs: $(Bf_*x)_{f(i)}=(Ax)_i$ and $(f_*x)^{\T}B(f_*x)=x^{\T}Ax$.
\subsection{Learners}
\begin{definition}
A \term{learner} on $S$ is an open dynamical system with input a payoff vector $\pi\in\R^S$, a state $z$, a
readout $x=r(z)\in\Dist(S)$, and an update: $\dot z=V(z,\pi)$ in continuous time or $z^+=U(z,\pi)$ in discrete
time.
\end{definition}
\noindent Five learners recur below. All read out by $\sm$, applied to $Q/\tau$ in the last case.
\begin{center}\small
\begin{tabular}{@{}lll@{}}
\toprule
learner & state & update\\
\midrule
replicator (R) & scores $z$ & $\dot z=\pi$\\
Hedge, MWU (H) & scores $z$ & $z^+=z+\eta\pi$\\
optimistic (O) & scores $z$ & $z^+=z+\eta(2\pi_t-\pi_{t-1})$\\
policy grad.\ (PG) & logits $\theta$ & $\dot\theta_i=x_i(\pi_i-x\!\cdot\!\pi)$\\
Boltzmann $Q$ (Q) & values $Q$ & $\dot Q=\alpha(\pi-Q)$\\
\bottomrule
\end{tabular}
\end{center}
\noindent (PG) is gradient ascent on expected payoff $x\cdot\pi$ in the logits; (Q) is the expected,
all-actions form of $Q$-learning with Boltzmann exploration \cite{TVL03,SaCr03}.
\begin{proposition}[Hedge is the replicator, sampled]\label{prop:mwu}
\Scl The replicator learner fed a payoff stream $\pi(\cdot)$ has
\[ x(t)=\sm\Bigl(\log x(0)+\textstyle\int_0^t\pi\Bigr), \]
and in the variable $x$ it obeys $\dot x_i=x_i(\pi_i-x\cdot\pi)$. One step of Hedge with step size $\eta$ is the
time-$\eta$ map of this flow under a payoff held constant over the step. \chk{B1}
\end{proposition}
\begin{proposition}[Policy gradient and its natural form]\label{prop:pg}
\Scl[\cite{Kak01,Ama98}] For $x=\sm(\theta)$, $\partial(x\cdot\pi)/\partial\theta_i=x_i(\pi_i-x\cdot\pi)$.
Multiplying this gradient by the inverse of the Fisher matrix $\operatorname{diag}(x)-xx^{\T}$, on the
tangent space, gives $\pi-(x\cdot\pi)\one$: natural policy gradient on a softmax policy is the replicator
learner. \chk{B2}
\end{proposition}
\noindent So (PG) differs from (R) by one factor of $x_i$ in the update of the logits. Neural Replicator
Dynamics makes exactly this change in a deep policy-gradient method, bypassing the gradient through the softmax,
and reduces to Hedge in the single-state tabular case \cite{HMO20}. In the variable $x$,
\begin{equation}\label{eq:pgx}
\text{(PG)}\qquad\dot x_i=x_i\Bigl(x_i(\pi_i-\bar\pi)-\sum_jx_j^2(\pi_j-\bar\pi)\Bigr),
\end{equation}
with $\bar\pi=x\cdot\pi$. The extra factor has a cost that is easy to state.
\begin{proposition}[Escape from a bad start]\label{prop:escape}
\Scl[cf.\ \cite{MXSS20}] Two actions, payoff gap $D>0$, the better action at initial probability $\epsilon$. The
time for it to reach $\tfrac12$ is
\begin{align*} t_{\mathrm R}&=\frac1D\log\frac{1-\epsilon}{\epsilon},\\
t_{\mathrm{PG}}&=\frac1{2D}\Bigl(\frac1\epsilon-\frac1{1-\epsilon}+2\log\frac{1-\epsilon}{\epsilon}\Bigr). \end{align*}
\end{proposition}
\begin{proof}
With $x$ the probability of the better action, (R) is $\dot x=Dx(1-x)$ and, by \eqref{eq:pgx}, (PG) is
$\dot x=2Dx^2(1-x)^2$. Integrate.
\end{proof}
\noindent For $D=1$ and $\epsilon=0.1$, $0.01$, $0.001$ the two times are $2.20$ and $6.64$, $4.60$ and
$54.1$, $6.91$ and $506$: ratios $3.0$, $11.8$ and $73.3$. \chk{B3} The replicator learner recovers from a bad
initialization in logarithmic time and policy gradient in time of order $1/\epsilon$. In evolutionary terms (PG)
treats a rare good action as the drift game treats a neutral mutant \cite{EGHC}.
\subsection{Closing the loop}
A learner becomes a dynamical system when its input is supplied. \term{Self-play} in a symmetric game feeds
back $\pi=A\,r(z)$; for (R) this is \eqref{eq:rep}. Two learners in a bimatrix game $(A,B)$ are wired
crosswise, $\pi^1=A\,r(z^2)$ and $\pi^2=B^{\T}r(z^1)$. A league or a population on a graph is a larger diagram.
Formally, open dynamical systems are algebras for an operad of wiring diagrams \cite{SSV20,Mye21}, and a
multi-agent learning problem is a diagram together with a choice of box for each node (\Cref{fig:wiring}).
\begin{figure}[t]
\centering
\begin{tikzpicture}[>=Stealth,every node/.style={font=\footnotesize},scale=0.82]
\node[draw,thick,minimum width=1.25cm,minimum height=0.62cm] (p) at (0,0) {learner};
\node[draw,minimum width=0.7cm,minimum height=0.5cm] (a) at (0,-1.05) {$A$};
\draw[->] (p.east) -- ++(0.45,0) |- node[pos=0.25,right]{$x$} (a.east);
\draw[->] (a.west) -- ++(-0.75,0) |- node[pos=0.25,left]{$\pi$} (p.west);
\node at (0,-1.75) {self-play};
\begin{scope}[xshift=4.6cm]
\node[draw,thick,minimum width=0.95cm,minimum height=0.62cm] (r) at (-1.25,0) {$L_1$};
\node[draw,thick,minimum width=0.95cm,minimum height=0.62cm] (c) at (1.25,0) {$L_2$};
\node[draw,minimum width=0.7cm,minimum height=0.5cm] (a2) at (0,0.62) {$B^{\T}$};
\node[draw,minimum width=0.7cm,minimum height=0.5cm] (b2) at (0,-0.62) {$A$};
\draw[->] (r.north) |- (a2.west);
\draw[->] (a2.east) -| (c.north);
\draw[->] (c.south) |- (b2.east);
\draw[->] (b2.west) -| (r.south);
\node at (0,-1.75) {two players};
\end{scope}
\end{tikzpicture}
\caption{A game is a wiring of learners. The boxes can be changed without changing the diagram, and the
diagram without changing the boxes.}\label{fig:wiring}
\end{figure}
\subsection{Naturality}
\begin{definition}
A family of learners, one for each finite set $S$, is \term{natural} for a class of payoff-preserving maps if
for every $f\colon(S,A)\to(T,B)$ in the class, $f_*$ carries self-play trajectories in $\Dist(S)$ to self-play
trajectories in $\Dist(T)$.
\end{definition}
\noindent Naturality for inclusions says that an action never tried is never missed, and the rest behaves as
if it did not exist. Naturality for clone maps says that listing an action twice, or splitting it into
indistinguishable variants, changes nothing about the total weight of the variants. Naturality for symmetries
is equivariance.
\begin{theorem}[Which learners are natural]\label{thm:natural}
\Sob
\begin{enumerate}[label=(\alph*),leftmargin=1.5em,itemsep=1pt,topsep=2pt]
\item \emph{(R)} and \emph{(H)} are natural for all payoff-preserving maps.
\item \emph{(PG)} is natural for inclusions and symmetries and not for clone maps. If an action of total
probability $y$ is split into copies with probabilities $x_1,\dots,x_k$, the leading term of its drift is
$\sum_cx_c^2$ where the unsplit learner has $y^2$.
\item In the two-action problem of \Cref{prop:escape}, with either action split into $k$ equal copies, the
probability $y$ of the better class obeys $\dot y=(1+\tfrac1k)Dy^2(1-y)^2$. Every learning time is multiplied
by exactly $2k/(k+1)$.
\item \emph{(Q)} is natural for symmetries and not for clone maps. Its rest points are the logit equilibria
$x=\sm(Ax/\tau)$, which depend on the number of copies of each action, and every face of the simplex is
repelling: no action stays unused.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) For (R), with $y=f_*x$, the pushforward identity gives
$\dot y_t=\sum_{f(i)=t}x_i((Ax)_i-x^{\T}Ax)=y_t((By)_t-y^{\T}By)$ for $t$ in the image and $y_t\equiv0$ otherwise.
(H) is a time-$\eta$ map of the same flow with frozen payoffs, and frozen payoffs are also preserved.
(b) In \eqref{eq:pgx} the factor $x_i$ gives invariance of faces. Summing \eqref{eq:pgx} over the copies of
one action gives $(\sum_cx_c^2)(\pi-\bar\pi)-y\sum_jx_j^2(\pi_j-\bar\pi)$, and $\sum_cx_c^2<y^2$ unless all but one
copy is empty. (c) By symmetry the copies stay equal. Substituting $x_c=y/k$ for the split action in
\eqref{eq:pgx} and summing gives the stated equation in both cases; compare with $2Dy^2(1-y)^2$. (d) In the
variable $x$, (Q) in self-play is
\begin{equation}\label{eq:q}
\dot x_i=x_i\Bigl(\tfrac\alpha\tau(\pi_i-\bar\pi)-\alpha\bigl(\log x_i-\textstyle\sum_jx_j\log x_j\bigr)\Bigr),
\end{equation}
whose rest points satisfy $\log x_i=\pi_i/\tau+\mathrm{const}$. The entropy term makes $\dot x_i/x_i\to+\infty$ as
$x_i\to0$, and it distinguishes one action of weight $y$ from two of weight $y/2$, since
$2\cdot\tfrac y2\log\tfrac y2\ne y\log y$.
\end{proof}
\noindent \emph{Computed defects.} For Rock--Paper--Scissors with Rock duplicated, the largest discrepancy over
fifty random states between the pushed-forward field and the field of the three-action game was $4\times10^{-17}$
for (R) and $0.028$ for (PG). \chk{B19} From $\epsilon=0.01$ the policy-gradient time to $\tfrac12$ was $54.1$,
$72.1$, $86.5$ and $96.2$ for $k=1,2,4,8$, and the replicator time $4.60$ for every $k$. \chk{B20} For (Q) the
logit equilibrium of Rock--Paper--Scissors is uniform at every temperature; with Rock duplicated its mass on the
three types is $(0.49,0.29,0.23)$ at $\tau=2$, $(0.40,0.36,0.24)$ at $\tau=0.5$ and $(0.34,0.35,0.31)$ at
$\tau=0.1$. \chk{B21}
\begin{remark}
Large action spaces in practice contain near-duplicates: synonymous tokens, equivalent moves, redundant
members of a league. \Cref{thm:natural} says which update rules are indifferent to them. It is the
finite-dimensional shadow of the invariance of natural gradient under reparametrization \cite{Ama98}, stated for
the maps that matter in games. Entropy bonuses are the clearest case of a method that counts names: at high
temperature the regularized equilibrium gives duplicated Rock half the mass.
\end{remark}
\section{Regret}\label{sec:regret}
A learner has \term{no regret} if, against any stream of payoffs, its average payoff is asymptotically at
least that of the best fixed action. For the replicator learner this is an identity.
\begin{theorem}[Continuous time]\label{thm:regc}
\Scl[\cite{Sor09,KwMe17}] For the replicator learner and any bounded measurable payoff stream,
\[ \int_0^T\bigl(\pi_i-x\cdot\pi\bigr)\,dt=\log\frac{x_i(T)}{x_i(0)}\le\log\frac1{x_i(0)} \]
for every action $i$ and every horizon $T$. From the uniform initial state the regret is at most $\ln n$.
\end{theorem}
\begin{proof}
$\frac d{dt}\log x_i=\pi_i-x\cdot\pi$ by \Cref{prop:mwu}, and $x_i(T)\le1$.
\end{proof}
\noindent The bound does not grow with $T$. Against a stream of six payoffs that alternates between random
vectors and vectors chosen to punish the current favourite, the regret at $T=80$ was $0.24$, against
$\ln6=1.79$, and the identity held to $5\times10^{-8}$. \chk{B4}
\begin{theorem}[Discrete time]\label{thm:regd}
\Scl[\cite{FrSc97,CBL06}] For Hedge with payoffs in $[0,1]$ and $\eta=\sqrt{8\ln n/T}$, the regret after $T$ steps
is at most $\sqrt{T\ln n/2}$.
\end{theorem}
\begin{proof}[Sketch]
One step multiplies the normalizing constant by $\E_xe^{\eta\pi}$, and Hoeffding's lemma gives
$\log\E_xe^{\eta(\pi-x\cdot\pi)}\le\eta^2/8$. Summing, the discrete learner satisfies the identity of
\Cref{thm:regc} up to an excess of at most $\eta/8$ per step: the regret is at most $\ln n/\eta+\eta T/8$.
\end{proof}
\noindent Over forty runs with $n\in\{2,5,10\}$, $T\in\{200,1000\}$, half against random payoffs and half
against an adversary, the largest ratio of regret to bound was $0.50$. \chk{B5} The whole difference between
$\ln n$ and $\sqrt T$ is the term
\begin{equation}\label{eq:excess}
\Lambda_\eta(x,\pi)=\log\E_x\,e^{\eta(\pi-x\cdot\pi)}=\tfrac12\eta^2\Var_x(\pi)+O(\eta^3),
\end{equation}
the price of holding the payoff fixed for a step. The same term decides the fate of discrete learning in
games.
\begin{corollary}\label{cor:avg}
\Scl If both players of a zero-sum game with payoffs in an interval of length $1$ use Hedge with the step of
\Cref{thm:regd}, their time-averaged strategies form a $2\sqrt{\ln n/(2T)}$-equilibrium. In a general game the
time-averaged joint play approaches the set of coarse correlated equilibria.
\end{corollary}
\noindent In self-play of Rock--Paper--Scissors, whose payoffs have range $2$, the time average over
$T=20000$ steps could be exploited for $0.0015$, against the corresponding bound $0.0105$. \chk{B8}
\section{Zero-sum games: recurrence, divergence, repair}\label{sec:zerosum}
Regret bounds speak of averages. What the strategies themselves do is another matter, and it is the matter
that concerns adversarial training \cite{Goo14}, where the iterate and not the average is the model.
\subsection{The flow cycles}
\begin{theorem}[Conservation and recurrence]\label{thm:rec}
\Sli[\cite{HoSi98,PiSh14,MPP18}] Let $A$ be antisymmetric with an interior equilibrium $x^*$. Along replicator
self-play $\KL{x^*}{x(t)}$ is constant. For two replicator learners in a zero-sum game with interior equilibrium
$(x^*,y^*)$ the sum $\KL{x^*}{x}+\KL{y^*}{y}$ is constant, and the flow is Poincar\'e recurrent: almost every
trajectory returns arbitrarily close to its starting point infinitely often.
\end{theorem}
\begin{proof}[Proof of conservation]
$\frac d{dt}\KL{x^*}{x}=-\sum_ix^*_i(\pi_i-\bar\pi)=-(x^*)^{\T}Ax+x^{\T}Ax$. Antisymmetry gives $x^{\T}Ax=0$, and
$(x^*)^{\T}Ax=-x^{\T}Ax^*=0$ because $Ax^*$ is constant on the support of an interior equilibrium and that
constant is the value $0$.
\end{proof}
\noindent Conservation says that such a game does no work on the learner. An antisymmetric matrix has no
symmetric part, so the gradient component of the replicator field vanishes \cite{EGHC}; and when the equilibrium
is uniform the game is purely cyclic in the sense of the Hodge decomposition of \Cref{sec:eval}.
\subsection{The discretization spirals out}
\begin{proposition}[Exact drift of Hedge]\label{prop:drift}
\Srf[\cite{BaPi18}] In Hedge self-play of an antisymmetric game with interior equilibrium $x^*$, each step
changes the relative entropy by exactly
\[ \KL{x^*}{x^+}-\KL{x^*}{x}=\Lambda_\eta(x,Ax)\ \ge0, \]
with equality only if all actions in use earn the same. Hence $\KL{x^*}{x_t}$ is non-decreasing, and it increases
by about $\tfrac12\eta^2\Var_x(Ax)$ per step.
\end{proposition}
\begin{proof}
$x^+_i=x_ie^{\eta\pi_i}/Z$ with $Z=\E_xe^{\eta\pi}$, so the difference is $-\eta\,x^*\!\cdot\pi+\log Z$. As in
\Cref{thm:rec}, $x^*\!\cdot\pi=0=x\cdot\pi$, so the difference is $\log Z=\Lambda_\eta$, and Jensen's inequality
gives the sign.
\end{proof}
\noindent At the state $(0.5,0.3,0.2)$ of Rock--Paper--Scissors the increase per step was $5.01\times10^{-5}$,
$2.01\times10^{-4}$ and $8.08\times10^{-4}$ for $\eta=0.05$, $0.1$, $0.2$, against
$\tfrac12\eta^2\Var=5.00\times10^{-5}$, $2.00\times10^{-4}$, $8.00\times10^{-4}$. \chk{B6} With $\eta=0.1$ the
relative entropy increased at every step and some action fell below probability $0.01$ after $1451$ steps.
\chk{B7} The quantity \eqref{eq:excess} that costs a learner $\sqrt T$ in regret is, in a game, energy pumped
into the cycle at every step. The time average still converges (\Cref{cor:avg}); the iterate leaves.
\subsection{Three repairs, in three places}
\begin{proposition}[Optimism]\label{prop:opt}
\Sli[\cite{RaSr13,DISZ18,DaPa19}] The optimistic learner \emph{(O)}, which uses $2\pi_t-\pi_{t-1}$ as its
prediction of the next payoff, converges in the last iterate in zero-sum games with a unique equilibrium, for
small enough $\eta$.
\end{proposition}
\noindent From the same initial state and $\eta=0.1$ the relative entropy to equilibrium fell from $0.070$ to
$1.5\times10^{-10}$ in $6000$ steps. \chk{B9}
\begin{theorem}[Entropy regularization, with its rate]\label{thm:q}
\Sob Let $A$ be antisymmetric with the uniform state $u$ as equilibrium. Under the dynamics \eqref{eq:q},
for every $\tau>0$,
\[ \frac d{dt}\KL ux=-\alpha\bigl(\KL ux+\KL xu\bigr), \]
and therefore $\KL u{x(t)}\le e^{-\alpha t}\KL u{x(0)}$.
\end{theorem}
\begin{proof}
$\frac d{dt}\KL ux=-\sum_iu_i\dot x_i/x_i$. The payoff terms vanish as in \Cref{thm:rec}. The entropy terms
give $\alpha\bigl(\sum_iu_i\log x_i-\sum_jx_j\log x_j\bigr)$, and
$\KL ux+\KL xu=\sum_jx_j\log x_j-\sum_iu_i\log x_i$.
\end{proof}
\noindent Convergence of $Q$-learning dynamics in competitive games is known in much greater generality
\cite{LPS21}; the point of \Cref{thm:q} is the exact identity, in which the rate is the learning rate $\alpha$
and does not depend on the game or the temperature. With $\alpha=0.7$ the relative entropy fell from $0.070$ to
$6.1\times10^{-8}$ at $t=10$, against the bound $6.4\times10^{-5}$. \chk{B10} In general the limit is a logit
equilibrium and not a Nash equilibrium \chk{B11}, and by \Cref{thm:natural}(d) it is not natural.
\begin{remark}[Where each repair acts]\label{rem:repairs}
Averaging changes the \emph{readout}: the box and the wiring are untouched and one reports $\frac1T\sum x_t$.
Optimism changes the \emph{box}: the learner carries one more step of memory. Regularization changes the
\emph{wiring}: \eqref{eq:q} is the replicator learner in the game with the additional self-payoff
$-\tau\log x_i$, a loop from each learner's output to its own input, up to a constant change of time scale. In
evolutionary terms these are
time-averaging of frequencies, anticipation, and mutation \cite{TVL03}. The trade is visible in the table of
\Cref{sec:summary}: averaging keeps naturality and loses the iterate; regularization gains a rate and loses
both naturality and the equilibrium.
\end{remark}
\section{Evaluating agents}\label{sec:eval}
Training by self-play, leagues and population-based methods all need to say which of a set of agents is
better \cite{Vin19,LZG17}. The data is an \term{evaluation matrix}: an antisymmetric $K$ with $K_{ij}$ the
advantage of agent $i$ over agent $j$, for instance the log-odds of winning. This is a symmetric zero-sum game
on the set of agents, and an \term{evaluation method} assigns to it a score vector or a distribution on agents.
\begin{theorem}[Ratings and cycles]\label{thm:hodge}
\Sli[\cite{JLYY11,BTPG18}] Every antisymmetric $K$ decomposes uniquely and orthogonally as
\[ K_{ij}=(r_i-r_j)+C_{ij},\quad r_i=\tfrac1n\textstyle\sum_jK_{ij},\quad\textstyle\sum_jC_{ij}=0. \]
The first part satisfies $G_{ij}+G_{jk}+G_{ki}=0$ on every triple, and every matrix satisfying this is of that
form. The second part carries the same curl $K_{ij}+K_{jk}+K_{ki}$ as $K$.
\end{theorem}
\noindent The ratings $r$ are the least-squares fit of an Elo-like scale; the \term{transitivity index}
$\|G\|^2/\|K\|^2$ measures how much of the data a scale can explain. For five agents with planted ratings and one
planted cycle of strength $0.5$ among three of them, the decomposition returned the ratings
$(0.7,0.3,0,-0.3,-0.7)$, index $0.885$ and curl $1.5$ on the planted triple. \chk{B12} That closed implies
exact is the vanishing of the first cohomology of a simplex.
\begin{proposition}[Incomplete comparisons]\label{prop:harmonic}
\Sli[\cite{JLYY11}] If only some pairs of agents have been compared, the decomposition acquires a third,
harmonic component: circulations around loops of the comparison graph that bound no triple of mutually
compared agents. It is locally consistent and globally not a rating, and its dimension is the first Betti
number of the comparison complex.
\end{proposition}
\noindent Four agents compared only around a square, each beating the next by one unit, give the smallest
example: zero divergence, no triangles, nothing to fit. \chk{B15} Sparse tournament schedules create such
classes by design.
The naturality criterion of \Cref{sec:setting} applies to evaluation unchanged. A league that contains two
copies of an agent, or ten near-copies obtained from one checkpoint, should not be ranked differently for
that reason.
\begin{proposition}[Naturality of evaluation methods]\label{prop:evalnat}
\Sob Let $f\colon(S,K)\to(T,K')$ be a clone map between evaluation games.
\begin{enumerate}[label=(\alph*),leftmargin=1.5em,itemsep=1pt,topsep=2pt]
\item Mean score against the field, $s_i=\frac1n\sum_jK_{ij}$, is not natural: duplicating an agent raises the
score of everything that beats it.
\item The set of Nash equilibria is natural: $x$ is an equilibrium of $(S,K)$ if and only if $f_*x$ is one of
$(T,K')$. When the equilibrium of $(T,K')$ is unique, every rule that selects an equilibrium is therefore
natural on types. Invariance to redundant agents is the stated aim of Nash averaging \cite{BTPG18}.
\item The time average of replicator self-play is natural, and when $(T,K')$ has a unique interior equilibrium
it assigns to each type its equilibrium mass.
\item The stationary distribution of the small-mutation chain in which a monomorphic population is invaded by
a mutant drawn uniformly from the other \emph{names}, with transition probability given by the fixation
probability, is not natural.
\end{enumerate}
\end{proposition}
\begin{proof}
(a) and (d) by example, below. (b) Payoffs against $f_*x$ at the types are the payoffs against $x$ at the
agents, and a clone map omits no type \cite[Prop.~1.10]{EGHC}. (c) By \Cref{thm:natural}(a), $f_*$ carries
trajectories to trajectories and hence time averages to time averages; the time-average theorem for the
replicator equation \cite{HoSi98} identifies the limit.
\end{proof}
\noindent \emph{The example.} Rock, a second copy of Rock, Paper, Scissors. Mean scores are $(0,0,0.25,-0.25)$:
Paper looks best and Scissors worst, although the game is symmetric among the three types. The
time-averaged replicator mass on the types is $(0.3334,0.3333,0.3333)$. \chk{B13} The small-mutation chain,
uniform on plain Rock--Paper--Scissors, gives $(0.2,0.2,0.4,0.2)$ on the four names: Rock's type has mass $0.4$,
Paper $0.4$ and Scissors $0.2$. \chk{B14} The reason is visible in the chain: a population of Scissors is now
invaded twice as often.
\begin{remark}[Chain components]\label{rem:mcc}
The chain in (d) is in the manner of $\alpha$-Rank \cite{OPP19}, which ranks agents by the stationary
distribution of a Markov chain on strategy profiles built from fixation probabilities, and which grounds the
construction in Conley's theory: the long-run behaviour of a dynamical system is carried by its chain
recurrent set, and Markov--Conley chains are a finite model of the sink components. This is the right
generalization of equilibrium for games that cycle \cite{PaPi19}, and the dynamical content is natural: chain
components of a natural flow map into chain components. Statement (d) concerns only the mutation kernel of
our small implementation, which counts names; we make no claim about $\alpha$-Rank as published or as used.
A kernel that draws mutants by type would restore invariance, at the price of knowing the types.
\end{remark}
\section{Finite populations of learners}\label{sec:pop}
Population-based training keeps $N$ agents, lets better ones overwrite worse ones, and perturbs the copies
\cite{Jad17}. With copying in proportion to fitness and one replacement at a time this is the Moran process
of population genetics, and its classical theory applies \cite{Now06,EGHC}.
\begin{theorem}[Fixation and the one-third law]\label{thm:third}
\Sli[\cite{NSTF04}] Let two variants play a coordination game: each does best against its own kind, and
the newcomer must exceed a fraction $x^*$ of the population before it is favoured. Under weak selection in a
large population a single newcomer takes over with probability greater than the neutral value $1/N$ if and
only if $x^*<\tfrac13$.
\end{theorem}
\noindent With $N=400$ and selection intensity $10^{-3}$, $N\rho$ was $1.021$, $1.009$, $0.991$ and $0.958$ at
$x^*=0.25$, $0.30$, $0.36$ and $0.45$. \chk{B16} The deterministic dynamics says that a new convention, protocol
or shared representation cannot invade from rarity at all when it needs partners; the finite population says
it can, by drift, and when it is more likely than chance to do so.
\begin{theorem}[The copying topology matters]\label{thm:graph}
\Sli[\cite{LHN05}] Let offspring of agent $i$ replace agent $j$ with probability $w_{ij}$. The fixation
probability of a variant of relative fitness $r$ equals the well-mixed value $(1-r^{-1})/(1-r^{-N})$ for all $r$
if and only if $W$ is doubly stochastic. A star amplifies selection. On a directed line, in which each
agent copies only its predecessor, fixation has probability $1/N$ whatever the fitness.
\end{theorem}
\noindent For $N=6$ and $r=1.2$ the exact values are $0.167$ on the line, $0.251$ well-mixed and $0.287$ on the
star. \chk{B17} The design reading is direct. Symmetric exchange between agents, of any shape, neither helps
nor hurts selection. A hub that many agents copy from and that copies from them sharpens it. A strictly
feed-forward pipeline of copying removes selection altogether: which variant ends up everywhere is decided by
where it started.
\begin{remark}[Populations of language-model agents]\label{rem:llm}
Recent work studies populations of agents built on large language models with exactly these tools: replicator
dynamics over agent types in open-source games \cite{OSG25}, evolutionarily stable strategies as a design
target for trust mechanisms \cite{EvT25}, group selection and replicator--mutator models for the evolution of
cooperative prompts \cite{GSP26}, evolutionary game theory as a probe of strategic behaviour \cite{SIL25}, and a
survey of the area \cite{XLLS26}. The titles and identifiers of these five items were checked on 17 November
2026; we have not evaluated their results. Our contribution to that programme is a set of constraints.
Two prompts that induce the same behaviour are clones, so an analysis of a prompt population should be natural
in the sense of \Cref{sec:setting}; stability should be asked of sets of prompts and not of single ones,
because stability of a point is destroyed by cloning and stability of a set is not \cite[Thm.~2.8]{EGHC}; and
conclusions drawn from the replicator equation about invasion by a single new agent are subject to the
correction of \Cref{thm:third}.
\end{remark}
\section{Inference}\label{sec:bayes}
\begin{proposition}[Bayes is the replicator]\label{prop:bayes}
\Scl[\cite{Sha09,Har09b}] Let $S$ be a set of hypotheses with likelihoods $\ell_h$. Bayesian updating of a prior
$x$ on an observation $o$ is one step of Hedge with $\eta=1$ and payoff $\pi_h=\log\ell_h(o)$. \chk{B18}
\end{proposition}
\noindent Bayesian inference is the replicator learner fed log-likelihoods, and the dictionary is exact.
Naturality for clone maps is the statement that splitting a hypothesis into two indistinguishable ones, with the
prior mass divided, changes no posterior probability of any event. The regret identity of \Cref{thm:regc} is
the statement that the log-loss of the Bayesian mixture exceeds that of the best hypothesis by at most
$\log(1/x_h(0))$. And the elimination of dominated strategies with its rate \cite{EGHC} is posterior
concentration: if observations come from $h^*$, the log-odds of any other $h$ against $h^*$ fall at the
expected rate $\KL{\ell_{h^*}}{\ell_h}$ per observation.
\section{Summary of results}\label{sec:summary}
\begin{table*}[t]
\centering\small
\begin{tabular}{@{}>{\raggedright}p{0.16\textwidth}>{\raggedright}p{0.3\textwidth}>{\raggedright}p{0.34\textwidth}l@{}}
\toprule
Topic & Qualitative statement & Quantitative statement & Checks\\
\midrule
Hedge and replicator & one is the other, sampled & exact at every step size & B1\\
Policy gradient & replicator with an extra factor $x_i$ & escape time $\sim1/(2D\epsilon)$ against $\log(1/\epsilon)/D$; ratios $3.0$, $11.8$, $73.3$ & B2, B3\\
Duplicated actions & (R), (H) natural; (PG), (Q) not & slowdown $2k/(k+1)$; defect $0.028$ & B19--B21\\
Regret & no regret & $\le\ln n$ continuous; $\le\sqrt{T\ln n/2}$ discrete; observed ratio $\le0.50$ & B4, B5\\
Zero-sum, flow & relative entropy conserved; recurrence & numerical drift $1.3\times10^{-15}$ over time $40$ & R4 of \cite{EGHC}\\
Zero-sum, Hedge & iterate leaves, average converges & $+\tfrac12\eta^2\Var$ per step; boundary after $1451$ steps; exploitability $0.0015$ & B6--B8\\
Optimism & last iterate converges & $0.070\to1.5\times10^{-10}$ in $6000$ steps & B9\\
Entropy regularization & converges; not natural; biased & rate at least $\alpha$, by an identity; type masses $(0.40,0.36,0.24)$ at $\tau=0.5$ & B10, B11, B21\\
Evaluation & ratings plus cycles plus harmonic & index $0.885$; curl $1.5$ & B12, B15\\
Redundant agents & averaging and name-mutation chain not natural; replicator average natural & $(0,0,0.25,-0.25)$; $(0.4,0.4,0.2)$; $(\tfrac13,\tfrac13,\tfrac13)$ & B13, B14\\
New conventions & can invade by drift & iff barrier $<\tfrac13$; $N\rho=1.021,\dots,0.958$ & B16\\
Copying topology & amplifiers and suppressors & $0.167<0.251<0.287$ & B17\\
Inference & Bayes is Hedge at $\eta=1$ & exact & B18\\
\bottomrule
\end{tabular}
\caption{Qualitative statements and their quantitative forms. All numbers are produced by the companion
script.}\label{tab:summary}
\end{table*}
\Cref{tab:summary} collects the results. Read down the middle column it is a list of facts most of which are
individually known; read across, each has a constant attached; and the recurring entry is naturality, which
sorts the methods into those that depend on how actions and agents are named and those that do not.
\section{Limits}\label{sec:limits}
\emph{Scope.} Everything here is tabular: a finite set of actions and exact payoffs. Function approximation,
sampling noise, and partial observation are outside it; they are where methods such as \cite{HMO20} do their
work. The numbers come from games with at most six actions and populations of at most a few hundred, chosen
to be exactly computable, and are illustrations of the statements and not benchmarks. No experiment on a
trained model was run.
\emph{Novelty.} The identifications of \Cref{prop:mwu,prop:pg,prop:bayes}, the regret bounds, recurrence, the
Hodge decomposition, the one-third law and the isothermal theorem are from the literature. The observations
are \Cref{thm:natural}, the identity of \Cref{thm:q}, \Cref{prop:evalnat}, and the placing of the three
repairs in \Cref{rem:repairs}. They are short, and parts of them are likely known in other forms; the
invariance of Nash averaging under redundancy, for one, is a stated motivation of \cite{BTPG18}.
\emph{How categorical.} Modestly. The category $\Game$, the functor to dynamical systems, and the operad of
wiring diagrams are ordinary category theory; the only cohomology is in degree one; the homotopy theory is in
the Conley index behind \Cref{rem:mcc}, which we invoke and do not compute. The language earns its place by one
question, whether a construction commutes with the maps, which turned out to separate methods that the
literature usually compares on other grounds.
\emph{References.} The bibliography was assembled from memory, except for \cite{OPP19,HMO20} and the five items
of \Cref{rem:llm}, whose bibliographic details were checked online on 17 November 2026. It should be verified
before it is relied on.
\emph{Authorship.} This paper was drafted with the assistance of an AI system (Anthropic's Claude), which also
wrote and ran the companion script.
\paragraph{Open problems.} \Spg (1) Characterize the learners that are natural for clone maps among those
derived from a regularizer; we expect entropy on types and not on names to be forced. (2) Give the
rate in \Cref{thm:q} for a non-uniform equilibrium, where the limit moves with $\tau$. (3) Extend
\Cref{prop:evalnat}(c) from time averages to chain components, so that a ranking by recurrent classes is
natural by construction. (4) Find the analogue of the isothermal theorem for populations of learners whose
fitness depends on the population, where the graph and the game interact.
\appendix
\section{The companion script}\label{app:checks}
\texttt{verify-evolutionary-learning.py} uses only the Python standard library, fixes its random seed, runs
in under half a minute, and exits with status zero if and only if its 21 checks pass. Flows are integrated by
a fourth-order Runge--Kutta scheme; fixation probabilities are exact. The checks test instances and recompute
the numbers quoted in the text. They do not prove the theorems.
\begin{center}\footnotesize
\begin{tabular}{@{}lp{0.82\columnwidth}@{}}
\toprule
B1 & Hedge equals the time-$\eta$ replicator flow, 30 random cases\\
B2 & policy gradient in logits; Fisher preconditioning\\
B3 & escape times, \Cref{prop:escape}, against numerical integration\\
B4 & continuous-time regret identity and bound\\
B5 & discrete regret bound, 40 runs, random and adversarial\\
B6 & exact drift of Hedge, \Cref{prop:drift}, three step sizes\\
B7 & monotone divergence to the boundary\\
B8 & exploitability of the time average\\
B9 & optimistic Hedge, last iterate\\
B10 & identity and rate of \Cref{thm:q}\\
B11 & rest points of \eqref{eq:q} are logit equilibria\\
B12 & Hodge decomposition of a planted example\\
B13 & mean score and replicator average under cloning\\
B14 & small-mutation chain: uniform, dominated, cloned\\
B15 & a harmonic class on a square of comparisons\\
B16 & one-third law, exact fixation, $N=400$\\
B17 & line, well-mixed, star: exact fixation, $N=6$\\
B18 & Bayes equals Hedge with $\eta=1$\\
B19 & naturality defect of (R) and (PG) under cloning\\
B20 & slowdown $2k/(k+1)$ for $k=1,2,4,8$\\
B21 & logit equilibrium with a cloned action, three temperatures\\
\bottomrule
\end{tabular}
\end{center}
\begin{thebibliography}{XLLS26}\footnotesize\setlength{\itemsep}{0.5pt}
\newcommand{\bi}[1]{\bibitem[#1]{#1}}
\bi{Alex23} J.~M.~Alexander, \emph{Evolutionary Game Theory}, Elements in Decision Theory and Philosophy, Cambridge University Press, 2023.
\bi{Ama98} S.~Amari, Natural gradient works efficiently in learning, \emph{Neural Comput.} 10 (1998), 251--276.
\bi{BaPi18} J.~P.~Bailey, G.~Piliouras, Multiplicative weights update in zero-sum games, in \emph{Proc.\ ACM Conf.\ Economics and Computation (EC)}, 2018, 321--338.
\bi{BoSa97} T.~B\"orgers, R.~Sarin, Learning through reinforcement and replicator dynamics, \emph{J.~Econom.\ Theory} 77 (1997), 1--14.
\bi{BTPG18} D.~Balduzzi, K.~Tuyls, J.~P\'erolat, T.~Graepel, Re-evaluating evaluation, in \emph{Advances in Neural Information Processing Systems} 31, 2018.
\bi{BTHK15} D.~Bloembergen, K.~Tuyls, D.~Hennes, M.~Kaisers, Evolutionary dynamics of multi-agent learning: a survey, \emph{J.~Artificial Intelligence Res.} 53 (2015), 659--697.
\bi{CBL06} N.~Cesa-Bianchi, G.~Lugosi, \emph{Prediction, Learning, and Games}, Cambridge University Press, 2006.
\bi{CGG22} G.~S.~H.~Cruttwell, B.~Gavranovi\'c, N.~Ghani, P.~Wilson, F.~Zanasi, Categorical foundations of gradient-based learning, in \emph{European Symposium on Programming (ESOP)}, 2022.
\bi{DaPa19} C.~Daskalakis, I.~Panageas, Last-iterate convergence: zero-sum games and constrained min-max optimization, in \emph{Innovations in Theoretical Computer Science (ITCS)}, 2019.
\bi{DISZ18} C.~Daskalakis, A.~Ilyas, V.~Syrgkanis, H.~Zeng, Training GANs with optimism, in \emph{Int.\ Conf.\ Learning Representations (ICLR)}, 2018.
\bi{EGHC} Anonymous, \emph{Evolutionary Game Theory in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.
\bi{EvT25} Wang et al., Ev-Trust, a trust mechanism for multi-agent systems of language models built on evolutionary stability, preprint, arXiv:2512.16167, 2025. Full title and author list not recorded when the listing was checked.
\bi{FrSc97} Y.~Freund, R.~E.~Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, \emph{J.~Comput.\ System Sci.} 55 (1997), 119--139.
\bi{FST19} B.~Fong, D.~I.~Spivak, R.~Tuy\'eras, Backprop as functor: a compositional perspective on supervised learning, in \emph{LICS 2019}, IEEE, 2019.
\bi{Goo14} I.~Goodfellow, J.~Pouget-Abadie, M.~Mirza, B.~Xu, D.~Warde-Farley, S.~Ozair, A.~Courville, Y.~Bengio, Generative adversarial nets, in \emph{Advances in Neural Information Processing Systems} 27, 2014.
\bi{GSP26} Group selection to evolve cooperative prompts in populations of LLM agents, listed for NeurIPS 2026; authors not recorded when the listing was checked.
\bi{Har09b} M.~Harper, The replicator equation as an inference dynamic, preprint, arXiv:0911.1763.
\bi{HMO20} D.~Hennes, D.~Morrill, S.~Omidshafiei, R.~Munos, J.~Perolat, M.~Lanctot, A.~Gruslys, J.-B.~Lespiau, P.~Parmas, E.~Du\'e\~nez-Guzm\'an, K.~Tuyls, Neural replicator dynamics: multiagent learning via hedging policy gradients, in \emph{Proc.\ AAMAS}, 2020; arXiv:1906.00190.
\bi{HoSi98} J.~Hofbauer, K.~Sigmund, \emph{Evolutionary Games and Population Dynamics}, Cambridge University Press, 1998.
\bi{Jad17} M.~Jaderberg, V.~Dalibard, S.~Osindero, W.~M.~Czarnecki, J.~Donahue, A.~Razavi, O.~Vinyals, T.~Green, I.~Dunning, K.~Simonyan, C.~Fernando, K.~Kavukcuoglu, Population based training of neural networks, preprint, arXiv:1711.09846.
\bi{JLYY11} X.~Jiang, L.-H.~Lim, Y.~Yao, Y.~Ye, Statistical ranking and combinatorial Hodge theory, \emph{Math.\ Program.} 127 (2011), 203--244.
\bi{Kak01} S.~Kakade, A natural policy gradient, in \emph{Advances in Neural Information Processing Systems} 14, 2001.
\bi{KwMe17} J.~Kwon, P.~Mertikopoulos, A continuous-time approach to online optimization, \emph{J.~Dyn.\ Games} 4 (2017), 125--148.
\bi{LHN05} E.~Lieberman, C.~Hauert, M.~A.~Nowak, Evolutionary dynamics on graphs, \emph{Nature} 433 (2005), 312--316.
\bi{LPS21} S.~Leonardos, G.~Piliouras, K.~Spendlove, Exploration-exploitation in multi-agent competition: convergence with bounded rationality, in \emph{Advances in Neural Information Processing Systems} 34, 2021.
\bi{LZG17} M.~Lanctot, V.~Zambaldi, A.~Gruslys, A.~Lazaridou, K.~Tuyls, J.~P\'erolat, D.~Silver, T.~Graepel, A unified game-theoretic approach to multiagent reinforcement learning, in \emph{Advances in Neural Information Processing Systems} 30, 2017.
\bi{MPP18} P.~Mertikopoulos, C.~Papadimitriou, G.~Piliouras, Cycles in adversarial regularized learning, in \emph{Proc.\ ACM-SIAM Symp.\ Discrete Algorithms (SODA)}, 2018, 2703--2717.
\bi{MSP73} J.~Maynard Smith, G.~R.~Price, The logic of animal conflict, \emph{Nature} 246 (1973), 15--18.
\bi{MXSS20} J.~Mei, C.~Xiao, C.~Szepesv\'ari, D.~Schuurmans, On the global convergence rates of softmax policy gradient methods, in \emph{Proc.\ Int.\ Conf.\ Machine Learning (ICML)}, 2020.
\bi{Mye21} D.~J.~Myers, Double categories of open dynamical systems (extended abstract), in \emph{Applied Category Theory 2020}, EPTCS 333, 2021, 154--167.
\bi{Now06} M.~A.~Nowak, \emph{Evolutionary Dynamics: Exploring the Equations of Life}, Harvard University Press, 2006.
\bi{NSTF04} M.~A.~Nowak, A.~Sasaki, C.~Taylor, D.~Fudenberg, Emergence of cooperation and evolutionary stability in finite populations, \emph{Nature} 428 (2004), 646--650.
\bi{OPP19} S.~Omidshafiei, C.~Papadimitriou, G.~Piliouras, K.~Tuyls, M.~Rowland, J.-B.~Lespiau, W.~M.~Czarnecki, M.~Lanctot, J.~Perolat, R.~Munos, $\alpha$-Rank: multi-agent evaluation by evolution, \emph{Sci.\ Rep.} 9 (2019); arXiv:1903.01373.
\bi{OSG25} Evaluating LLMs in open-source games, preprint, arXiv:2512.00371, 2025; authors not recorded when the listing was checked.
\bi{PaPi19} C.~Papadimitriou, G.~Piliouras, Game dynamics as the meaning of a game, \emph{ACM SIGecom Exchanges} 16 (2019), no.~2, 53--63.
\bi{PiSh14} G.~Piliouras, J.~S.~Shamma, Optimization despite chaos: convex relaxations to complex limit sets via Poincar\'e recurrence, in \emph{Proc.\ ACM-SIAM Symp.\ Discrete Algorithms (SODA)}, 2014, 861--873.
\bi{RaSr13} A.~Rakhlin, K.~Sridharan, Optimization, learning, and games with predictable sequences, in \emph{Advances in Neural Information Processing Systems} 26, 2013.
\bi{SaCr03} Y.~Sato, J.~P.~Crutchfield, Coupled replicator equations for the dynamics of learning in multiagent systems, \emph{Phys.\ Rev.\ E} 67 (2003), 015206.
\bi{Sha09} C.~R.~Shalizi, Dynamics of Bayesian updating with dependent data and misspecified models, \emph{Electron.\ J.~Stat.} 3 (2009), 1039--1074.
\bi{SIL25} Strategic intelligence in large language models: evidence from evolutionary game theory, preprint, arXiv:2507.02618, 2025; authors not recorded when the listing was checked.
\bi{Sor09} S.~Sorin, Exponential weight algorithm in continuous time, \emph{Math.\ Program.} 116 (2009), 513--528.
\bi{SSV20} P.~Schultz, D.~I.~Spivak, C.~Vasilakopoulou, Dynamical systems and sheaves, \emph{Appl.\ Categ.\ Structures} 28 (2020), 1--57.
\bi{TaJo78} P.~D.~Taylor, L.~B.~Jonker, Evolutionarily stable strategies and game dynamics, \emph{Math.\ Biosci.} 40 (1978), 145--156.
\bi{TVL03} K.~Tuyls, K.~Verbeeck, T.~Lenaerts, A selection-mutation model for Q-learning in multi-agent systems, in \emph{Proc.\ AAMAS}, 2003, 693--700.
\bi{Vin19} O.~Vinyals et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, \emph{Nature} 575 (2019), 350--354.
\bi{XLLS26} Xie, Liu, Lan, Szolnoki, Evolutionary games in LLM-agent societies: early milestones and perspectives, \emph{Chaos Solitons Fractals} 213 (2026), 119158. Initials not recorded when the listing was checked.
\end{thebibliography}
\end{document}