Indexed text attachment
evolutionary-game-theory-in-machine-learning-and-ai.tex
Source for Evolutionary Game Theory in Machine Learning and AI: Naturality, Regret, and Recurrence
View paper · Download original file
Lines 625–724 of 725. Source text is displayed literally.
625\emph{References.} The bibliography was assembled from memory, except for \cite{OPP19,HMO20} and the five items626of \Cref{rem:llm}, whose bibliographic details were checked online on 17 November 2026. It should be verified627before it is relied on.628 629\emph{Authorship.} This paper was drafted with the assistance of an AI system (Anthropic's Claude), which also630wrote and ran the companion script.631 632\paragraph{Open problems.} \Spg (1) Characterize the learners that are natural for clone maps among those633derived from a regularizer; we expect entropy on types and not on names to be forced. (2) Give the634rate in \Cref{thm:q} for a non-uniform equilibrium, where the limit moves with $\tau$. (3) Extend635\Cref{prop:evalnat}(c) from time averages to chain components, so that a ranking by recurrent classes is636natural by construction. (4) Find the analogue of the isothermal theorem for populations of learners whose637fitness depends on the population, where the graph and the game interact.638 639\appendix640\section{The companion script}\label{app:checks}641 642\texttt{verify-evolutionary-learning.py} uses only the Python standard library, fixes its random seed, runs643in under half a minute, and exits with status zero if and only if its 21 checks pass. Flows are integrated by644a fourth-order Runge--Kutta scheme; fixation probabilities are exact. The checks test instances and recompute645the numbers quoted in the text. They do not prove the theorems.646 647\begin{center}\footnotesize648\begin{tabular}{@{}lp{0.82\columnwidth}@{}}649\toprule650B1 & Hedge equals the time-$\eta$ replicator flow, 30 random cases\\651B2 & policy gradient in logits; Fisher preconditioning\\652B3 & escape times, \Cref{prop:escape}, against numerical integration\\653B4 & continuous-time regret identity and bound\\654B5 & discrete regret bound, 40 runs, random and adversarial\\655B6 & exact drift of Hedge, \Cref{prop:drift}, three step sizes\\656B7 & monotone divergence to the boundary\\657B8 & exploitability of the time average\\658B9 & optimistic Hedge, last iterate\\659B10 & identity and rate of \Cref{thm:q}\\660B11 & rest points of \eqref{eq:q} are logit equilibria\\661B12 & Hodge decomposition of a planted example\\662B13 & mean score and replicator average under cloning\\663B14 & small-mutation chain: uniform, dominated, cloned\\664B15 & a harmonic class on a square of comparisons\\665B16 & one-third law, exact fixation, $N=400$\\666B17 & line, well-mixed, star: exact fixation, $N=6$\\667B18 & Bayes equals Hedge with $\eta=1$\\668B19 & naturality defect of (R) and (PG) under cloning\\669B20 & slowdown $2k/(k+1)$ for $k=1,2,4,8$\\670B21 & logit equilibrium with a cloned action, three temperatures\\671\bottomrule672\end{tabular}673\end{center}674 675\begin{thebibliography}{XLLS26}\footnotesize\setlength{\itemsep}{0.5pt}676\newcommand{\bi}[1]{\bibitem[#1]{#1}}677\bi{Alex23} J.~M.~Alexander, \emph{Evolutionary Game Theory}, Elements in Decision Theory and Philosophy, Cambridge University Press, 2023.678\bi{Ama98} S.~Amari, Natural gradient works efficiently in learning, \emph{Neural Comput.} 10 (1998), 251--276.679\bi{BaPi18} J.~P.~Bailey, G.~Piliouras, Multiplicative weights update in zero-sum games, in \emph{Proc.\ ACM Conf.\ Economics and Computation (EC)}, 2018, 321--338.680\bi{BoSa97} T.~B\"orgers, R.~Sarin, Learning through reinforcement and replicator dynamics, \emph{J.~Econom.\ Theory} 77 (1997), 1--14.681\bi{BTPG18} D.~Balduzzi, K.~Tuyls, J.~P\'erolat, T.~Graepel, Re-evaluating evaluation, in \emph{Advances in Neural Information Processing Systems} 31, 2018.682\bi{BTHK15} D.~Bloembergen, K.~Tuyls, D.~Hennes, M.~Kaisers, Evolutionary dynamics of multi-agent learning: a survey, \emph{J.~Artificial Intelligence Res.} 53 (2015), 659--697.683\bi{CBL06} N.~Cesa-Bianchi, G.~Lugosi, \emph{Prediction, Learning, and Games}, Cambridge University Press, 2006.684\bi{CGG22} G.~S.~H.~Cruttwell, B.~Gavranovi\'c, N.~Ghani, P.~Wilson, F.~Zanasi, Categorical foundations of gradient-based learning, in \emph{European Symposium on Programming (ESOP)}, 2022.685\bi{DaPa19} C.~Daskalakis, I.~Panageas, Last-iterate convergence: zero-sum games and constrained min-max optimization, in \emph{Innovations in Theoretical Computer Science (ITCS)}, 2019.686\bi{DISZ18} C.~Daskalakis, A.~Ilyas, V.~Syrgkanis, H.~Zeng, Training GANs with optimism, in \emph{Int.\ Conf.\ Learning Representations (ICLR)}, 2018.687\bi{EGHC} Anonymous, \emph{Evolutionary Game Theory in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.688\bi{EvT25} Wang et al., Ev-Trust, a trust mechanism for multi-agent systems of language models built on evolutionary stability, preprint, arXiv:2512.16167, 2025. Full title and author list not recorded when the listing was checked.689\bi{FrSc97} Y.~Freund, R.~E.~Schapire, A decision-theoretic generalization of on-line learning and an application to boosting, \emph{J.~Comput.\ System Sci.} 55 (1997), 119--139.690\bi{FST19} B.~Fong, D.~I.~Spivak, R.~Tuy\'eras, Backprop as functor: a compositional perspective on supervised learning, in \emph{LICS 2019}, IEEE, 2019.691\bi{Goo14} I.~Goodfellow, J.~Pouget-Abadie, M.~Mirza, B.~Xu, D.~Warde-Farley, S.~Ozair, A.~Courville, Y.~Bengio, Generative adversarial nets, in \emph{Advances in Neural Information Processing Systems} 27, 2014.692\bi{GSP26} Group selection to evolve cooperative prompts in populations of LLM agents, listed for NeurIPS 2026; authors not recorded when the listing was checked.693\bi{Har09b} M.~Harper, The replicator equation as an inference dynamic, preprint, arXiv:0911.1763.694\bi{HMO20} D.~Hennes, D.~Morrill, S.~Omidshafiei, R.~Munos, J.~Perolat, M.~Lanctot, A.~Gruslys, J.-B.~Lespiau, P.~Parmas, E.~Du\'e\~nez-Guzm\'an, K.~Tuyls, Neural replicator dynamics: multiagent learning via hedging policy gradients, in \emph{Proc.\ AAMAS}, 2020; arXiv:1906.00190.695\bi{HoSi98} J.~Hofbauer, K.~Sigmund, \emph{Evolutionary Games and Population Dynamics}, Cambridge University Press, 1998.696\bi{Jad17} M.~Jaderberg, V.~Dalibard, S.~Osindero, W.~M.~Czarnecki, J.~Donahue, A.~Razavi, O.~Vinyals, T.~Green, I.~Dunning, K.~Simonyan, C.~Fernando, K.~Kavukcuoglu, Population based training of neural networks, preprint, arXiv:1711.09846.697\bi{JLYY11} X.~Jiang, L.-H.~Lim, Y.~Yao, Y.~Ye, Statistical ranking and combinatorial Hodge theory, \emph{Math.\ Program.} 127 (2011), 203--244.698\bi{Kak01} S.~Kakade, A natural policy gradient, in \emph{Advances in Neural Information Processing Systems} 14, 2001.699\bi{KwMe17} J.~Kwon, P.~Mertikopoulos, A continuous-time approach to online optimization, \emph{J.~Dyn.\ Games} 4 (2017), 125--148.700\bi{LHN05} E.~Lieberman, C.~Hauert, M.~A.~Nowak, Evolutionary dynamics on graphs, \emph{Nature} 433 (2005), 312--316.701\bi{LPS21} S.~Leonardos, G.~Piliouras, K.~Spendlove, Exploration-exploitation in multi-agent competition: convergence with bounded rationality, in \emph{Advances in Neural Information Processing Systems} 34, 2021.702\bi{LZG17} M.~Lanctot, V.~Zambaldi, A.~Gruslys, A.~Lazaridou, K.~Tuyls, J.~P\'erolat, D.~Silver, T.~Graepel, A unified game-theoretic approach to multiagent reinforcement learning, in \emph{Advances in Neural Information Processing Systems} 30, 2017.703\bi{MPP18} P.~Mertikopoulos, C.~Papadimitriou, G.~Piliouras, Cycles in adversarial regularized learning, in \emph{Proc.\ ACM-SIAM Symp.\ Discrete Algorithms (SODA)}, 2018, 2703--2717.704\bi{MSP73} J.~Maynard Smith, G.~R.~Price, The logic of animal conflict, \emph{Nature} 246 (1973), 15--18.705\bi{MXSS20} J.~Mei, C.~Xiao, C.~Szepesv\'ari, D.~Schuurmans, On the global convergence rates of softmax policy gradient methods, in \emph{Proc.\ Int.\ Conf.\ Machine Learning (ICML)}, 2020.706\bi{Mye21} D.~J.~Myers, Double categories of open dynamical systems (extended abstract), in \emph{Applied Category Theory 2020}, EPTCS 333, 2021, 154--167.707\bi{Now06} M.~A.~Nowak, \emph{Evolutionary Dynamics: Exploring the Equations of Life}, Harvard University Press, 2006.708\bi{NSTF04} M.~A.~Nowak, A.~Sasaki, C.~Taylor, D.~Fudenberg, Emergence of cooperation and evolutionary stability in finite populations, \emph{Nature} 428 (2004), 646--650.709\bi{OPP19} S.~Omidshafiei, C.~Papadimitriou, G.~Piliouras, K.~Tuyls, M.~Rowland, J.-B.~Lespiau, W.~M.~Czarnecki, M.~Lanctot, J.~Perolat, R.~Munos, $\alpha$-Rank: multi-agent evaluation by evolution, \emph{Sci.\ Rep.} 9 (2019); arXiv:1903.01373.710\bi{OSG25} Evaluating LLMs in open-source games, preprint, arXiv:2512.00371, 2025; authors not recorded when the listing was checked.711\bi{PaPi19} C.~Papadimitriou, G.~Piliouras, Game dynamics as the meaning of a game, \emph{ACM SIGecom Exchanges} 16 (2019), no.~2, 53--63.712\bi{PiSh14} G.~Piliouras, J.~S.~Shamma, Optimization despite chaos: convex relaxations to complex limit sets via Poincar\'e recurrence, in \emph{Proc.\ ACM-SIAM Symp.\ Discrete Algorithms (SODA)}, 2014, 861--873.713\bi{RaSr13} A.~Rakhlin, K.~Sridharan, Optimization, learning, and games with predictable sequences, in \emph{Advances in Neural Information Processing Systems} 26, 2013.714\bi{SaCr03} Y.~Sato, J.~P.~Crutchfield, Coupled replicator equations for the dynamics of learning in multiagent systems, \emph{Phys.\ Rev.\ E} 67 (2003), 015206.715\bi{Sha09} C.~R.~Shalizi, Dynamics of Bayesian updating with dependent data and misspecified models, \emph{Electron.\ J.~Stat.} 3 (2009), 1039--1074.716\bi{SIL25} Strategic intelligence in large language models: evidence from evolutionary game theory, preprint, arXiv:2507.02618, 2025; authors not recorded when the listing was checked.717\bi{Sor09} S.~Sorin, Exponential weight algorithm in continuous time, \emph{Math.\ Program.} 116 (2009), 513--528.718\bi{SSV20} P.~Schultz, D.~I.~Spivak, C.~Vasilakopoulou, Dynamical systems and sheaves, \emph{Appl.\ Categ.\ Structures} 28 (2020), 1--57.719\bi{TaJo78} P.~D.~Taylor, L.~B.~Jonker, Evolutionarily stable strategies and game dynamics, \emph{Math.\ Biosci.} 40 (1978), 145--156.720\bi{TVL03} K.~Tuyls, K.~Verbeeck, T.~Lenaerts, A selection-mutation model for Q-learning in multi-agent systems, in \emph{Proc.\ AAMAS}, 2003, 693--700.721\bi{Vin19} O.~Vinyals et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, \emph{Nature} 575 (2019), 350--354.722\bi{XLLS26} Xie, Liu, Lan, Szolnoki, Evolutionary games in LLM-agent societies: early milestones and perspectives, \emph{Chaos Solitons Fractals} 213 (2026), 119158. Initials not recorded when the listing was checked.723\end{thebibliography}724\end{document}