Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 1–100 of 1174. Source text is displayed literally.
1\documentclass[10pt,letterpaper,twocolumn]{article}2\usepackage[T1]{fontenc}3\usepackage{lmodern}4\usepackage[margin=.74in,includeheadfoot,headheight=14pt,headsep=13pt]{geometry}5\usepackage{microtype,mathtools,amssymb,amsthm}6\usepackage{xcolor,tikz-cd,booktabs,array,enumitem,fancyhdr}7\usepackage{xurl,hyperref}8\tikzcdset{labels={/tikz/auto=false,description,/tikz/inner sep=0.8pt}}9\definecolor{accent}{HTML}{155852}10\definecolor{ink}{HTML}{162724}11\hypersetup{colorlinks=true,linkcolor=accent,citecolor=accent,urlcolor=accent,12 pdftitle={Operads, Modules, and Differentiation in Machine Learning},13 pdfauthor={Anonymous},14 pdfsubject={Kriz--May operadic modules and differentiable programs}}15\AtBeginDocument{\color{ink}}16\setlength{\columnsep}{.28in}17\setlength{\parskip}{1.5pt plus .5pt}18\setlength{\emergencystretch}{1em}19\raggedbottom20\allowdisplaybreaks[1]21\setlist[itemize]{leftmargin=*,nosep,topsep=3pt}22\setlist[enumerate]{leftmargin=*,itemsep=2pt}23\newtheorem{theorem}{Theorem}[section]24\newtheorem{proposition}[theorem]{Proposition}25\newtheorem{lemma}[theorem]{Lemma}26\newtheorem{corollary}[theorem]{Corollary}27\theoremstyle{definition}28\newtheorem{definition}[theorem]{Definition}29\newtheorem{example}[theorem]{Example}30\theoremstyle{remark}31\newtheorem{remark}[theorem]{Remark}32\DeclareMathOperator{\Hom}{Hom}33\DeclareMathOperator{\End}{End}34\DeclareMathOperator{\Tor}{Tor}35\DeclareMathOperator{\im}{im}36\DeclareMathOperator{\id}{id}37\pagestyle{fancy}38\fancyhf{}39\fancyhead[L]{\small\sffamily\color{accent} Operadic modules in machine learning}40\fancyhead[R]{\small\sffamily\thepage}41\renewcommand{\headrulewidth}{.35pt}42\fancypagestyle{plain}{\fancyhf{}\fancyfoot[C]{\small\sffamily\thepage}\renewcommand{\headrulewidth}{0pt}}43\title{\bfseries Operads, Modules, and Differentiation\\in Machine Learning}44\author{Anonymous}45\date{}46 47\begin{document}48\maketitle49\begin{abstract}50We identify an exact connection between the one-marked-input modules of K\v{r}\'{\i}\v{z} and May and differentiation in machine learning. Freely linearizing the colors and operations of a smooth computation turns the direct sum of its tangent fibers into an operadic module. The square-zero extension is a quotient of the free vector space on the total tangent bundle by fiberwise affine relations. These constructions recover composition of partial derivatives, reverse accumulation, and gradients for tied parameters. A truncated Weil algebra gives a separate construction for mixed derivatives. We also compute derived coinvariants of finite permutation modules by stabilizers, separating real-valued parameter averaging from higher homology in modular coefficients. Worked examples test tied nonlinear parameters and permutation-invariant pooling. For finite stochastic AI components, signed zero-mass perturbations form another operadic module, whose contraction factors bound consistency errors.51\end{abstract}52 53\section{Introduction}54\label{sec:introduction}55 56A neural architecture contains choices of layer, topology, parameter-sharing pattern, and symmetry. Each choice affects the maps that training differentiates. An operad describes legal substitutions among such choices. Its algebras interpret the operations, and its modules describe what happens when one input carries a linear perturbation. The question is whether the last statement can be made literal for smooth architectures. K\v{r}\'{\i}\v{z} and May work with operads in chain complexes, whose algebra operations are multilinear. A nonlinear smooth map with two inputs is a function on their Cartesian product. Any use of their module formula must account for that change of ambient category.57 58We give a passage from smooth semantics to the differential graded setting concentrated in degree zero. For each input type \(X\), the algebra color is the free vector space on \(X\), and the module color is the direct sum of tangent fibers over the points of \(X\). This separates the coefficient supplied by an unmarked input from the direction attached to a marked input. It makes the derivative of each operation multilinear in the enlarged vector spaces even if the original operation is nonlinear. The module axioms then follow from the chain rule. The construction traces back to Part I of the monograph, especially its one-marked-input operations and enveloping algebra~\cite{kriz-may}.59 60The main results and applications are as follows.61\begin{enumerate}62\item A smooth algebra over a typed operad induces a genuine module over its real linearization. Theorem~\ref{thm:tangent-module} proves this statement and identifies the fiberwise derivatives.63\item Proposition~\ref{prop:affine-quotient} identifies the square-zero extension with the linearization of the tangent algebra modulo fiberwise affine relations. This describes the information retained by the module.64\item Theorem~\ref{thm:reverse} derives reverse accumulation on finite computation graphs. Explicit diagonals account for tied parameters. Section~\ref{sec:jets} treats mixed second derivatives using nilpotents and records their extra data.65\item Sections~\ref{sec:examples} and~\ref{sec:derived} give concrete calculations in learning and in symmetry balancing. Section~\ref{sec:stochastic-module} gives a module of probability perturbations and uses it to bound discrepancies in stochastic substitution.66\end{enumerate}67 68This treatment is mathematical. The worked computations check identities on finite examples. Claims about training performance, generalization, and language-model reliability would require separate empirical or statistical evidence.69 70\section{Modules over operadic algebras}71\label{sec:kriz-may}72 73K\v{r}\'{\i}\v{z} and May develop operads in the symmetric monoidal category74of chain complexes over a commutative ring \(k\), with differential of75degree \(-1\) in Parts~I and II. We write \(\mathcal O(n)\) for their76parameter complex \(C(n)\). A symmetric operad has a right77\(\Sigma_n\)-action on \(\mathcal O(n)\), a unit \(k\to\mathcal O(1)\),78and substitution maps79\[80 \mathcal O(r)\otimes\bigotimes_{i=1}^{r}\mathcal O(n_i)81 \longrightarrow \mathcal O\bigl(\textstyle\sum_i n_i\bigr).82\]83The associativity, unit, and equivariance axioms make substitution84independent of how a tree of operations is assembled. Interchanging85factors of degrees \(p\) and \(q\) produces the Koszul sign86\((-1)^{pq}\). An \(\mathcal O\)-algebra \(A\) has equivariant87chain maps \(\theta_n:\mathcal O(n)\otimes A^{\otimes n}\to A\)88compatible with substitution and the unit. These are Definitions~1.189and 2.1 of Part~I of the monograph~\cite{kriz-may}.90 91The free algebra monad on complexes is92\[93 T_{\mathcal O}(X)=94 \bigoplus_{n\geq 0}\mathcal O(n)\otimes_{k[\Sigma_n]}X^{\otimes n}.95\]96Its multiplication substitutes operations, and its algebra axiom97expresses the compatibility of \(\theta\) with that substitution.98Part~I, Definition~3.3 and Proposition~3.4, identify99\(\mathcal O\)-algebras with algebras over this monad. The tensor100powers make the structure maps multilinear in their input vectors.