Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 41–140 of 1174. Source text is displayed literally.
41\renewcommand{\headrulewidth}{.35pt}42\fancypagestyle{plain}{\fancyhf{}\fancyfoot[C]{\small\sffamily\thepage}\renewcommand{\headrulewidth}{0pt}}43\title{\bfseries Operads, Modules, and Differentiation\\in Machine Learning}44\author{Anonymous}45\date{}46 47\begin{document}48\maketitle49\begin{abstract}50We identify an exact connection between the one-marked-input modules of K\v{r}\'{\i}\v{z} and May and differentiation in machine learning. Freely linearizing the colors and operations of a smooth computation turns the direct sum of its tangent fibers into an operadic module. The square-zero extension is a quotient of the free vector space on the total tangent bundle by fiberwise affine relations. These constructions recover composition of partial derivatives, reverse accumulation, and gradients for tied parameters. A truncated Weil algebra gives a separate construction for mixed derivatives. We also compute derived coinvariants of finite permutation modules by stabilizers, separating real-valued parameter averaging from higher homology in modular coefficients. Worked examples test tied nonlinear parameters and permutation-invariant pooling. For finite stochastic AI components, signed zero-mass perturbations form another operadic module, whose contraction factors bound consistency errors.51\end{abstract}52 53\section{Introduction}54\label{sec:introduction}55 56A neural architecture contains choices of layer, topology, parameter-sharing pattern, and symmetry. Each choice affects the maps that training differentiates. An operad describes legal substitutions among such choices. Its algebras interpret the operations, and its modules describe what happens when one input carries a linear perturbation. The question is whether the last statement can be made literal for smooth architectures. K\v{r}\'{\i}\v{z} and May work with operads in chain complexes, whose algebra operations are multilinear. A nonlinear smooth map with two inputs is a function on their Cartesian product. Any use of their module formula must account for that change of ambient category.57 58We give a passage from smooth semantics to the differential graded setting concentrated in degree zero. For each input type \(X\), the algebra color is the free vector space on \(X\), and the module color is the direct sum of tangent fibers over the points of \(X\). This separates the coefficient supplied by an unmarked input from the direction attached to a marked input. It makes the derivative of each operation multilinear in the enlarged vector spaces even if the original operation is nonlinear. The module axioms then follow from the chain rule. The construction traces back to Part I of the monograph, especially its one-marked-input operations and enveloping algebra~\cite{kriz-may}.59 60The main results and applications are as follows.61\begin{enumerate}62\item A smooth algebra over a typed operad induces a genuine module over its real linearization. Theorem~\ref{thm:tangent-module} proves this statement and identifies the fiberwise derivatives.63\item Proposition~\ref{prop:affine-quotient} identifies the square-zero extension with the linearization of the tangent algebra modulo fiberwise affine relations. This describes the information retained by the module.64\item Theorem~\ref{thm:reverse} derives reverse accumulation on finite computation graphs. Explicit diagonals account for tied parameters. Section~\ref{sec:jets} treats mixed second derivatives using nilpotents and records their extra data.65\item Sections~\ref{sec:examples} and~\ref{sec:derived} give concrete calculations in learning and in symmetry balancing. Section~\ref{sec:stochastic-module} gives a module of probability perturbations and uses it to bound discrepancies in stochastic substitution.66\end{enumerate}67 68This treatment is mathematical. The worked computations check identities on finite examples. Claims about training performance, generalization, and language-model reliability would require separate empirical or statistical evidence.69 70\section{Modules over operadic algebras}71\label{sec:kriz-may}72 73K\v{r}\'{\i}\v{z} and May develop operads in the symmetric monoidal category74of chain complexes over a commutative ring \(k\), with differential of75degree \(-1\) in Parts~I and II. We write \(\mathcal O(n)\) for their76parameter complex \(C(n)\). A symmetric operad has a right77\(\Sigma_n\)-action on \(\mathcal O(n)\), a unit \(k\to\mathcal O(1)\),78and substitution maps79\[80 \mathcal O(r)\otimes\bigotimes_{i=1}^{r}\mathcal O(n_i)81 \longrightarrow \mathcal O\bigl(\textstyle\sum_i n_i\bigr).82\]83The associativity, unit, and equivariance axioms make substitution84independent of how a tree of operations is assembled. Interchanging85factors of degrees \(p\) and \(q\) produces the Koszul sign86\((-1)^{pq}\). An \(\mathcal O\)-algebra \(A\) has equivariant87chain maps \(\theta_n:\mathcal O(n)\otimes A^{\otimes n}\to A\)88compatible with substitution and the unit. These are Definitions~1.189and 2.1 of Part~I of the monograph~\cite{kriz-may}.90 91The free algebra monad on complexes is92\[93 T_{\mathcal O}(X)=94 \bigoplus_{n\geq 0}\mathcal O(n)\otimes_{k[\Sigma_n]}X^{\otimes n}.95\]96Its multiplication substitutes operations, and its algebra axiom97expresses the compatibility of \(\theta\) with that substitution.98Part~I, Definition~3.3 and Proposition~3.4, identify99\(\mathcal O\)-algebras with algebras over this monad. The tensor100powers make the structure maps multilinear in their input vectors.101A nonlinear smooth primitive therefore calls for a semantic102encoding before these structure maps can represent it.103 104\subsection{The marked slot and the enveloping algebra}105 106For a fixed \(\mathcal O\)-algebra \(A\), a module \(M\) in the107sense of K\v{r}\'{\i}\v{z} and May has chain maps108\begin{equation}109 \lambda_n:\mathcal O(n)\otimes A^{\otimes(n-1)}110 \otimes M\longrightarrow M,\qquad n\geq 1 .111 \label{eq:operadic-module}112\end{equation}113Here the last slot is marked. Equivariance under \(\Sigma_{n-1}\)114permutes the unmarked slots. To put \(M\) in position \(i\), use the115permutation moving that position to the last slot, with the116corresponding action on \(\mathcal O(n)\) and the Koszul signs.117Write the resulting operation as \(\lambda_{n,i}\).118Equivariance under the subgroup fixing the mark makes this119description independent of the permutation used. When such120operations are nested, every subtree outside the path from121the mark to the root is evaluated in \(A\). The operations along122the marked path act on \(M\). The module axiom identifies this123evaluation with applying the composite operation once124(Part~I, Definition~4.1).125 126Define the complex of raw marked operations by127\[128 V_A=\bigoplus_{n\geq 1}129 \mathcal O(n)\otimes_{k[\Sigma_{n-1}]}A^{\otimes(n-1)} .130\]131The action maps assemble into a single map \(V_A\otimes M\to M\). The universal enveloping algebra \(U_{\mathcal O}(A)\)132is the quotient of the tensor algebra on \(V_A\) by the relations133for the unit and for operadic substitution. A product of two134marked operations represents insertion of the inner operation135at the mark of the outer one. Substitutions in unmarked slots136can be evaluated in \(A\) before the marked operation acts.137The maps in~\eqref{eq:operadic-module} give a representation of the138tensor algebra, and the module axioms make it factor through the139quotient. Conversely, restricting a \(U_{\mathcal O}(A)\)-action140to \(V_A\) recovers the maps and their axioms. This proves the