Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 61–160 of 1174. Source text is displayed literally.

61\begin{enumerate}62\item A smooth algebra over a typed operad induces a genuine module over its real linearization. Theorem~\ref{thm:tangent-module} proves this statement and identifies the fiberwise derivatives.63\item Proposition~\ref{prop:affine-quotient} identifies the square-zero extension with the linearization of the tangent algebra modulo fiberwise affine relations. This describes the information retained by the module.64\item Theorem~\ref{thm:reverse} derives reverse accumulation on finite computation graphs. Explicit diagonals account for tied parameters. Section~\ref{sec:jets} treats mixed second derivatives using nilpotents and records their extra data.65\item Sections~\ref{sec:examples} and~\ref{sec:derived} give concrete calculations in learning and in symmetry balancing. Section~\ref{sec:stochastic-module} gives a module of probability perturbations and uses it to bound discrepancies in stochastic substitution.66\end{enumerate}67 68This treatment is mathematical. The worked computations check identities on finite examples. Claims about training performance, generalization, and language-model reliability would require separate empirical or statistical evidence.69 70\section{Modules over operadic algebras}71\label{sec:kriz-may}72 73K\v{r}\'{\i}\v{z} and May develop operads in the symmetric monoidal category74of chain complexes over a commutative ring \(k\), with differential of75degree \(-1\) in Parts~I and II. We write \(\mathcal O(n)\) for their76parameter complex \(C(n)\). A symmetric operad has a right77\(\Sigma_n\)-action on \(\mathcal O(n)\), a unit \(k\to\mathcal O(1)\),78and substitution maps79\[80 \mathcal O(r)\otimes\bigotimes_{i=1}^{r}\mathcal O(n_i)81 \longrightarrow \mathcal O\bigl(\textstyle\sum_i n_i\bigr).82\]83The associativity, unit, and equivariance axioms make substitution84independent of how a tree of operations is assembled. Interchanging85factors of degrees \(p\) and \(q\) produces the Koszul sign86\((-1)^{pq}\). An \(\mathcal O\)-algebra \(A\) has equivariant87chain maps \(\theta_n:\mathcal O(n)\otimes A^{\otimes n}\to A\)88compatible with substitution and the unit. These are Definitions~1.189and 2.1 of Part~I of the monograph~\cite{kriz-may}.90 91The free algebra monad on complexes is92\[93 T_{\mathcal O}(X)=94 \bigoplus_{n\geq 0}\mathcal O(n)\otimes_{k[\Sigma_n]}X^{\otimes n}.95\]96Its multiplication substitutes operations, and its algebra axiom97expresses the compatibility of \(\theta\) with that substitution.98Part~I, Definition~3.3 and Proposition~3.4, identify99\(\mathcal O\)-algebras with algebras over this monad. The tensor100powers make the structure maps multilinear in their input vectors.101A nonlinear smooth primitive therefore calls for a semantic102encoding before these structure maps can represent it.103 104\subsection{The marked slot and the enveloping algebra}105 106For a fixed \(\mathcal O\)-algebra \(A\), a module \(M\) in the107sense of K\v{r}\'{\i}\v{z} and May has chain maps108\begin{equation}109 \lambda_n:\mathcal O(n)\otimes A^{\otimes(n-1)}110                 \otimes M\longrightarrow M,\qquad n\geq 1 .111 \label{eq:operadic-module}112\end{equation}113Here the last slot is marked. Equivariance under \(\Sigma_{n-1}\)114permutes the unmarked slots. To put \(M\) in position \(i\), use the115permutation moving that position to the last slot, with the116corresponding action on \(\mathcal O(n)\) and the Koszul signs.117Write the resulting operation as \(\lambda_{n,i}\).118Equivariance under the subgroup fixing the mark makes this119description independent of the permutation used. When such120operations are nested, every subtree outside the path from121the mark to the root is evaluated in \(A\). The operations along122the marked path act on \(M\). The module axiom identifies this123evaluation with applying the composite operation once124(Part~I, Definition~4.1).125 126Define the complex of raw marked operations by127\[128 V_A=\bigoplus_{n\geq 1}129       \mathcal O(n)\otimes_{k[\Sigma_{n-1}]}A^{\otimes(n-1)} .130\]131The action maps assemble into a single map \(V_A\otimes M\to M\). The universal enveloping algebra \(U_{\mathcal O}(A)\)132is the quotient of the tensor algebra on \(V_A\) by the relations133for the unit and for operadic substitution. A product of two134marked operations represents insertion of the inner operation135at the mark of the outer one. Substitutions in unmarked slots136can be evaluated in \(A\) before the marked operation acts.137The maps in~\eqref{eq:operadic-module} give a representation of the138tensor algebra, and the module axioms make it factor through the139quotient. Conversely, restricting a \(U_{\mathcal O}(A)\)-action140to \(V_A\) recovers the maps and their axioms. This proves the141identification of categories in Part~I, Definition~4.9 and142Proposition~4.10. The associative and commutative examples143give \(A\otimes A^{\mathrm{op}}\) and \(A\), respectively144(Part~I, Examples~4.11).145 146\subsection{Square-zero extensions}147 148\begin{proposition}149\label{prop:operadic-square-zero}150An \(\mathcal O\)-module \(M\) over \(A\) determines a split151\(\mathcal O\)-algebra extension \(A\oplus M\to A\) on which152every operation with at least two inputs in \(M\) is zero.153Conversely, a split extension with this vanishing property154determines an \(A\)-module on its kernel.155\end{proposition}156 157\begin{proof}158Define the operations on \(A\oplus M\) by multilinearity.159With every argument in \(A\), use \(\theta_n\) and include160the result in the \(A\)-summand. With one argument in \(M\),