Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 901–1000 of 1174. Source text is displayed literally.
901together systems of differential equations902\cite[Sec.~3 and Prop.~4.5]{vagner2015}. In neural programs, the903analogous syntax can record port types, parameter occurrences,904and which subprograms consume each intermediate value. When a905wire feeds multiple consumers, the reverse pass sums all the906cotangents returned along that wire. Consequently, substitutions907can be compiled while keeping the equality of that local sum908with the pullback along the forward wiring map.909 910Gavranovi\'c et al.\ treat constraints and implementations of911deep learning architectures by monads and endofunctor algebras in912a \(2\)-category of parametric maps \cite{gavranovic2024}.913This accounts for recurrent architectures and gives an algebraic914language for equivariance. The chain-complex setting used here915specifies multilinear operations and modules over a fixed916algebra, so explicit passage to an algebra and its module data917is needed when applying it to a smooth parameterized program.918For example, a differentiated relation between two919substitutions compares their effects on inputs and parameter920directions. Such a relation is additional information about921which perturbations preserve the algebra.922 923Fong, Spivak, and Tuy\'eras give a symmetric monoidal functor924from parametrized differentiable functions to learners under925an invertibility hypothesis for the scalar error derivative926\cite[Thm.~III.2]{fong2019}. Cruttwell et al.\ encode reverse927derivatives as lenses and thereby treat models, losses, and928optimizers as composable data \cite[Secs.~2--3]{cruttwell2022}.929Spivak gives operadic dynamics of smooth adaptive arrangements,930with gradient descent as one of the semantics931\cite{spivak2026}. Those formulations describe learning dynamics.932Here the cotangent pullback is the dual of the linearized933substitution. Turning the resulting covector into a parameter934update also requires a choice of metric or optimizer.935 936\subsection{A module of probability perturbations}937\label{sec:stochastic-module}938 939For nonempty finite sets \(X_1,\dots,X_n,Y\), a stochastic operation \(K\)940assigns a probability measure \(K(-\mid\boldsymbol x)\) on \(Y\)941to each input tuple. To substitute a stochastic operation942\(H_i\) into slot \(i\), first sample their results independently943conditional on their respective input blocks \(\boldsymbol u_i\).944The substituted kernel is945\[946 (K(H_1,\ldots,H_n))(y\mid\boldsymbol u)947 =\sum_{\boldsymbol x}K(y\mid\boldsymbol x)948 \prod_i H_i(x_i\mid\boldsymbol u_i).949\]950Rearranging the finite sums verifies associativity. Permuting951inputs reindexes the sums, and identity kernels are units.952Taking these kernels as operations therefore defines a colored953operad in sets.954 955Let \(A_X=\mathbb R^{X}\) be the vector space of signed measures,956\(\epsilon_X(a)=\sum_{x\in X}a(x)\), and \(M_X=\ker\epsilon_X\).957The linearization of the kernel operad acts on \(A\) by958\[959 K_A(a_1,\ldots,a_n)(y)960 =\sum_{\boldsymbol x}K(y\mid\boldsymbol x)\prod_i a_i(x_i).961\]962Since \(\sum_y K(y\mid\boldsymbol x)=1\), its output mass is963\(\prod_i\epsilon_{X_i}(a_i)\). Hence the operation restricts964to \(M_Y\) whenever a chosen input lies in \(M_{X_i}\). These965one-marked-input restrictions satisfy the module axioms by966restricting the algebra substitution identities. In particular967the first-order change in the result under varying one968probability law is the marked-input map on its signed969zero-mass perturbation. Baseline laws are nonnegative and970normalized. At a boundary point of the probability simplex,971a zero-mass direction can be feasible in only one sign.972 973If \(M_X\) is equipped with \(\|m\|_{\rm TV}=\tfrac12\sum_x|m(x)|\),974the marked-input map at probability laws in the remaining slots975has operator norm at most the diameter976\(\sup_{\boldsymbol x,\boldsymbol x'}\|K(-\mid\boldsymbol x)-977K(-\mid\boldsymbol x')\|_{\rm TV}\). Indeed a nonzero \(m\)978is \(\|m\|_{\rm TV}\) times the difference of two probability979laws obtained by normalizing its positive and negative parts.980The two outputs after replacing the marked input are convex981mixtures of rows of \(K\). Their distance is bounded by that982diameter, by comparing all pairs of rows with the product983mixture weights. This is the contraction factor used in the984following consistency test.985 986\subsection{Stochastic substitution and answer consistency}987 988Bottman and Richardson model question decomposition by a989question operad. Their operations are question templates, and990composition plugs the output of one question into a blank991of another. They propose992comparing answers across partial evaluations of a decomposition993tree \cite{bottman2026}. Bottman, Liu, and Richardson report a994companion empirical study of this consistency signal on995multi-hop question-answering datasets \cite{bottmanliu2026}.996One mathematical use of an operadic specification is to say997which evaluation orders should agree under a particular998answer semantics. The following bound quantifies the propagation999of local discrepancies for finite stochastic evaluations.1000