Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 381–480 of 1174. Source text is displayed literally.
381\[382 \langle\lambda_w,h_w\rangle383 =\sum_i \langle L_{w,i}^{\vee}\lambda_w,h_{v(w,i)}\rangle .384\]385Processing in reverse order puts all downstream contributions at \(w\) before it is eliminated. If an input occurs repeatedly, the identical \(h_v\) appears with a sum of covectors. At the end the original differential has been written as a pairing with all input variations independently. Its coefficients are the desired partial differentials.386\end{proof}387 388No inner product is required for the reverse sweep. Choosing an inner product identifies each covector with a gradient. Under Euclidean coordinates \(L^\vee\) is represented by the transpose. This is reverse-mode automatic differentiation~\cite{baydin2018}, which also underlies the categorical formulations in~\cite{fong2019,cruttwell2022}. Here its linear pieces are obtained as operators in a module over an explicit linearized operad.389 390The cotangents of this calculation are local to the evaluated fibers.391The algebraic dual of \(M_c\) is \(\prod_{x\in X_c}T_x^*X_c\).392A transpose on the whole dual need not preserve finite support.393For example, under the unary map \(x\mapsto\sin x\), a covector394supported at the output \(0\) acquires an entry at every \(n\pi\)395when pulled back. Each such entry is multiplied by396\(\cos(n\pi)=(-1)^n\). A reverse pass on a fixed graph uses only397the fibers specified by its forward evaluation.398 399\subsection{Tying parameters along a diagonal}400 401Let \(\Phi:P_1\times\cdots\times P_r\times X\to Y\) be a model with independent parameter slots. A parameterization \(\tau:Q\to\prod_i P_i\) gives the tied model \(\Phi(\tau(q),x)\). At fixed \(x\) put \(L(p)=\ell(\Phi(p,x))\). The tied differential is402\[403 D(L\circ\tau)(q)=(D\tau(q))^\vee\, DL(\tau(q)).404\]405If all \(P_i=P\) and \(\tau(q)=(q,\ldots,q)\), it follows that406\[407 D(L\circ\tau)(q)=\sum_{i=1}^r D_i L(q,\ldots,q).408\]409The summands are covectors on \(P\). The sum arises from evaluating the module on the smooth diagonal \(P\to P^r\), viewed as a unary operation whose output color is a product. Its tangent is \(v\mapsto(v,\ldots,v)\), so dualizing produces addition. It would give a different result to treat repeated mentions as newly independent parameters.410 411For equipped parameter spaces, the Euclidean gradient used by an optimizer depends on the metric chosen on \(Q\). In particular, taking the pullback of the product metric to the diagonal rescales the repeated block by \(r\) and changes the gradient from a sum to an average. The parameter differential already determines the chain rule and is independent of this decision. Section~\ref{sec:derived} computes the same distinction for permutation orbits.412 413\subsection{Differentiating a symmetry}414 415Let \(\sigma:X\to X\) and \(\rho:Y\to Y\) be invertible smooth maps with \(F\circ\sigma=\rho\circ F\). The chain rule gives416\[417 DF(\sigma x)\,D\sigma(x)=D\rho(F(x))\,DF(x).418\]419For a linear action and an invariant \(F\), take \(\rho=\id\). Then \(DF(\sigma x)\sigma=DF(x)\). If the action on \(X\) is orthogonal and \(F\) is scalar, the corresponding gradient satisfies \(\nabla F(\sigma x)=\sigma\nabla F(x)\). This gives a concrete check of a permutation-invariant architecture and of its backward operation. The pooling example below exhibits the check without assuming a universality theorem for invariant networks.420 421\section{Mixed derivatives from nilpotent coefficients}422\label{sec:jets}423 424At a given evaluation point a first-order module operation stores a value and the linear derivative slots. Two maps can have identical first-order data at that point and different quadratic terms. Indeed \(t\mapsto t^2\) and \(t\mapsto 0\) have equal values and first derivatives at \(0\), with second derivatives \(2\) and \(0\). To retain those terms we use an algebra with two nilpotent parameters.425 426\begin{definition}427A real Weil algebra for this discussion is a finite-dimensional commutative real algebra \(W=\mathbb R\oplus\mathfrak m\), with \(\mathfrak m\) a nilpotent ideal and quotient map \(W\to\mathbb R\). Suppose \(\mathfrak m^{s+1}=0\). For a smooth map \(F:X\to Y\), the extension to \(W\) is428\[429 F_W(x+\xi)=F(x)+\sum_{a=1}^{s}430 \frac{1}{a!} D^aF(x)[\xi,\ldots,\xi].431\]432The nilpotent part \(\xi\) lies in \(X\otimes\mathfrak m\).433Here each derivative is extended by multiplying its coefficients in \(W\).434\end{definition}435 436\begin{proposition}[Composition of finite jets]437\label{prop:weil}438Real-space products and composition are preserved by \(F\mapsto F_W\). This applies to substitutions among the operations of any smooth \(P\)-algebra.439\end{proposition}440 441\begin{proof}442Choose coordinates and expand the Taylor polynomial at the real point \(x\) through order \(s\). The Taylor polynomial of \(G\circ F\) is obtained, to this order, by substituting the polynomial of \(F\) at \(x\) into that of \(G\) at \(F(x)\) and discarding terms of degree \(>s\). In detail, for real \(\eta\) near zero the omitted term for a smooth map is \(O(\|\eta\|^{s+1})\); composing with the other Taylor polynomial preserves this order since \(F(x+\eta)-F(x)=O(\|\eta\|)\). Thus the two degree-\(\le s\) polynomials have the same coefficients by uniqueness of Taylor coefficients. On substituting any \(\xi\in X\otimes\mathfrak m\), the discarded monomials vanish. Tuples of Taylor polynomials give the assertion about products.443\end{proof}444 445Take the hyperdual algebra \(W=\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\)~\cite{fike2011}. For \(u,v,w\in X\), the coefficient of \(\epsilon\delta\) in \(F_W(x+\epsilon u+\delta v+\epsilon\delta w)\) is446\[447 DF(x)w+D^2F(x)[u,v].448\]449Using both directions gives, for composable \(F\) and \(G\),450\begin{equation*}451 \begin{split}452 D^2(G\circ F)(x)[u,v]453 &=DG(F(x))\,D^2F(x)[u,v] \\454 &\quad+D^2G(F(x))[DF(x)u,DF(x)v].455 \end{split}456\end{equation*}457The mixed term involves two independently marked directions. A single module slot cannot hold it as a linear map in just one direction, evaluated at the same base point. With a nonlinear reparameterization this is also the origin of the additional term in the Hessian of \(L\circ\tau\). For a linear tie \(\tau\), the second derivative of \(\tau\) is zero, but cross terms in \(D^2L\) still contribute. For example, for a scalar tie \(\tau(t)=(t,t)\),458\[459 (L\circ\tau)''(t)460 = L_{11}(t,t)+L_{12}(t,t)+L_{21}(t,t)+L_{22}(t,t).461\]462This provides a direct rule for curvature calculations in tied models.463 464\section{Worked examples in learning}465\label{sec:examples}466 467\subsection{A nonlinear layer with a tied parameter}468 469Consider a two-stage scalar model with input \(x\), offset \(b\), and470initially independent parameters \(\alpha,\beta\). Write471\[472 \begin{gathered}473 z=\alpha x+b,\quad h=\tanh z,\quad y=\beta h,\\474 L=\tfrac12(y-t)^2,475 \end{gathered}476\]477where \(t\) is a fixed target. Each operation is smooth on its478Euclidean input space. The graph uses the multiplication operation479twice and inserts the nonpolynomial activation \(\tanh\) between480the two uses. We first take the marked-slot maps of the untied graph