Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 401–500 of 1174. Source text is displayed literally.

401Let \(\Phi:P_1\times\cdots\times P_r\times X\to Y\) be a model with independent parameter slots. A parameterization \(\tau:Q\to\prod_i P_i\) gives the tied model \(\Phi(\tau(q),x)\). At fixed \(x\) put \(L(p)=\ell(\Phi(p,x))\). The tied differential is402\[403 D(L\circ\tau)(q)=(D\tau(q))^\vee\, DL(\tau(q)).404\]405If all \(P_i=P\) and \(\tau(q)=(q,\ldots,q)\), it follows that406\[407 D(L\circ\tau)(q)=\sum_{i=1}^r D_i L(q,\ldots,q).408\]409The summands are covectors on \(P\). The sum arises from evaluating the module on the smooth diagonal \(P\to P^r\), viewed as a unary operation whose output color is a product. Its tangent is \(v\mapsto(v,\ldots,v)\), so dualizing produces addition. It would give a different result to treat repeated mentions as newly independent parameters.410 411For equipped parameter spaces, the Euclidean gradient used by an optimizer depends on the metric chosen on \(Q\). In particular, taking the pullback of the product metric to the diagonal rescales the repeated block by \(r\) and changes the gradient from a sum to an average. The parameter differential already determines the chain rule and is independent of this decision. Section~\ref{sec:derived} computes the same distinction for permutation orbits.412 413\subsection{Differentiating a symmetry}414 415Let \(\sigma:X\to X\) and \(\rho:Y\to Y\) be invertible smooth maps with \(F\circ\sigma=\rho\circ F\). The chain rule gives416\[417 DF(\sigma x)\,D\sigma(x)=D\rho(F(x))\,DF(x).418\]419For a linear action and an invariant \(F\), take \(\rho=\id\). Then \(DF(\sigma x)\sigma=DF(x)\). If the action on \(X\) is orthogonal and \(F\) is scalar, the corresponding gradient satisfies \(\nabla F(\sigma x)=\sigma\nabla F(x)\). This gives a concrete check of a permutation-invariant architecture and of its backward operation. The pooling example below exhibits the check without assuming a universality theorem for invariant networks.420 421\section{Mixed derivatives from nilpotent coefficients}422\label{sec:jets}423 424At a given evaluation point a first-order module operation stores a value and the linear derivative slots. Two maps can have identical first-order data at that point and different quadratic terms. Indeed \(t\mapsto t^2\) and \(t\mapsto 0\) have equal values and first derivatives at \(0\), with second derivatives \(2\) and \(0\). To retain those terms we use an algebra with two nilpotent parameters.425 426\begin{definition}427A real Weil algebra for this discussion is a finite-dimensional commutative real algebra \(W=\mathbb R\oplus\mathfrak m\), with \(\mathfrak m\) a nilpotent ideal and quotient map \(W\to\mathbb R\). Suppose \(\mathfrak m^{s+1}=0\). For a smooth map \(F:X\to Y\), the extension to \(W\) is428\[429 F_W(x+\xi)=F(x)+\sum_{a=1}^{s}430       \frac{1}{a!} D^aF(x)[\xi,\ldots,\xi].431\]432The nilpotent part \(\xi\) lies in \(X\otimes\mathfrak m\).433Here each derivative is extended by multiplying its coefficients in \(W\).434\end{definition}435 436\begin{proposition}[Composition of finite jets]437\label{prop:weil}438Real-space products and composition are preserved by \(F\mapsto F_W\). This applies to substitutions among the operations of any smooth \(P\)-algebra.439\end{proposition}440 441\begin{proof}442Choose coordinates and expand the Taylor polynomial at the real point \(x\) through order \(s\). The Taylor polynomial of \(G\circ F\) is obtained, to this order, by substituting the polynomial of \(F\) at \(x\) into that of \(G\) at \(F(x)\) and discarding terms of degree \(>s\). In detail, for real \(\eta\) near zero the omitted term for a smooth map is \(O(\|\eta\|^{s+1})\); composing with the other Taylor polynomial preserves this order since \(F(x+\eta)-F(x)=O(\|\eta\|)\). Thus the two degree-\(\le s\) polynomials have the same coefficients by uniqueness of Taylor coefficients. On substituting any \(\xi\in X\otimes\mathfrak m\), the discarded monomials vanish. Tuples of Taylor polynomials give the assertion about products.443\end{proof}444 445Take the hyperdual algebra \(W=\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\)~\cite{fike2011}. For \(u,v,w\in X\), the coefficient of \(\epsilon\delta\) in \(F_W(x+\epsilon u+\delta v+\epsilon\delta w)\) is446\[447 DF(x)w+D^2F(x)[u,v].448\]449Using both directions gives, for composable \(F\) and \(G\),450\begin{equation*}451 \begin{split}452 D^2(G\circ F)(x)[u,v]453    &=DG(F(x))\,D^2F(x)[u,v] \\454    &\quad+D^2G(F(x))[DF(x)u,DF(x)v].455 \end{split}456\end{equation*}457The mixed term involves two independently marked directions. A single module slot cannot hold it as a linear map in just one direction, evaluated at the same base point. With a nonlinear reparameterization this is also the origin of the additional term in the Hessian of \(L\circ\tau\). For a linear tie \(\tau\), the second derivative of \(\tau\) is zero, but cross terms in \(D^2L\) still contribute. For example, for a scalar tie \(\tau(t)=(t,t)\),458\[459 (L\circ\tau)''(t)460   = L_{11}(t,t)+L_{12}(t,t)+L_{21}(t,t)+L_{22}(t,t).461\]462This provides a direct rule for curvature calculations in tied models.463 464\section{Worked examples in learning}465\label{sec:examples}466 467\subsection{A nonlinear layer with a tied parameter}468 469Consider a two-stage scalar model with input \(x\), offset \(b\), and470initially independent parameters \(\alpha,\beta\). Write471\[472 \begin{gathered}473 z=\alpha x+b,\quad h=\tanh z,\quad y=\beta h,\\474 L=\tfrac12(y-t)^2,475 \end{gathered}476\]477where \(t\) is a fixed target. Each operation is smooth on its478Euclidean input space. The graph uses the multiplication operation479twice and inserts the nonpolynomial activation \(\tanh\) between480the two uses. We first take the marked-slot maps of the untied graph481and then impose the sharing relation.482 483Let \(\overline s\) denote the cotangent of an intermediate scalar484\(s\) for this loss. The last multiplication has marked-slot maps485given by multiplication by \(h\) in the \(\beta\) slot and by486\(\beta\) in the \(h\) slot. Taking adjoints in the standard487scalar pairing gives488\[489 \overline y=y-t,\qquad490 \overline\beta=h\,\overline y,\qquad491 \overline h=\beta\,\overline y.492\]493The activation and affine operation then give494\[495\begin{aligned}496 \overline z &= (1-h^2)\overline h,\\497 \overline\alpha &= x\,\overline z,\qquad498 \overline x=\alpha\,\overline z,\qquad499 \overline b=\overline z .500\end{aligned}