Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 421–520 of 1174. Source text is displayed literally.
421\section{Mixed derivatives from nilpotent coefficients}422\label{sec:jets}423 424At a given evaluation point a first-order module operation stores a value and the linear derivative slots. Two maps can have identical first-order data at that point and different quadratic terms. Indeed \(t\mapsto t^2\) and \(t\mapsto 0\) have equal values and first derivatives at \(0\), with second derivatives \(2\) and \(0\). To retain those terms we use an algebra with two nilpotent parameters.425 426\begin{definition}427A real Weil algebra for this discussion is a finite-dimensional commutative real algebra \(W=\mathbb R\oplus\mathfrak m\), with \(\mathfrak m\) a nilpotent ideal and quotient map \(W\to\mathbb R\). Suppose \(\mathfrak m^{s+1}=0\). For a smooth map \(F:X\to Y\), the extension to \(W\) is428\[429 F_W(x+\xi)=F(x)+\sum_{a=1}^{s}430 \frac{1}{a!} D^aF(x)[\xi,\ldots,\xi].431\]432The nilpotent part \(\xi\) lies in \(X\otimes\mathfrak m\).433Here each derivative is extended by multiplying its coefficients in \(W\).434\end{definition}435 436\begin{proposition}[Composition of finite jets]437\label{prop:weil}438Real-space products and composition are preserved by \(F\mapsto F_W\). This applies to substitutions among the operations of any smooth \(P\)-algebra.439\end{proposition}440 441\begin{proof}442Choose coordinates and expand the Taylor polynomial at the real point \(x\) through order \(s\). The Taylor polynomial of \(G\circ F\) is obtained, to this order, by substituting the polynomial of \(F\) at \(x\) into that of \(G\) at \(F(x)\) and discarding terms of degree \(>s\). In detail, for real \(\eta\) near zero the omitted term for a smooth map is \(O(\|\eta\|^{s+1})\); composing with the other Taylor polynomial preserves this order since \(F(x+\eta)-F(x)=O(\|\eta\|)\). Thus the two degree-\(\le s\) polynomials have the same coefficients by uniqueness of Taylor coefficients. On substituting any \(\xi\in X\otimes\mathfrak m\), the discarded monomials vanish. Tuples of Taylor polynomials give the assertion about products.443\end{proof}444 445Take the hyperdual algebra \(W=\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\)~\cite{fike2011}. For \(u,v,w\in X\), the coefficient of \(\epsilon\delta\) in \(F_W(x+\epsilon u+\delta v+\epsilon\delta w)\) is446\[447 DF(x)w+D^2F(x)[u,v].448\]449Using both directions gives, for composable \(F\) and \(G\),450\begin{equation*}451 \begin{split}452 D^2(G\circ F)(x)[u,v]453 &=DG(F(x))\,D^2F(x)[u,v] \\454 &\quad+D^2G(F(x))[DF(x)u,DF(x)v].455 \end{split}456\end{equation*}457The mixed term involves two independently marked directions. A single module slot cannot hold it as a linear map in just one direction, evaluated at the same base point. With a nonlinear reparameterization this is also the origin of the additional term in the Hessian of \(L\circ\tau\). For a linear tie \(\tau\), the second derivative of \(\tau\) is zero, but cross terms in \(D^2L\) still contribute. For example, for a scalar tie \(\tau(t)=(t,t)\),458\[459 (L\circ\tau)''(t)460 = L_{11}(t,t)+L_{12}(t,t)+L_{21}(t,t)+L_{22}(t,t).461\]462This provides a direct rule for curvature calculations in tied models.463 464\section{Worked examples in learning}465\label{sec:examples}466 467\subsection{A nonlinear layer with a tied parameter}468 469Consider a two-stage scalar model with input \(x\), offset \(b\), and470initially independent parameters \(\alpha,\beta\). Write471\[472 \begin{gathered}473 z=\alpha x+b,\quad h=\tanh z,\quad y=\beta h,\\474 L=\tfrac12(y-t)^2,475 \end{gathered}476\]477where \(t\) is a fixed target. Each operation is smooth on its478Euclidean input space. The graph uses the multiplication operation479twice and inserts the nonpolynomial activation \(\tanh\) between480the two uses. We first take the marked-slot maps of the untied graph481and then impose the sharing relation.482 483Let \(\overline s\) denote the cotangent of an intermediate scalar484\(s\) for this loss. The last multiplication has marked-slot maps485given by multiplication by \(h\) in the \(\beta\) slot and by486\(\beta\) in the \(h\) slot. Taking adjoints in the standard487scalar pairing gives488\[489 \overline y=y-t,\qquad490 \overline\beta=h\,\overline y,\qquad491 \overline h=\beta\,\overline y.492\]493The activation and affine operation then give494\[495\begin{aligned}496 \overline z &= (1-h^2)\overline h,\\497 \overline\alpha &= x\,\overline z,\qquad498 \overline x=\alpha\,\overline z,\qquad499 \overline b=\overline z .500\end{aligned}501\]502Each expression is determined by a one-slot map and its downstream503cotangent. In the algebra-module construction above, the underlying504slot is based at its particular forward value. For example, the slot505map from \(T_z\mathbb R\) to \(T_h\mathbb R\) multiplies by \(1-h^2\),506with the coefficient evaluated at the given pass through the graph.507 508Tie the parameters using the diagonal509\(\Delta(w)=(\alpha,\beta)=(w,w)\). The parameter variation510\(\dot w\) has image \((\dot w,\dot w)\). Pairing this image with511\((\overline\alpha,\overline\beta)\) gives512\[513 \overline w=\overline\alpha+\overline\beta514 =(y-t)\big(h+wx(1-h^2)\big).515\]516The first term inside the parentheses comes from the direct scaling517of \(h\), and the second comes from the use of \(w\) before the518activation. In a forward calculation the same two paths give519\[520\begin{aligned}