Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 441–540 of 1174. Source text is displayed literally.
441\begin{proof}442Choose coordinates and expand the Taylor polynomial at the real point \(x\) through order \(s\). The Taylor polynomial of \(G\circ F\) is obtained, to this order, by substituting the polynomial of \(F\) at \(x\) into that of \(G\) at \(F(x)\) and discarding terms of degree \(>s\). In detail, for real \(\eta\) near zero the omitted term for a smooth map is \(O(\|\eta\|^{s+1})\); composing with the other Taylor polynomial preserves this order since \(F(x+\eta)-F(x)=O(\|\eta\|)\). Thus the two degree-\(\le s\) polynomials have the same coefficients by uniqueness of Taylor coefficients. On substituting any \(\xi\in X\otimes\mathfrak m\), the discarded monomials vanish. Tuples of Taylor polynomials give the assertion about products.443\end{proof}444 445Take the hyperdual algebra \(W=\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\)~\cite{fike2011}. For \(u,v,w\in X\), the coefficient of \(\epsilon\delta\) in \(F_W(x+\epsilon u+\delta v+\epsilon\delta w)\) is446\[447 DF(x)w+D^2F(x)[u,v].448\]449Using both directions gives, for composable \(F\) and \(G\),450\begin{equation*}451 \begin{split}452 D^2(G\circ F)(x)[u,v]453 &=DG(F(x))\,D^2F(x)[u,v] \\454 &\quad+D^2G(F(x))[DF(x)u,DF(x)v].455 \end{split}456\end{equation*}457The mixed term involves two independently marked directions. A single module slot cannot hold it as a linear map in just one direction, evaluated at the same base point. With a nonlinear reparameterization this is also the origin of the additional term in the Hessian of \(L\circ\tau\). For a linear tie \(\tau\), the second derivative of \(\tau\) is zero, but cross terms in \(D^2L\) still contribute. For example, for a scalar tie \(\tau(t)=(t,t)\),458\[459 (L\circ\tau)''(t)460 = L_{11}(t,t)+L_{12}(t,t)+L_{21}(t,t)+L_{22}(t,t).461\]462This provides a direct rule for curvature calculations in tied models.463 464\section{Worked examples in learning}465\label{sec:examples}466 467\subsection{A nonlinear layer with a tied parameter}468 469Consider a two-stage scalar model with input \(x\), offset \(b\), and470initially independent parameters \(\alpha,\beta\). Write471\[472 \begin{gathered}473 z=\alpha x+b,\quad h=\tanh z,\quad y=\beta h,\\474 L=\tfrac12(y-t)^2,475 \end{gathered}476\]477where \(t\) is a fixed target. Each operation is smooth on its478Euclidean input space. The graph uses the multiplication operation479twice and inserts the nonpolynomial activation \(\tanh\) between480the two uses. We first take the marked-slot maps of the untied graph481and then impose the sharing relation.482 483Let \(\overline s\) denote the cotangent of an intermediate scalar484\(s\) for this loss. The last multiplication has marked-slot maps485given by multiplication by \(h\) in the \(\beta\) slot and by486\(\beta\) in the \(h\) slot. Taking adjoints in the standard487scalar pairing gives488\[489 \overline y=y-t,\qquad490 \overline\beta=h\,\overline y,\qquad491 \overline h=\beta\,\overline y.492\]493The activation and affine operation then give494\[495\begin{aligned}496 \overline z &= (1-h^2)\overline h,\\497 \overline\alpha &= x\,\overline z,\qquad498 \overline x=\alpha\,\overline z,\qquad499 \overline b=\overline z .500\end{aligned}501\]502Each expression is determined by a one-slot map and its downstream503cotangent. In the algebra-module construction above, the underlying504slot is based at its particular forward value. For example, the slot505map from \(T_z\mathbb R\) to \(T_h\mathbb R\) multiplies by \(1-h^2\),506with the coefficient evaluated at the given pass through the graph.507 508Tie the parameters using the diagonal509\(\Delta(w)=(\alpha,\beta)=(w,w)\). The parameter variation510\(\dot w\) has image \((\dot w,\dot w)\). Pairing this image with511\((\overline\alpha,\overline\beta)\) gives512\[513 \overline w=\overline\alpha+\overline\beta514 =(y-t)\big(h+wx(1-h^2)\big).515\]516The first term inside the parentheses comes from the direct scaling517of \(h\), and the second comes from the use of \(w\) before the518activation. In a forward calculation the same two paths give519\[520\begin{aligned}521 \dot z &= x\dot w+w\dot x+\dot b,\\522 \dot y &= h\dot w+w(1-h^2)\dot z .523\end{aligned}524\]525The coefficient of \(\dot w\) in \(\dot L=(y-t)\dot y\)526agrees with the reverse computation. This calculation separates527one-hole operadic substitution along a path from the addition528that occurs at a copied input.529 530For an exact evaluation take \(w=x=1\), \(t=0\), and531\(532 b=\operatorname{artanh}(1/2)-1.533\)534Then \(h=y=1/2\) and \(L=1/8\). The two branches contribute535\(\overline\alpha=3/8\) and \(\overline\beta=1/4\), giving536\[537 (\partial_w L,\partial_b L,\partial_x L)538 =(5/8,\,3/8,\,3/8).539\]540Here \(\partial_x L\) measures input sensitivity. For a gradient