Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 454–553 of 1174. Source text is displayed literally.
454 &\quad+D^2G(F(x))[DF(x)u,DF(x)v].455 \end{split}456\end{equation*}457The mixed term involves two independently marked directions. A single module slot cannot hold it as a linear map in just one direction, evaluated at the same base point. With a nonlinear reparameterization this is also the origin of the additional term in the Hessian of \(L\circ\tau\). For a linear tie \(\tau\), the second derivative of \(\tau\) is zero, but cross terms in \(D^2L\) still contribute. For example, for a scalar tie \(\tau(t)=(t,t)\),458\[459 (L\circ\tau)''(t)460 = L_{11}(t,t)+L_{12}(t,t)+L_{21}(t,t)+L_{22}(t,t).461\]462This provides a direct rule for curvature calculations in tied models.463 464\section{Worked examples in learning}465\label{sec:examples}466 467\subsection{A nonlinear layer with a tied parameter}468 469Consider a two-stage scalar model with input \(x\), offset \(b\), and470initially independent parameters \(\alpha,\beta\). Write471\[472 \begin{gathered}473 z=\alpha x+b,\quad h=\tanh z,\quad y=\beta h,\\474 L=\tfrac12(y-t)^2,475 \end{gathered}476\]477where \(t\) is a fixed target. Each operation is smooth on its478Euclidean input space. The graph uses the multiplication operation479twice and inserts the nonpolynomial activation \(\tanh\) between480the two uses. We first take the marked-slot maps of the untied graph481and then impose the sharing relation.482 483Let \(\overline s\) denote the cotangent of an intermediate scalar484\(s\) for this loss. The last multiplication has marked-slot maps485given by multiplication by \(h\) in the \(\beta\) slot and by486\(\beta\) in the \(h\) slot. Taking adjoints in the standard487scalar pairing gives488\[489 \overline y=y-t,\qquad490 \overline\beta=h\,\overline y,\qquad491 \overline h=\beta\,\overline y.492\]493The activation and affine operation then give494\[495\begin{aligned}496 \overline z &= (1-h^2)\overline h,\\497 \overline\alpha &= x\,\overline z,\qquad498 \overline x=\alpha\,\overline z,\qquad499 \overline b=\overline z .500\end{aligned}501\]502Each expression is determined by a one-slot map and its downstream503cotangent. In the algebra-module construction above, the underlying504slot is based at its particular forward value. For example, the slot505map from \(T_z\mathbb R\) to \(T_h\mathbb R\) multiplies by \(1-h^2\),506with the coefficient evaluated at the given pass through the graph.507 508Tie the parameters using the diagonal509\(\Delta(w)=(\alpha,\beta)=(w,w)\). The parameter variation510\(\dot w\) has image \((\dot w,\dot w)\). Pairing this image with511\((\overline\alpha,\overline\beta)\) gives512\[513 \overline w=\overline\alpha+\overline\beta514 =(y-t)\big(h+wx(1-h^2)\big).515\]516The first term inside the parentheses comes from the direct scaling517of \(h\), and the second comes from the use of \(w\) before the518activation. In a forward calculation the same two paths give519\[520\begin{aligned}521 \dot z &= x\dot w+w\dot x+\dot b,\\522 \dot y &= h\dot w+w(1-h^2)\dot z .523\end{aligned}524\]525The coefficient of \(\dot w\) in \(\dot L=(y-t)\dot y\)526agrees with the reverse computation. This calculation separates527one-hole operadic substitution along a path from the addition528that occurs at a copied input.529 530For an exact evaluation take \(w=x=1\), \(t=0\), and531\(532 b=\operatorname{artanh}(1/2)-1.533\)534Then \(h=y=1/2\) and \(L=1/8\). The two branches contribute535\(\overline\alpha=3/8\) and \(\overline\beta=1/4\), giving536\[537 (\partial_w L,\partial_b L,\partial_x L)538 =(5/8,\,3/8,\,3/8).539\]540Here \(\partial_x L\) measures input sensitivity. For a gradient541update of the two model parameters alone with step \(1/16\), the542new values are \(w'=123/128\) and \(b'=b-3/128\). With \(x\) fixed543at \(1\), the new preactivation is \(z'=z-1/16\). Indeed the two544parameter changes contribute \(5/128\) and \(3/128\) to the decrease545in \(z\). From \(\operatorname{artanh}(1/2)>1/2\) we have546\(0<z'<z\). Strict monotonicity of \(\tanh\) and \(0<w'<1\)547give \(0<y'<y\), hence the loss decreases for this update.548The accompanying script computes its new value.549 550The same point gives a mixed-derivative check. For551\(y=w\tanh(wx+b)\), denote the activation by \(\sigma\).552The relevant derivatives can be read off from the affine553operation and the two uses of \(w\):