Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 461–560 of 1174. Source text is displayed literally.

461\]462This provides a direct rule for curvature calculations in tied models.463 464\section{Worked examples in learning}465\label{sec:examples}466 467\subsection{A nonlinear layer with a tied parameter}468 469Consider a two-stage scalar model with input \(x\), offset \(b\), and470initially independent parameters \(\alpha,\beta\). Write471\[472 \begin{gathered}473 z=\alpha x+b,\quad h=\tanh z,\quad y=\beta h,\\474 L=\tfrac12(y-t)^2,475 \end{gathered}476\]477where \(t\) is a fixed target. Each operation is smooth on its478Euclidean input space. The graph uses the multiplication operation479twice and inserts the nonpolynomial activation \(\tanh\) between480the two uses. We first take the marked-slot maps of the untied graph481and then impose the sharing relation.482 483Let \(\overline s\) denote the cotangent of an intermediate scalar484\(s\) for this loss. The last multiplication has marked-slot maps485given by multiplication by \(h\) in the \(\beta\) slot and by486\(\beta\) in the \(h\) slot. Taking adjoints in the standard487scalar pairing gives488\[489 \overline y=y-t,\qquad490 \overline\beta=h\,\overline y,\qquad491 \overline h=\beta\,\overline y.492\]493The activation and affine operation then give494\[495\begin{aligned}496 \overline z &= (1-h^2)\overline h,\\497 \overline\alpha &= x\,\overline z,\qquad498 \overline x=\alpha\,\overline z,\qquad499 \overline b=\overline z .500\end{aligned}501\]502Each expression is determined by a one-slot map and its downstream503cotangent. In the algebra-module construction above, the underlying504slot is based at its particular forward value. For example, the slot505map from \(T_z\mathbb R\) to \(T_h\mathbb R\) multiplies by \(1-h^2\),506with the coefficient evaluated at the given pass through the graph.507 508Tie the parameters using the diagonal509\(\Delta(w)=(\alpha,\beta)=(w,w)\). The parameter variation510\(\dot w\) has image \((\dot w,\dot w)\). Pairing this image with511\((\overline\alpha,\overline\beta)\) gives512\[513 \overline w=\overline\alpha+\overline\beta514   =(y-t)\big(h+wx(1-h^2)\big).515\]516The first term inside the parentheses comes from the direct scaling517of \(h\), and the second comes from the use of \(w\) before the518activation. In a forward calculation the same two paths give519\[520\begin{aligned}521 \dot z &= x\dot w+w\dot x+\dot b,\\522 \dot y &= h\dot w+w(1-h^2)\dot z .523\end{aligned}524\]525The coefficient of \(\dot w\) in \(\dot L=(y-t)\dot y\)526agrees with the reverse computation. This calculation separates527one-hole operadic substitution along a path from the addition528that occurs at a copied input.529 530For an exact evaluation take \(w=x=1\), \(t=0\), and531\(532 b=\operatorname{artanh}(1/2)-1.533\)534Then \(h=y=1/2\) and \(L=1/8\). The two branches contribute535\(\overline\alpha=3/8\) and \(\overline\beta=1/4\), giving536\[537 (\partial_w L,\partial_b L,\partial_x L)538   =(5/8,\,3/8,\,3/8).539\]540Here \(\partial_x L\) measures input sensitivity. For a gradient541update of the two model parameters alone with step \(1/16\), the542new values are \(w'=123/128\) and \(b'=b-3/128\). With \(x\) fixed543at \(1\), the new preactivation is \(z'=z-1/16\). Indeed the two544parameter changes contribute \(5/128\) and \(3/128\) to the decrease545in \(z\). From \(\operatorname{artanh}(1/2)>1/2\) we have546\(0<z'<z\). Strict monotonicity of \(\tanh\) and \(0<w'<1\)547give \(0<y'<y\), hence the loss decreases for this update.548The accompanying script computes its new value.549 550The same point gives a mixed-derivative check. For551\(y=w\tanh(wx+b)\), denote the activation by \(\sigma\).552The relevant derivatives can be read off from the affine553operation and the two uses of \(w\):554\[555\begin{aligned}556 y_w&=\sigma +wx\sigma',557 &y_x&=w^2\sigma',\\558 y_{wx}&=2w\sigma'+w^2x\sigma'',559 & L_{wx}&=y_wy_x+(y-t)y_{wx}.560\end{aligned}