Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 481–580 of 1174. Source text is displayed literally.

481and then impose the sharing relation.482 483Let \(\overline s\) denote the cotangent of an intermediate scalar484\(s\) for this loss. The last multiplication has marked-slot maps485given by multiplication by \(h\) in the \(\beta\) slot and by486\(\beta\) in the \(h\) slot. Taking adjoints in the standard487scalar pairing gives488\[489 \overline y=y-t,\qquad490 \overline\beta=h\,\overline y,\qquad491 \overline h=\beta\,\overline y.492\]493The activation and affine operation then give494\[495\begin{aligned}496 \overline z &= (1-h^2)\overline h,\\497 \overline\alpha &= x\,\overline z,\qquad498 \overline x=\alpha\,\overline z,\qquad499 \overline b=\overline z .500\end{aligned}501\]502Each expression is determined by a one-slot map and its downstream503cotangent. In the algebra-module construction above, the underlying504slot is based at its particular forward value. For example, the slot505map from \(T_z\mathbb R\) to \(T_h\mathbb R\) multiplies by \(1-h^2\),506with the coefficient evaluated at the given pass through the graph.507 508Tie the parameters using the diagonal509\(\Delta(w)=(\alpha,\beta)=(w,w)\). The parameter variation510\(\dot w\) has image \((\dot w,\dot w)\). Pairing this image with511\((\overline\alpha,\overline\beta)\) gives512\[513 \overline w=\overline\alpha+\overline\beta514   =(y-t)\big(h+wx(1-h^2)\big).515\]516The first term inside the parentheses comes from the direct scaling517of \(h\), and the second comes from the use of \(w\) before the518activation. In a forward calculation the same two paths give519\[520\begin{aligned}521 \dot z &= x\dot w+w\dot x+\dot b,\\522 \dot y &= h\dot w+w(1-h^2)\dot z .523\end{aligned}524\]525The coefficient of \(\dot w\) in \(\dot L=(y-t)\dot y\)526agrees with the reverse computation. This calculation separates527one-hole operadic substitution along a path from the addition528that occurs at a copied input.529 530For an exact evaluation take \(w=x=1\), \(t=0\), and531\(532 b=\operatorname{artanh}(1/2)-1.533\)534Then \(h=y=1/2\) and \(L=1/8\). The two branches contribute535\(\overline\alpha=3/8\) and \(\overline\beta=1/4\), giving536\[537 (\partial_w L,\partial_b L,\partial_x L)538   =(5/8,\,3/8,\,3/8).539\]540Here \(\partial_x L\) measures input sensitivity. For a gradient541update of the two model parameters alone with step \(1/16\), the542new values are \(w'=123/128\) and \(b'=b-3/128\). With \(x\) fixed543at \(1\), the new preactivation is \(z'=z-1/16\). Indeed the two544parameter changes contribute \(5/128\) and \(3/128\) to the decrease545in \(z\). From \(\operatorname{artanh}(1/2)>1/2\) we have546\(0<z'<z\). Strict monotonicity of \(\tanh\) and \(0<w'<1\)547give \(0<y'<y\), hence the loss decreases for this update.548The accompanying script computes its new value.549 550The same point gives a mixed-derivative check. For551\(y=w\tanh(wx+b)\), denote the activation by \(\sigma\).552The relevant derivatives can be read off from the affine553operation and the two uses of \(w\):554\[555\begin{aligned}556 y_w&=\sigma +wx\sigma',557 &y_x&=w^2\sigma',\\558 y_{wx}&=2w\sigma'+w^2x\sigma'',559 & L_{wx}&=y_wy_x+(y-t)y_{wx}.560\end{aligned}561\]562The values \(\sigma'=3/4\) and \(\sigma''=-3/4\) at the563displayed point give \(y_{wx}=3/4\) and \(L_{wx}=21/16\).564Equivalently this is the coefficient of \(\epsilon\delta\)565when varying \(w\) by \(\epsilon\) and \(x\) by \(\delta\) in566\(\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\). It detects567the second derivative of the activation in addition to the568compositions of first derivatives.569 570\subsection{Permutation-invariant pooling}571 572We use the sum-pooling architecture of Deep Sets~\cite{zaheer2017}573for a fixed number of input elements.574 575Let the latent space be \(Z=\mathbb R^2\), and put576\[577 \mu_n(z_1,\ldots,z_n)=\sum_{i=1}^n z_i,\qquad \mu_0=0 .578\]579Every permutation of the input slots of \(\mu_n\) gives the same580map. Substituting sums of \(m_j\) latent vectors in each slot