Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 521–620 of 1174. Source text is displayed literally.
521 \dot z &= x\dot w+w\dot x+\dot b,\\522 \dot y &= h\dot w+w(1-h^2)\dot z .523\end{aligned}524\]525The coefficient of \(\dot w\) in \(\dot L=(y-t)\dot y\)526agrees with the reverse computation. This calculation separates527one-hole operadic substitution along a path from the addition528that occurs at a copied input.529 530For an exact evaluation take \(w=x=1\), \(t=0\), and531\(532 b=\operatorname{artanh}(1/2)-1.533\)534Then \(h=y=1/2\) and \(L=1/8\). The two branches contribute535\(\overline\alpha=3/8\) and \(\overline\beta=1/4\), giving536\[537 (\partial_w L,\partial_b L,\partial_x L)538 =(5/8,\,3/8,\,3/8).539\]540Here \(\partial_x L\) measures input sensitivity. For a gradient541update of the two model parameters alone with step \(1/16\), the542new values are \(w'=123/128\) and \(b'=b-3/128\). With \(x\) fixed543at \(1\), the new preactivation is \(z'=z-1/16\). Indeed the two544parameter changes contribute \(5/128\) and \(3/128\) to the decrease545in \(z\). From \(\operatorname{artanh}(1/2)>1/2\) we have546\(0<z'<z\). Strict monotonicity of \(\tanh\) and \(0<w'<1\)547give \(0<y'<y\), hence the loss decreases for this update.548The accompanying script computes its new value.549 550The same point gives a mixed-derivative check. For551\(y=w\tanh(wx+b)\), denote the activation by \(\sigma\).552The relevant derivatives can be read off from the affine553operation and the two uses of \(w\):554\[555\begin{aligned}556 y_w&=\sigma +wx\sigma',557 &y_x&=w^2\sigma',\\558 y_{wx}&=2w\sigma'+w^2x\sigma'',559 & L_{wx}&=y_wy_x+(y-t)y_{wx}.560\end{aligned}561\]562The values \(\sigma'=3/4\) and \(\sigma''=-3/4\) at the563displayed point give \(y_{wx}=3/4\) and \(L_{wx}=21/16\).564Equivalently this is the coefficient of \(\epsilon\delta\)565when varying \(w\) by \(\epsilon\) and \(x\) by \(\delta\) in566\(\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\). It detects567the second derivative of the activation in addition to the568compositions of first derivatives.569 570\subsection{Permutation-invariant pooling}571 572We use the sum-pooling architecture of Deep Sets~\cite{zaheer2017}573for a fixed number of input elements.574 575Let the latent space be \(Z=\mathbb R^2\), and put576\[577 \mu_n(z_1,\ldots,z_n)=\sum_{i=1}^n z_i,\qquad \mu_0=0 .578\]579Every permutation of the input slots of \(\mu_n\) gives the same580map. Substituting sums of \(m_j\) latent vectors in each slot581of \(\mu_k\) gives \(\mu_{\sum_jm_j}\). Thus the sum family582gives a commutative operad algebra in smooth spaces, including583the nullary and unary structure maps. In the real linearization,584the algebra on the free vector space generated by the points585of \(Z\) is the monoid algebra \(\mathbb R[Z,+]\), since586\([z][z']=[z+z']\) and the unit is \([0]\). In the usual587trivialization \(T_z Z=Z\), the derivative of addition in588either slot sends a vector unchanged to the fiber over the589sum. The resulting tangent module of the linearized sum590algebra is \(\mathbb R[Z,+]\otimes_{\mathbb R} Z\), with the591first factor acted on by multiplication. The identification592sends \([z]\otimes v\) to \(v\) in the summand indexed by \(z\).593 594To obtain a Deep Sets model, use the shared smooth feature map595\(\phi_a(s)=(as,as^2)\), a sum, and the readout596\(\rho(u,v)=u^2+v\). Here \(a\) is a trainable scalar. The feature597and readout maps are smooth operations in the surrounding colored598operad. At a fixed size \(n\) the composite is599\[600 \begin{gathered}601 F_a(\mathbf x)=\rho\!\left(\sum_i\phi_a(x_i)\right)602 =(aS_1)^2+aS_2,\\603 S_j=\sum_i x_i^j .604 \end{gathered}605\]606A permutation of \(\mathbf x\) permutes the latent vectors in607the same way because every slot uses the same \(\phi_a\).608The sum coequalizes the permutation action, so \(F_a\) is609invariant and descends set-theoretically to the orbit set.610The Euclidean gradient is calculated on the ordered input space.611For a permutation matrix \(P\), differentiate612\(F_a(P\mathbf x)=F_a(\mathbf x)\) in \(\mathbf x\).613The chain rule and \(P^{\mathsf T}P=I\) give614\[615 \nabla_{\mathbf x}F_a(P\mathbf x)616 =P\nabla_{\mathbf x}F_a(\mathbf x).617\]618Differentiation in \(a\) instead gives an invariant scalar619\(\partial_a F_a(P\mathbf x)=\partial_a F_a(\mathbf x)\).620These two transformation laws also hold after composing with