Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 561–660 of 1174. Source text is displayed literally.

561\]562The values \(\sigma'=3/4\) and \(\sigma''=-3/4\) at the563displayed point give \(y_{wx}=3/4\) and \(L_{wx}=21/16\).564Equivalently this is the coefficient of \(\epsilon\delta\)565when varying \(w\) by \(\epsilon\) and \(x\) by \(\delta\) in566\(\mathbb R[\epsilon,\delta]/(\epsilon^2,\delta^2)\). It detects567the second derivative of the activation in addition to the568compositions of first derivatives.569 570\subsection{Permutation-invariant pooling}571 572We use the sum-pooling architecture of Deep Sets~\cite{zaheer2017}573for a fixed number of input elements.574 575Let the latent space be \(Z=\mathbb R^2\), and put576\[577 \mu_n(z_1,\ldots,z_n)=\sum_{i=1}^n z_i,\qquad \mu_0=0 .578\]579Every permutation of the input slots of \(\mu_n\) gives the same580map. Substituting sums of \(m_j\) latent vectors in each slot581of \(\mu_k\) gives \(\mu_{\sum_jm_j}\). Thus the sum family582gives a commutative operad algebra in smooth spaces, including583the nullary and unary structure maps. In the real linearization,584the algebra on the free vector space generated by the points585of \(Z\) is the monoid algebra \(\mathbb R[Z,+]\), since586\([z][z']=[z+z']\) and the unit is \([0]\). In the usual587trivialization \(T_z Z=Z\), the derivative of addition in588either slot sends a vector unchanged to the fiber over the589sum. The resulting tangent module of the linearized sum590algebra is \(\mathbb R[Z,+]\otimes_{\mathbb R} Z\), with the591first factor acted on by multiplication. The identification592sends \([z]\otimes v\) to \(v\) in the summand indexed by \(z\).593 594To obtain a Deep Sets model, use the shared smooth feature map595\(\phi_a(s)=(as,as^2)\), a sum, and the readout596\(\rho(u,v)=u^2+v\). Here \(a\) is a trainable scalar. The feature597and readout maps are smooth operations in the surrounding colored598operad. At a fixed size \(n\) the composite is599\[600 \begin{gathered}601 F_a(\mathbf x)=\rho\!\left(\sum_i\phi_a(x_i)\right)602          =(aS_1)^2+aS_2,\\603 S_j=\sum_i x_i^j .604 \end{gathered}605\]606A permutation of \(\mathbf x\) permutes the latent vectors in607the same way because every slot uses the same \(\phi_a\).608The sum coequalizes the permutation action, so \(F_a\) is609invariant and descends set-theoretically to the orbit set.610The Euclidean gradient is calculated on the ordered input space.611For a permutation matrix \(P\), differentiate612\(F_a(P\mathbf x)=F_a(\mathbf x)\) in \(\mathbf x\).613The chain rule and \(P^{\mathsf T}P=I\) give614\[615 \nabla_{\mathbf x}F_a(P\mathbf x)616       =P\nabla_{\mathbf x}F_a(\mathbf x).617\]618Differentiation in \(a\) instead gives an invariant scalar619\(\partial_a F_a(P\mathbf x)=\partial_a F_a(\mathbf x)\).620These two transformation laws also hold after composing with621a smooth scalar loss and fixing its target.622 623The readout sends the unit output cotangent to624\(\lambda=(2aS_1,1)\). Each pooling slot receives \(\lambda\),625since all marked-slot derivatives of the latent sum are identities626in the given trivialization. At slot \(i\), the \(x_i\)-derivative of627the feature map is \((a,2ax_i)\), and the \(a\)-derivative is628\((x_i,x_i^2)\). The pairing and the parameter diagonal now give629\[630\begin{aligned}631 \partial_{x_i} F_a &= 2a^2S_1+2ax_i,\\632 \partial_a F_a &=\sum_i(2aS_1x_i+x_i^2)633                      =2aS_1^2+S_2 .634\end{aligned}635\]636The three stages have distinct roles. The symmetry of the sum637establishes forward invariance, the cotangent of the sum distributes638the readout covector to each feature branch, and the cotangent of639parameter copying adds the resulting parameter covectors.640 641At \(a=2\) and \(\mathbf x=(1,-2,3)\), the sums are \(S_1=2\),642\(S_2=14\). The resulting values are643\[644 F_a(\mathbf x)=44,\quad645 \partial_a F_a=30,\quad646 \nabla_{\mathbf x}F_a=(20,8,28).647\]648The parameter contributions from the three branches read649\(9,-12,33\) before accumulation. The attached verification650script computes these values by a reverse pass and checks651all six permutations of this input. Its central differences652also check the parameter and input gradients of both examples653and the tied model's mixed loss derivative. The latter value654is also calculated by direct multiplication and Taylor expansion655in the four-dimensional algebra with basis656\(1,\epsilon,\delta,\epsilon\delta\).657\section{Tied parameters and derived symmetry quotients}658\label{sec:derived}659 660Suppose a finite set \(S\) indexes parameter slots and a finite group