Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 581–680 of 1174. Source text is displayed literally.

581of \(\mu_k\) gives \(\mu_{\sum_jm_j}\). Thus the sum family582gives a commutative operad algebra in smooth spaces, including583the nullary and unary structure maps. In the real linearization,584the algebra on the free vector space generated by the points585of \(Z\) is the monoid algebra \(\mathbb R[Z,+]\), since586\([z][z']=[z+z']\) and the unit is \([0]\). In the usual587trivialization \(T_z Z=Z\), the derivative of addition in588either slot sends a vector unchanged to the fiber over the589sum. The resulting tangent module of the linearized sum590algebra is \(\mathbb R[Z,+]\otimes_{\mathbb R} Z\), with the591first factor acted on by multiplication. The identification592sends \([z]\otimes v\) to \(v\) in the summand indexed by \(z\).593 594To obtain a Deep Sets model, use the shared smooth feature map595\(\phi_a(s)=(as,as^2)\), a sum, and the readout596\(\rho(u,v)=u^2+v\). Here \(a\) is a trainable scalar. The feature597and readout maps are smooth operations in the surrounding colored598operad. At a fixed size \(n\) the composite is599\[600 \begin{gathered}601 F_a(\mathbf x)=\rho\!\left(\sum_i\phi_a(x_i)\right)602          =(aS_1)^2+aS_2,\\603 S_j=\sum_i x_i^j .604 \end{gathered}605\]606A permutation of \(\mathbf x\) permutes the latent vectors in607the same way because every slot uses the same \(\phi_a\).608The sum coequalizes the permutation action, so \(F_a\) is609invariant and descends set-theoretically to the orbit set.610The Euclidean gradient is calculated on the ordered input space.611For a permutation matrix \(P\), differentiate612\(F_a(P\mathbf x)=F_a(\mathbf x)\) in \(\mathbf x\).613The chain rule and \(P^{\mathsf T}P=I\) give614\[615 \nabla_{\mathbf x}F_a(P\mathbf x)616       =P\nabla_{\mathbf x}F_a(\mathbf x).617\]618Differentiation in \(a\) instead gives an invariant scalar619\(\partial_a F_a(P\mathbf x)=\partial_a F_a(\mathbf x)\).620These two transformation laws also hold after composing with621a smooth scalar loss and fixing its target.622 623The readout sends the unit output cotangent to624\(\lambda=(2aS_1,1)\). Each pooling slot receives \(\lambda\),625since all marked-slot derivatives of the latent sum are identities626in the given trivialization. At slot \(i\), the \(x_i\)-derivative of627the feature map is \((a,2ax_i)\), and the \(a\)-derivative is628\((x_i,x_i^2)\). The pairing and the parameter diagonal now give629\[630\begin{aligned}631 \partial_{x_i} F_a &= 2a^2S_1+2ax_i,\\632 \partial_a F_a &=\sum_i(2aS_1x_i+x_i^2)633                      =2aS_1^2+S_2 .634\end{aligned}635\]636The three stages have distinct roles. The symmetry of the sum637establishes forward invariance, the cotangent of the sum distributes638the readout covector to each feature branch, and the cotangent of639parameter copying adds the resulting parameter covectors.640 641At \(a=2\) and \(\mathbf x=(1,-2,3)\), the sums are \(S_1=2\),642\(S_2=14\). The resulting values are643\[644 F_a(\mathbf x)=44,\quad645 \partial_a F_a=30,\quad646 \nabla_{\mathbf x}F_a=(20,8,28).647\]648The parameter contributions from the three branches read649\(9,-12,33\) before accumulation. The attached verification650script computes these values by a reverse pass and checks651all six permutations of this input. Its central differences652also check the parameter and input gradients of both examples653and the tied model's mixed loss derivative. The latter value654is also calculated by direct multiplication and Taylor expansion655in the four-dimensional algebra with basis656\(1,\epsilon,\delta,\epsilon\delta\).657\section{Tied parameters and derived symmetry quotients}658\label{sec:derived}659 660Suppose a finite set \(S\) indexes parameter slots and a finite group661\(G\) acts on \(S\). Examples include relabeling interchangeable662branches and applying automorphisms of a fixed graph to its node slots.663Write \(G_s\) for the stabilizer of \(s\) and \(S/G\) for the set664of orbits. This symmetry gives two constructions on the same665representation. Tying restricts parameters to an invariant subspace.666Orbit pooling identifies slots by taking coinvariants. Their667relationship depends on the coefficient field and, for gradients,668on the metric chosen on the tied parameters.669 670\subsection{Balancing and averaging}671 672Let \(k\) be a field, \(A=k[G]\), and let \(M\) be a left673\(A\)-module. Regard \(A\) as a differential graded algebra674concentrated in degree zero, with augmentation675\(\epsilon(\sum_g a_g g)=\sum_g a_g\).676Use this augmentation for the right \(A\)-action on \(k\). Then677\[678 k\otimes_A M \cong M_G679 := M/\langle gm-m:g\in G,\ m\in M\rangle_k .680\]