Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 601–700 of 1174. Source text is displayed literally.

601 F_a(\mathbf x)=\rho\!\left(\sum_i\phi_a(x_i)\right)602          =(aS_1)^2+aS_2,\\603 S_j=\sum_i x_i^j .604 \end{gathered}605\]606A permutation of \(\mathbf x\) permutes the latent vectors in607the same way because every slot uses the same \(\phi_a\).608The sum coequalizes the permutation action, so \(F_a\) is609invariant and descends set-theoretically to the orbit set.610The Euclidean gradient is calculated on the ordered input space.611For a permutation matrix \(P\), differentiate612\(F_a(P\mathbf x)=F_a(\mathbf x)\) in \(\mathbf x\).613The chain rule and \(P^{\mathsf T}P=I\) give614\[615 \nabla_{\mathbf x}F_a(P\mathbf x)616       =P\nabla_{\mathbf x}F_a(\mathbf x).617\]618Differentiation in \(a\) instead gives an invariant scalar619\(\partial_a F_a(P\mathbf x)=\partial_a F_a(\mathbf x)\).620These two transformation laws also hold after composing with621a smooth scalar loss and fixing its target.622 623The readout sends the unit output cotangent to624\(\lambda=(2aS_1,1)\). Each pooling slot receives \(\lambda\),625since all marked-slot derivatives of the latent sum are identities626in the given trivialization. At slot \(i\), the \(x_i\)-derivative of627the feature map is \((a,2ax_i)\), and the \(a\)-derivative is628\((x_i,x_i^2)\). The pairing and the parameter diagonal now give629\[630\begin{aligned}631 \partial_{x_i} F_a &= 2a^2S_1+2ax_i,\\632 \partial_a F_a &=\sum_i(2aS_1x_i+x_i^2)633                      =2aS_1^2+S_2 .634\end{aligned}635\]636The three stages have distinct roles. The symmetry of the sum637establishes forward invariance, the cotangent of the sum distributes638the readout covector to each feature branch, and the cotangent of639parameter copying adds the resulting parameter covectors.640 641At \(a=2\) and \(\mathbf x=(1,-2,3)\), the sums are \(S_1=2\),642\(S_2=14\). The resulting values are643\[644 F_a(\mathbf x)=44,\quad645 \partial_a F_a=30,\quad646 \nabla_{\mathbf x}F_a=(20,8,28).647\]648The parameter contributions from the three branches read649\(9,-12,33\) before accumulation. The attached verification650script computes these values by a reverse pass and checks651all six permutations of this input. Its central differences652also check the parameter and input gradients of both examples653and the tied model's mixed loss derivative. The latter value654is also calculated by direct multiplication and Taylor expansion655in the four-dimensional algebra with basis656\(1,\epsilon,\delta,\epsilon\delta\).657\section{Tied parameters and derived symmetry quotients}658\label{sec:derived}659 660Suppose a finite set \(S\) indexes parameter slots and a finite group661\(G\) acts on \(S\). Examples include relabeling interchangeable662branches and applying automorphisms of a fixed graph to its node slots.663Write \(G_s\) for the stabilizer of \(s\) and \(S/G\) for the set664of orbits. This symmetry gives two constructions on the same665representation. Tying restricts parameters to an invariant subspace.666Orbit pooling identifies slots by taking coinvariants. Their667relationship depends on the coefficient field and, for gradients,668on the metric chosen on the tied parameters.669 670\subsection{Balancing and averaging}671 672Let \(k\) be a field, \(A=k[G]\), and let \(M\) be a left673\(A\)-module. Regard \(A\) as a differential graded algebra674concentrated in degree zero, with augmentation675\(\epsilon(\sum_g a_g g)=\sum_g a_g\).676Use this augmentation for the right \(A\)-action on \(k\). Then677\[678 k\otimes_A M \cong M_G679 := M/\langle gm-m:g\in G,\ m\in M\rangle_k .680\]681Indeed the balancing relation identifies \(1\otimes gm\) with682\(1\otimes m\), and the inverse sends \(1\otimes m\) to the class of683\(m\). The fixed subspace \(M^G\) consists of those \(m\) satisfying684\(gm=m\) for every \(g\). For \(M=k[S]\otimes_k V\) with \(V\) a685trivial \(G\)-module, the fixed subspace consists of \(V\)-valued686vectors constant on each orbit. The quotient map to687\(k[S/G]\otimes_k V\) sums coordinates over each orbit. On a688constant vector \(v\) on an orbit \(O\) this quotient takes value689\(|O|v\).690 691\begin{proposition}[Averaging]692\label{prop:derived-averaging}693If \(|G|\) is invertible in \(k\), the composite694\(M^G\hookrightarrow M\twoheadrightarrow M_G\) is a natural695isomorphism. Its inverse sends \([m]\) to \(em\), where696\(e=|G|^{-1}\sum_{g\in G}g\).697For every \(M\), \(\operatorname{Tor}^{A}_q(k,M)=0\) for \(q>0\).698\end{proposition}699 700\begin{proof}