Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 621–720 of 1174. Source text is displayed literally.

621a smooth scalar loss and fixing its target.622 623The readout sends the unit output cotangent to624\(\lambda=(2aS_1,1)\). Each pooling slot receives \(\lambda\),625since all marked-slot derivatives of the latent sum are identities626in the given trivialization. At slot \(i\), the \(x_i\)-derivative of627the feature map is \((a,2ax_i)\), and the \(a\)-derivative is628\((x_i,x_i^2)\). The pairing and the parameter diagonal now give629\[630\begin{aligned}631 \partial_{x_i} F_a &= 2a^2S_1+2ax_i,\\632 \partial_a F_a &=\sum_i(2aS_1x_i+x_i^2)633                      =2aS_1^2+S_2 .634\end{aligned}635\]636The three stages have distinct roles. The symmetry of the sum637establishes forward invariance, the cotangent of the sum distributes638the readout covector to each feature branch, and the cotangent of639parameter copying adds the resulting parameter covectors.640 641At \(a=2\) and \(\mathbf x=(1,-2,3)\), the sums are \(S_1=2\),642\(S_2=14\). The resulting values are643\[644 F_a(\mathbf x)=44,\quad645 \partial_a F_a=30,\quad646 \nabla_{\mathbf x}F_a=(20,8,28).647\]648The parameter contributions from the three branches read649\(9,-12,33\) before accumulation. The attached verification650script computes these values by a reverse pass and checks651all six permutations of this input. Its central differences652also check the parameter and input gradients of both examples653and the tied model's mixed loss derivative. The latter value654is also calculated by direct multiplication and Taylor expansion655in the four-dimensional algebra with basis656\(1,\epsilon,\delta,\epsilon\delta\).657\section{Tied parameters and derived symmetry quotients}658\label{sec:derived}659 660Suppose a finite set \(S\) indexes parameter slots and a finite group661\(G\) acts on \(S\). Examples include relabeling interchangeable662branches and applying automorphisms of a fixed graph to its node slots.663Write \(G_s\) for the stabilizer of \(s\) and \(S/G\) for the set664of orbits. This symmetry gives two constructions on the same665representation. Tying restricts parameters to an invariant subspace.666Orbit pooling identifies slots by taking coinvariants. Their667relationship depends on the coefficient field and, for gradients,668on the metric chosen on the tied parameters.669 670\subsection{Balancing and averaging}671 672Let \(k\) be a field, \(A=k[G]\), and let \(M\) be a left673\(A\)-module. Regard \(A\) as a differential graded algebra674concentrated in degree zero, with augmentation675\(\epsilon(\sum_g a_g g)=\sum_g a_g\).676Use this augmentation for the right \(A\)-action on \(k\). Then677\[678 k\otimes_A M \cong M_G679 := M/\langle gm-m:g\in G,\ m\in M\rangle_k .680\]681Indeed the balancing relation identifies \(1\otimes gm\) with682\(1\otimes m\), and the inverse sends \(1\otimes m\) to the class of683\(m\). The fixed subspace \(M^G\) consists of those \(m\) satisfying684\(gm=m\) for every \(g\). For \(M=k[S]\otimes_k V\) with \(V\) a685trivial \(G\)-module, the fixed subspace consists of \(V\)-valued686vectors constant on each orbit. The quotient map to687\(k[S/G]\otimes_k V\) sums coordinates over each orbit. On a688constant vector \(v\) on an orbit \(O\) this quotient takes value689\(|O|v\).690 691\begin{proposition}[Averaging]692\label{prop:derived-averaging}693If \(|G|\) is invertible in \(k\), the composite694\(M^G\hookrightarrow M\twoheadrightarrow M_G\) is a natural695isomorphism. Its inverse sends \([m]\) to \(em\), where696\(e=|G|^{-1}\sum_{g\in G}g\).697For every \(M\), \(\operatorname{Tor}^{A}_q(k,M)=0\) for \(q>0\).698\end{proposition}699 700\begin{proof}701Both \(ge=e\) and \(eg=e\) hold by reindexing the sum. Also702\(e^2=e\) and \(e\) acts as the identity on fixed vectors.703Hence \(m\mapsto em\) factors over \(M_G\) and takes values in704\(M^G\). In \(M_G\), each \([gm]\) is \([m]\), so705\([em]=[m]\). This proves the isomorphism. The right ideal \(eA\)706is a summand of the free right module \(A\), via left multiplication707by \(e\). The map \(k\to eA\) sending \(1\) to \(e\) is an708isomorphism of right modules since \(ea=\epsilon(a)e\) and709\(\epsilon(e)=1\). The right module \(k\) is therefore projective.710\end{proof}711 712\subsection{The metric on shared parameters}713 714Over \(k=\mathbb R\), let \(V\) be a finite-dimensional Euclidean715space and \(P=V^S\) with the product inner product. The permutation716representation acts by isometries. Put \(U=V^{S/G}\). The tying717map \(i:U\to P\) sends \(\theta\) to the vector with718\(i(\theta)_s=\theta_{[s]}\), where \([s]\) denotes the orbit of719\(s\). Its image is \(P^G\). For a differentiable loss \(L:P\to720\mathbb R\), write \(g_s\) for the \(s\)-component of