Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 674–773 of 1174. Source text is displayed literally.

674concentrated in degree zero, with augmentation675\(\epsilon(\sum_g a_g g)=\sum_g a_g\).676Use this augmentation for the right \(A\)-action on \(k\). Then677\[678 k\otimes_A M \cong M_G679 := M/\langle gm-m:g\in G,\ m\in M\rangle_k .680\]681Indeed the balancing relation identifies \(1\otimes gm\) with682\(1\otimes m\), and the inverse sends \(1\otimes m\) to the class of683\(m\). The fixed subspace \(M^G\) consists of those \(m\) satisfying684\(gm=m\) for every \(g\). For \(M=k[S]\otimes_k V\) with \(V\) a685trivial \(G\)-module, the fixed subspace consists of \(V\)-valued686vectors constant on each orbit. The quotient map to687\(k[S/G]\otimes_k V\) sums coordinates over each orbit. On a688constant vector \(v\) on an orbit \(O\) this quotient takes value689\(|O|v\).690 691\begin{proposition}[Averaging]692\label{prop:derived-averaging}693If \(|G|\) is invertible in \(k\), the composite694\(M^G\hookrightarrow M\twoheadrightarrow M_G\) is a natural695isomorphism. Its inverse sends \([m]\) to \(em\), where696\(e=|G|^{-1}\sum_{g\in G}g\).697For every \(M\), \(\operatorname{Tor}^{A}_q(k,M)=0\) for \(q>0\).698\end{proposition}699 700\begin{proof}701Both \(ge=e\) and \(eg=e\) hold by reindexing the sum. Also702\(e^2=e\) and \(e\) acts as the identity on fixed vectors.703Hence \(m\mapsto em\) factors over \(M_G\) and takes values in704\(M^G\). In \(M_G\), each \([gm]\) is \([m]\), so705\([em]=[m]\). This proves the isomorphism. The right ideal \(eA\)706is a summand of the free right module \(A\), via left multiplication707by \(e\). The map \(k\to eA\) sending \(1\) to \(e\) is an708isomorphism of right modules since \(ea=\epsilon(a)e\) and709\(\epsilon(e)=1\). The right module \(k\) is therefore projective.710\end{proof}711 712\subsection{The metric on shared parameters}713 714Over \(k=\mathbb R\), let \(V\) be a finite-dimensional Euclidean715space and \(P=V^S\) with the product inner product. The permutation716representation acts by isometries. Put \(U=V^{S/G}\). The tying717map \(i:U\to P\) sends \(\theta\) to the vector with718\(i(\theta)_s=\theta_{[s]}\), where \([s]\) denotes the orbit of719\(s\). Its image is \(P^G\). For a differentiable loss \(L:P\to720\mathbb R\), write \(g_s\) for the \(s\)-component of721\(\nabla L(i\theta)\). The tied derivative is the pullback722\(i^* DL(i\theta)\). On \(v\in U\) it is723\[724 D(L\circ i)(\theta)[v]725 =\sum_{O\in S/G}726   \left\langle\sum_{s\in O}g_s,v_O\right\rangle .727\]728With the ordinary product metric on \(U\), the \(O\)-component729of the gradient is \(\sum_{s\in O}g_s\). The metric induced by730\(i\) has inner product731\[732 \langle u,v\rangle_i733 =\sum_O |O|\langle u_O,v_O\rangle.734\]735In the induced metric, the component for \(O\) becomes736the average of the \(g_s\) over \(O\). Let \(R_G\) be the737Reynolds operator \(m\mapsto em\) on \(P\). Then738\[739 i\nabla_i(L\circ i)(\theta)=R_G\nabla L(i\theta).740\]741Indeed for \(s\in O\) every index \(t\in O\) occurs as \(g^{-1}s\)742for exactly \(|G_s|\) group elements, so \(R_G\) averages743uniformly over \(O\). This is also the orthogonal projection744onto \(P^G\): the operator is idempotent, and reindexing by745\(g^{-1}\) gives self-adjointness.746 747For example, let \(C_2\) swap \(a,b\) and fix \(c\), and take748\(V=\mathbb R\). The map \(i\) sends \((\theta,\eta)\) to749\((\theta,\theta,\eta)\). For \(y=(1,3,2)\) and750\(L(p)=\tfrac12\sum_{s\in S}(p_s-y_s)^2\), the product-metric751gradient in \(U\) is \((2\theta-4,\eta-2)\), while the induced752metric gives \((\theta-2,\eta-2)\). At the origin the first753shared coordinate receives two contributions to the derivative,754and the induced metric divides that sum by the orbit size.755This difference matters when a single learning rate is used756across orbits of unequal sizes.757 758When the ambient loss is invariant, its ambient gradient at a759tied point is itself fixed. Differentiating \(L\circ g=L\)760at \(p\in P^G\) gives \(DL(p)[g v]=DL(p)[v]\). Since \(g\)761acts orthogonally, uniqueness of the Riesz representative762gives \(g\nabla L(p)=\nabla L(p)\). Consequently the induced763gradient agrees with the ambient gradient after inclusion by764\(i\), and the product-metric gradient still includes the765orbit-size factors. This identifies the metric implicit in766restricting Euclidean gradient flow to the tied subspace.767 768Tying also has a direct architectural consequence. Suppose a769parameterized family \(F:P\times X\to Y\) satisfies diagonal770equivariance, \(F(gp,gx)=gF(p,x)\), for given \(G\)-actions on771\(X,Y\). Each tied parameter \(i\theta\) gives an equivariant772model, since \(g(i\theta)=i\theta\) and thus773\(F(i\theta,gx)=gF(i\theta,x)\). The hypothesis is a property of