Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 701–800 of 1174. Source text is displayed literally.

701Both \(ge=e\) and \(eg=e\) hold by reindexing the sum. Also702\(e^2=e\) and \(e\) acts as the identity on fixed vectors.703Hence \(m\mapsto em\) factors over \(M_G\) and takes values in704\(M^G\). In \(M_G\), each \([gm]\) is \([m]\), so705\([em]=[m]\). This proves the isomorphism. The right ideal \(eA\)706is a summand of the free right module \(A\), via left multiplication707by \(e\). The map \(k\to eA\) sending \(1\) to \(e\) is an708isomorphism of right modules since \(ea=\epsilon(a)e\) and709\(\epsilon(e)=1\). The right module \(k\) is therefore projective.710\end{proof}711 712\subsection{The metric on shared parameters}713 714Over \(k=\mathbb R\), let \(V\) be a finite-dimensional Euclidean715space and \(P=V^S\) with the product inner product. The permutation716representation acts by isometries. Put \(U=V^{S/G}\). The tying717map \(i:U\to P\) sends \(\theta\) to the vector with718\(i(\theta)_s=\theta_{[s]}\), where \([s]\) denotes the orbit of719\(s\). Its image is \(P^G\). For a differentiable loss \(L:P\to720\mathbb R\), write \(g_s\) for the \(s\)-component of721\(\nabla L(i\theta)\). The tied derivative is the pullback722\(i^* DL(i\theta)\). On \(v\in U\) it is723\[724 D(L\circ i)(\theta)[v]725 =\sum_{O\in S/G}726   \left\langle\sum_{s\in O}g_s,v_O\right\rangle .727\]728With the ordinary product metric on \(U\), the \(O\)-component729of the gradient is \(\sum_{s\in O}g_s\). The metric induced by730\(i\) has inner product731\[732 \langle u,v\rangle_i733 =\sum_O |O|\langle u_O,v_O\rangle.734\]735In the induced metric, the component for \(O\) becomes736the average of the \(g_s\) over \(O\). Let \(R_G\) be the737Reynolds operator \(m\mapsto em\) on \(P\). Then738\[739 i\nabla_i(L\circ i)(\theta)=R_G\nabla L(i\theta).740\]741Indeed for \(s\in O\) every index \(t\in O\) occurs as \(g^{-1}s\)742for exactly \(|G_s|\) group elements, so \(R_G\) averages743uniformly over \(O\). This is also the orthogonal projection744onto \(P^G\): the operator is idempotent, and reindexing by745\(g^{-1}\) gives self-adjointness.746 747For example, let \(C_2\) swap \(a,b\) and fix \(c\), and take748\(V=\mathbb R\). The map \(i\) sends \((\theta,\eta)\) to749\((\theta,\theta,\eta)\). For \(y=(1,3,2)\) and750\(L(p)=\tfrac12\sum_{s\in S}(p_s-y_s)^2\), the product-metric751gradient in \(U\) is \((2\theta-4,\eta-2)\), while the induced752metric gives \((\theta-2,\eta-2)\). At the origin the first753shared coordinate receives two contributions to the derivative,754and the induced metric divides that sum by the orbit size.755This difference matters when a single learning rate is used756across orbits of unequal sizes.757 758When the ambient loss is invariant, its ambient gradient at a759tied point is itself fixed. Differentiating \(L\circ g=L\)760at \(p\in P^G\) gives \(DL(p)[g v]=DL(p)[v]\). Since \(g\)761acts orthogonally, uniqueness of the Riesz representative762gives \(g\nabla L(p)=\nabla L(p)\). Consequently the induced763gradient agrees with the ambient gradient after inclusion by764\(i\), and the product-metric gradient still includes the765orbit-size factors. This identifies the metric implicit in766restricting Euclidean gradient flow to the tied subspace.767 768Tying also has a direct architectural consequence. Suppose a769parameterized family \(F:P\times X\to Y\) satisfies diagonal770equivariance, \(F(gp,gx)=gF(p,x)\), for given \(G\)-actions on771\(X,Y\). Each tied parameter \(i\theta\) gives an equivariant772model, since \(g(i\theta)=i\theta\) and thus773\(F(i\theta,gx)=gF(i\theta,x)\). The hypothesis is a property of774the untied family, checkable at its generating operations.775 776\subsection{The two-sided bar and stabilizer homology}777 778The balancing construction is the degree-zero part of a derived779tensor product. K\v{r}\'i\v{z} and May develop derived tensor780products of differential graded modules by cell approximation in781Part~III, \S4, starting on p.~53 \cite{kriz-may}. For augmented782algebras the two-sided tensor bar, reviewed in Part~IV, \S6,783p.~71, supplies a concrete model. We use its simplicial formula784for the associative algebra \(A=k[G]\), with homological degrees785increasing with bar length. Their operadic comparison in786Part~II, \S4, p.~35 uses a monad bar \(B(F,C,A)\) with operadic787monad \(C\) and a \(C\)-algebra \(A\). The bar here uses tensor788powers of the group algebra and its ordinary right and left789modules.790 791Explicitly, use \(B_q(k,A,M)=k\otimes_k A^{\otimes q}\otimes_k M\)792with differential given by the alternating sum of faces. The first793face applies the augmentation to the first algebra factor,794interior faces multiply two adjacent factors, and the last face795acts on \(M\). Degeneracies insert the unit. Replacing \(M\) by796\(A\) gives a free right \(A\)-module in every degree. The797augmented underlying complex contracts by appending \(1\in A\)798as an extra last factor, with sign \((-1)^{q+1}\) in degree \(q\).799Indeed the last face of this extra degeneracy is the identity800and every earlier face commutes with it in the adjacent degree.