Indexed text attachment
topos-actions-in-machine-learning-and-ai.tex
Source for Topos Actions in Machine Learning and Artificial Intelligence
View paper · Download original file
Lines 787–886 of 1207. Source text is displayed literally.
787discrete step \cite{lee-riemannian}. Using a new metric788without transporting it can change the update even789when the predictions and losses agree.790 791For example, let \(q=2p\) and \(\ell(p)=p^2/2\).792The unit metric in \(p\) transports to793\(\tfrac14\,dq^2\), giving gradient component \(q\).794Independently choosing the unit metric in \(q\) gives795component \(q/4\) for the same loss \(q^2/8\).796Thus the descent test for an optimizer concerns both797its prediction and its optimization data. For a798stateful optimizer the state and its transport can be799included in the updated parameter object, with the800same comparison condition on the enlarged update.801\par802\section{Two uses of quotient and local symmetry}803\label{sec:applications}804 805The first use chooses a nuisance action by a monoidal806topos and computes the resulting space of feasible layers.807The second uses a symmetry that varies as an internal group808and computes the admissible coefficients of a linear network.809 810\subsection{A central action and nuisance-invariant learning}811\label{sec:nuisance}812 813Let $A$ be an abelian discrete group, let $H$ be a814discrete group, and let $\chi:A\to Z(H)$ be a homomorphism.815Write $N=\chi(A)$ and $K=H/N$. The induced action of816$\mathbf B A$ on $\mathbf B H$ has geometric quotient817$q:\mathbf B H\to\mathbf B K$. Its inverse image inflates818a $K$-action along $H\twoheadrightarrow K$.819An $H$-object descends precisely when its $N$-action820is trivial \cite{topos-actions}. This is a literal action821between topoi of the kind discussed in822Section~\ref{sec:introduction}.823 824\begin{proposition}[Orbit features from a central quotient]825\label{prop:orbit-layer}826Let $P,W$ be $H$-sets with trivial $N$-action, and $V$ an827arbitrary $H$-set. An $H$-equivariant map $F:P\times V\to W$828corresponds uniquely to a $K$-equivariant map829$\overline F:P\times(V/N)\to W$.830For topological actions and continuous layers the same831statement holds with the orbit topology on $V/N$.832\end{proposition}833 834\begin{proof}835Equivariance and the hypotheses give836$F(p,nv)=nF(p,v)=F(p,v)$. Therefore $F$ is constant837on the fibers of $(p,v)\mapsto(p,[v])$ and factors.838Normality of $N$ lets $H$ act on the orbits through $K$.839The factor map is consequently equivariant for $K$,840and the converse follows by composing with the quotient.841For continuous actions, $V\to V/N$ is an open surjection842since the saturation of an open set is the union of its843translates. Its product with $P$ is also an open surjection,844so the factor map is continuous by the quotient property.845\end{proof}846 847Suppose now that $N$ is finite and $V,W$ are real linear848representations. For $W$ with trivial $N$-action, an849$N$-invariant \emph{linear} map out of $V$ satisfies850$L=L\Pi_N$, where851\[852 \Pi_N v=\frac{1}{|N|}\sum_{n\in N} n v .853\]854The operator is a projector onto $V^N$, and for an855$H$-representation it commutes with $H$ because $N$ is856normal. The same factorization holds for an affine map857by adding its constant part. The orbit feature $V/N$858instead retains the information needed for all invariant859nonlinear maps. To see the difference, take860$H=C_4$ acting by quarter turns on $V=\mathbb R^2$ and861$N=C_2$ acting by $v\mapsto-v$. This is obtained from862$\chi:C_2\to C_4$ with image the half turn. Here863$\Pi_N=0$. A label on the input set864$\{0,\pm e_1,\pm e_2\}$ indicating whether it is nonzero865is predicted exactly by the invariant square-activation866score $\|v\|^2$. If half of the distribution is at $0$867and half is uniform on the other four points, a predictor868fed only $\Pi_N v$ is constant. Its minimum mean square869error is $1/4$, the variance of the label. The nonlinear870invariant predictor has zero error. The quotient871determines which information may be discarded, and an872implementation still has to choose an orbit encoding.873 874There is a related operation on predictors rather than875input features. If $\mu$ is the joint distribution of876input and label, assume $\mu$ is invariant under877$(v,y)\mapsto(nv,y)$. For a vector score $s$ and a loss878$\ell(y,-)$ convex in the score, define879$\overline s(v)=|N|^{-1}\sum_n s(nv)$.880For measurable scores and a nonnegative loss with finite risk,881Jensen's inequality gives882\begin{align*}883 \mathbb E_\mu[\ell(y,\overline s(v))]884 &\leq \frac1{|N|}\sum_{n\in N}885 \mathbb E_\mu[\ell(y,s(nv))]\\886 &=\mathbb E_\mu[\ell(y,s(v))].