Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 523–622 of 1343. Source text is displayed literally.

523information beyond the set of distinct output laws. Even the524infinitesimal null space can include directions that change the525output at higher order: along \((a,\varepsilon,\varepsilon)\) through526\((a,0,0)\), \(d=\varepsilon^2\) changes the prediction to second order.527Consequently equality of output laws, Fisher-null directions, and528parameter equivalence answer three different questions.529 530\subsection{A manifold decoder as a statistical model}531 532Let $Z$ be a latent manifold with fixed probability $\nu$ and533$d:P\times Z\to M$ a smooth decoder. In the absence of observation534noise the law at $p$ is $(d_p)_*\nu$. A noise channel $N:M\to Y$535then gives the observed law536\[537 \mathbb P_p(A)=\int_Z N(d_p(z),A)\,d\nu(z)538\]539for measurable $A\subseteq Y$. This is ordinary composition of540Markov kernels on these measurable spaces. The manifold541evaluation $d$ can be represented by the smooth slice542construction of Section~\ref{sec:relative}. The pushforward543additionally uses the chosen law $\nu$ and noise $N$.544If $a:Z\to Z$ preserves $\nu$, then replacing $d_p$ by545$d_p\circ a$ leaves $\mathbb P_p$ unchanged, by the546change-of-variables identity for pushforwards. A law that547observes $z$ and the output together can distinguish the two.548Thus the observation protocol determines which parameter549symmetries are invisible.550 551For an explicit case take $Z=S^1$ with Haar probability and552the ellipse $e(t)=(a\cos t,b\sin t)$ in $\mathbb R^2$ for553$a,b>0$ with $a\ne b$. A hidden phase $\theta$ in554$t\mapsto e(t+\theta)$ is entirely unidentifiable after555integrating out $t$. Adding centered isotropic Gaussian556noise with covariance $\sigma^2 I$ preserves this557invariance. The integrated Gaussian mixture density is558independent of $\theta$, so its Fisher information in559the phase direction is zero. In contrast, an output560orientation in $t\mapsto R_\theta e(t)$ rotates the561signal covariance:562\[563 \operatorname{Cov}_\theta(Y)=564 \tfrac12 R_\theta565 \begin{pmatrix}a^2&0\\0&b^2\end{pmatrix}R_\theta^\top566 +\sigma^2 I .567\]568Because $a^2\ne b^2$, equality of these covariances forces569coincidence of the principal axes and hence equality of570$\theta$ modulo $\pi$. Conversely a rotation by $\pi$ is571the latent shift $t\mapsto t+\pi$ on this ellipse, so572orientations differing by $\pi$ give the same law.573A symmetry of latent coordinates and a rotation of the574observed manifold therefore have distinct statistical effects575even within the same two-dimensional harmonic decoder.576\par577\section{Homogeneous feature bundles}578\label{geo:section}579 580Let \(G\) be a compact Lie group, \(H,K\subseteq G\) closed581subgroups, and \(P\) a topological \(G\)-space of parameters.582Write \(X=G/H\), \(Y=G/K\), and let \(\rho\) and \(\sigma\)583be finite-dimensional unitary representations of \(H\) and \(K\)584on \(V\) and \(W\). Use \(E=G\times_H V\) and \(F=G\times_K W\)585as input and output feature bundles. Our convention is586\([bh,v]=[b,\rho(h)v]\). An input field is therefore encoded by587\(f:G\to V\) with \(f(bh)=\rho(h)^{-1}f(b)\), and a group element \(g\)588acts by left translation \((g f)(b)=f(g^{-1}b)\). Haar probability589induces the invariant measures on \(X,Y\). The fixed-parameter590kernel model is the familiar homogeneous-space convolution591\cite{cohen-geiger-weiler,reprbook}; the parameter action below592allows the kernel itself to move with the symmetry.593 594\begin{proposition}[Kernels with moving parameters]595\label{geo:kernels}596Let \(L_p(y,x):E_x\to F_y\) be jointly continuous in the597parameter and the two base variables, with diagonal covariance598\[599 L_{gp}(gy,gx)=g L_p(y,x)g^{-1}.600\]601Such families correspond bijectively to continuous functions602\(B:P\times G\to\operatorname{Hom}(V,W)\) satisfying603\begin{equation}604 B(kp,kzh)=\sigma(k) B(p,z)\rho(h).605 \label{geo:bi}606\end{equation}607Here \(k\in K,h\in H\). The corresponding family of bounded608linear layers is609\begin{equation}610 (T_p f)(a)=\int_G B(a^{-1}p,a^{-1}b)f(b)\,db .611 \label{geo:conv}612\end{equation}613It satisfies \(T_{gp}(g f)=g(T_p f)\).614\end{proposition}615 616\begin{proof}617Evaluate \(L_p\) at the output basepoint \(1K\) and write618\(L_p(1K,zH)[z,v]=[1,B(p,z)v]\). Changing input representative619\(z\) to \(zh\) introduces \(\rho(h)\). Acting by \(k\in K\)620fixes the output basepoint and acts there by \(\sigma(k)\).621Together these give \eqref{geo:bi}. Conversely, define622\[