Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 503–602 of 1343. Source text is displayed literally.

503 \delta_v(t)&=v_a\cos t+(c v_b+b v_c)\sin t.504\end{align*}505Positivity of the Bernoulli variance and continuity in \(t\) imply506that the integral vanishes exactly when \(\delta_v\) vanishes507identically. The independent harmonics give the two equations508\(v_a=0\) and \(c v_b+b v_c=0\). The rank is two away from \(b=c=0\)509and one along that locus. In particular the Fisher tensor is the510pullback of a positive definite form on the two-dimensional logistic511family with coordinates \((a,d)\), along the map \((a,b,c)\mapsto(a,bc)\). The strict monotonicity of512the logistic function and the independent harmonics also show that513this map labels exactly the distinct output laws. This is an elementary instance of singular parametrization514\cite{watanabe-2009}.515 516The multiplicative group \(\mathbb R^\times\) acts by517\(\lambda\cdot(a,b,c)=(a,\lambda b,\lambda^{-1}c)\), preserving the518readout. The action groupoid has stabilizer \(\mathbb R^\times\) at519\(b=c=0\) and trivial stabilizer elsewhere. Its three distinct520orbits over a given prediction with \(d=0\) correspond to the521origin and the two punctured coordinate axes in the \((b,c)\)-plane.522Thus the quotient groupoid of parameter symmetries contains523information beyond the set of distinct output laws. Even the524infinitesimal null space can include directions that change the525output at higher order: along \((a,\varepsilon,\varepsilon)\) through526\((a,0,0)\), \(d=\varepsilon^2\) changes the prediction to second order.527Consequently equality of output laws, Fisher-null directions, and528parameter equivalence answer three different questions.529 530\subsection{A manifold decoder as a statistical model}531 532Let $Z$ be a latent manifold with fixed probability $\nu$ and533$d:P\times Z\to M$ a smooth decoder. In the absence of observation534noise the law at $p$ is $(d_p)_*\nu$. A noise channel $N:M\to Y$535then gives the observed law536\[537 \mathbb P_p(A)=\int_Z N(d_p(z),A)\,d\nu(z)538\]539for measurable $A\subseteq Y$. This is ordinary composition of540Markov kernels on these measurable spaces. The manifold541evaluation $d$ can be represented by the smooth slice542construction of Section~\ref{sec:relative}. The pushforward543additionally uses the chosen law $\nu$ and noise $N$.544If $a:Z\to Z$ preserves $\nu$, then replacing $d_p$ by545$d_p\circ a$ leaves $\mathbb P_p$ unchanged, by the546change-of-variables identity for pushforwards. A law that547observes $z$ and the output together can distinguish the two.548Thus the observation protocol determines which parameter549symmetries are invisible.550 551For an explicit case take $Z=S^1$ with Haar probability and552the ellipse $e(t)=(a\cos t,b\sin t)$ in $\mathbb R^2$ for553$a,b>0$ with $a\ne b$. A hidden phase $\theta$ in554$t\mapsto e(t+\theta)$ is entirely unidentifiable after555integrating out $t$. Adding centered isotropic Gaussian556noise with covariance $\sigma^2 I$ preserves this557invariance. The integrated Gaussian mixture density is558independent of $\theta$, so its Fisher information in559the phase direction is zero. In contrast, an output560orientation in $t\mapsto R_\theta e(t)$ rotates the561signal covariance:562\[563 \operatorname{Cov}_\theta(Y)=564 \tfrac12 R_\theta565 \begin{pmatrix}a^2&0\\0&b^2\end{pmatrix}R_\theta^\top566 +\sigma^2 I .567\]568Because $a^2\ne b^2$, equality of these covariances forces569coincidence of the principal axes and hence equality of570$\theta$ modulo $\pi$. Conversely a rotation by $\pi$ is571the latent shift $t\mapsto t+\pi$ on this ellipse, so572orientations differing by $\pi$ give the same law.573A symmetry of latent coordinates and a rotation of the574observed manifold therefore have distinct statistical effects575even within the same two-dimensional harmonic decoder.576\par577\section{Homogeneous feature bundles}578\label{geo:section}579 580Let \(G\) be a compact Lie group, \(H,K\subseteq G\) closed581subgroups, and \(P\) a topological \(G\)-space of parameters.582Write \(X=G/H\), \(Y=G/K\), and let \(\rho\) and \(\sigma\)583be finite-dimensional unitary representations of \(H\) and \(K\)584on \(V\) and \(W\). Use \(E=G\times_H V\) and \(F=G\times_K W\)585as input and output feature bundles. Our convention is586\([bh,v]=[b,\rho(h)v]\). An input field is therefore encoded by587\(f:G\to V\) with \(f(bh)=\rho(h)^{-1}f(b)\), and a group element \(g\)588acts by left translation \((g f)(b)=f(g^{-1}b)\). Haar probability589induces the invariant measures on \(X,Y\). The fixed-parameter590kernel model is the familiar homogeneous-space convolution591\cite{cohen-geiger-weiler,reprbook}; the parameter action below592allows the kernel itself to move with the symmetry.593 594\begin{proposition}[Kernels with moving parameters]595\label{geo:kernels}596Let \(L_p(y,x):E_x\to F_y\) be jointly continuous in the597parameter and the two base variables, with diagonal covariance598\[599 L_{gp}(gy,gx)=g L_p(y,x)g^{-1}.600\]601Such families correspond bijectively to continuous functions602\(B:P\times G\to\operatorname{Hom}(V,W)\) satisfying