Indexed text attachment
representation-theory-of-parametrized-neural-models-on-spaces.tex
Source for Representation Theory of Parametrized Neural Models on Spaces
View paper · Download original file
Lines 443–542 of 1343. Source text is displayed literally.
443\begin{proof}444Omit output values \(j\) that are unreachable under \(K\).445For \(r_j=\sum_i K_{ji}q_i>0\), the score of \(r_j\) in direction446\(v\) is447\[448 d_p\log r_j(v)=\sum_i\frac{K_{ji}q_i}{r_j}\,s_v(i).449\]450The weights are the conditional probabilities of \(i\) given \(j\).451Jensen's inequality bounds the square by the weighted sum of squares.452Multiplying by \(r_j\) and summing over \(j\) yields453\(I^{r}_{p,x}(v,v)\le I^{q}_{p,x}(v,v)\). Integrating yields the claim454for \(I^\mu\). At a fixed \(x\), equality in direction \(v\) holds455exactly when \(s_v\) is constant on the intermediate outcomes that456contribute to each reachable output.457\end{proof}458 459The parameter-independence hypothesis keeps the postprocessor from460introducing its own score. Formula~\eqref{stat:fisher} is the finite461instance of the parametrized-measure Fisher tensor462\cite{ay-jost-le-schwachhoefer-2015}.463 464\begin{example}[Gaussian observations of a manifold family]465Let \(m:P\times X\to\mathbb R^k\) be a smooth readout and let the466conditional outcome have distribution \(N(m(p,x),\Sigma(x))\), where467\(\Sigma(x)\) is positive definite and independent of \(p\). For an468observed outcome \(z\), differentiation of the Gaussian log density469at fixed \(x\) gives the score470\[471 d_pm(v)^\top\Sigma(x)^{-1}\bigl(z-m(p,x)\bigr).472\]473The expectation of the product of two scores uses the covariance474\(\Sigma(x)\) and gives475\[476 I_{p,x}(v,w)=d_pm(v)^\top\Sigma(x)^{-1}d_pm(w).477\]478Thus an immersed mean manifold at fixed \(x\) has a Riemannian479metric by pullback of the Euclidean metric weighted by480\(\Sigma(x)^{-1}\). If the mean map has a kernel in its differential,481the pullback has the same null directions. An invertible affine482coordinate change of the output with linear part \(A\) replaces483\(\Sigma\) by \(A\Sigma A^\top\) and \(d_pm\) by \(A\,d_pm\), leaving the484form invariant. For a general smooth output coordinate change,485one transports the entire conditional law. The transformed density486contains a Jacobian factor independent of \(p\), so its score is the487transport of the original score even when the law exits the Gaussian488family.489\end{example}490 491\subsection{What the Fisher form forgets}492 493Consider a two-feature decoder on \(S^1\), with angle \(t\) understood494modulo \(2\pi\) and parameters \(p=(a,b,c)\in\mathbb R^3\). Its logit495is \(L_p(t)=a\cos t+bc\sin t\) and its Bernoulli success probability496is \(s_p(t)=(1+e^{-L_p(t)})^{-1}\). For a tangent vector497\(v=(v_a,v_b,v_c)\), the score at label \(y\in\{0,1\}\) is498\((y-s_p(t))\,dL_p(t)(v)\). With any full-support design probability499measure \(\mu\) on the circle, the joint Fisher form is500\begin{align*}501 I^\mu_p(v,v)502 &=\int_{S^1}s_p(t)(1-s_p(t))\,(\delta_v(t))^2\,d\mu(t),\\503 \delta_v(t)&=v_a\cos t+(c v_b+b v_c)\sin t.504\end{align*}505Positivity of the Bernoulli variance and continuity in \(t\) imply506that the integral vanishes exactly when \(\delta_v\) vanishes507identically. The independent harmonics give the two equations508\(v_a=0\) and \(c v_b+b v_c=0\). The rank is two away from \(b=c=0\)509and one along that locus. In particular the Fisher tensor is the510pullback of a positive definite form on the two-dimensional logistic511family with coordinates \((a,d)\), along the map \((a,b,c)\mapsto(a,bc)\). The strict monotonicity of512the logistic function and the independent harmonics also show that513this map labels exactly the distinct output laws. This is an elementary instance of singular parametrization514\cite{watanabe-2009}.515 516The multiplicative group \(\mathbb R^\times\) acts by517\(\lambda\cdot(a,b,c)=(a,\lambda b,\lambda^{-1}c)\), preserving the518readout. The action groupoid has stabilizer \(\mathbb R^\times\) at519\(b=c=0\) and trivial stabilizer elsewhere. Its three distinct520orbits over a given prediction with \(d=0\) correspond to the521origin and the two punctured coordinate axes in the \((b,c)\)-plane.522Thus the quotient groupoid of parameter symmetries contains523information beyond the set of distinct output laws. Even the524infinitesimal null space can include directions that change the525output at higher order: along \((a,\varepsilon,\varepsilon)\) through526\((a,0,0)\), \(d=\varepsilon^2\) changes the prediction to second order.527Consequently equality of output laws, Fisher-null directions, and528parameter equivalence answer three different questions.529 530\subsection{A manifold decoder as a statistical model}531 532Let $Z$ be a latent manifold with fixed probability $\nu$ and533$d:P\times Z\to M$ a smooth decoder. In the absence of observation534noise the law at $p$ is $(d_p)_*\nu$. A noise channel $N:M\to Y$535then gives the observed law536\[537 \mathbb P_p(A)=\int_Z N(d_p(z),A)\,d\nu(z)538\]539for measurable $A\subseteq Y$. This is ordinary composition of540Markov kernels on these measurable spaces. The manifold541evaluation $d$ can be represented by the smooth slice542construction of Section~\ref{sec:relative}. The pushforward