Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 403–502 of 1343. Source text is displayed literally.

403At a point \(q\) of the positive simplex, write \(u,v\) for tangent404vectors with \(\sum_i u_i=\sum_i v_i=0\). The Fisher metric is405\[406 g_q(u,v)=\sum_{i\in Y}\frac{u_iv_i}{q_i}.407\]408At a fixed input \(x\), the score in direction \(v\in T_pP\) is409\(s_v(i)=d_p\log q_i(p,x)(v)\). Its mean is zero by differentiation of410\(\sum_i q_i=1\). Thus the covariance of the scores in directions411\(v,w\) is412\begin{equation}413 I_{p,x}(v,w)=414 \sum_{i\in Y}\frac{d_pq_i(v)\,d_pq_i(w)}{q_i(p,x)}.415 \label{stat:fisher}416\end{equation}417Equivalently, \(I_x=q_x^*g\), where \(q_x(p)=q(p,x)\).418For a parameter-independent Borel probability measure \(\mu\) on \(X\),419the joint experiment observes both the input and the label with law420\(d\mu(x)\,q_i(p,x)\). Its Fisher form is421\(I_p^\mu=\int_X I_{p,x}\,d\mu(x)\), assuming the scores are square422integrable and the experiment is differentiable under the integral.423These conditions hold for a smooth positive readout when \(X\) is424compact, locally in the parameter.425 426For a smooth reparametrization \(r:P'\to P\), the conditional Fisher427form of \(q\circ(r\times\operatorname{id}_X)\) is \(r^*I_x\)428by the chain rule. This holds for \(I^\mu\) as well. The identity gives429the change-of-coordinates rule \(I'=Dr^\top I\,Dr\) on charts.430On input charts the same joint form is computed by transporting431\(\mu\) with the coordinates. Thus the tensor descends with the432model and the chosen design measure. Its pointwise null space at433\(x\) is \(\ker d_pq_x\), because all \(q_i>0\). The joint form434vanishes on \(v\) precisely when \(d_pq_i(v)\) vanishes435\(\mu\)-almost everywhere for every \(i\).436 437\begin{proposition}[Finite Fisher contraction]438\label{stat:contraction}439A stochastic postprocessor \(K\), fixed with respect to \(p\), weakly440decreases both the conditional Fisher form and its design average.441\end{proposition}442 443\begin{proof}444Omit output values \(j\) that are unreachable under \(K\).445For \(r_j=\sum_i K_{ji}q_i>0\), the score of \(r_j\) in direction446\(v\) is447\[448 d_p\log r_j(v)=\sum_i\frac{K_{ji}q_i}{r_j}\,s_v(i).449\]450The weights are the conditional probabilities of \(i\) given \(j\).451Jensen's inequality bounds the square by the weighted sum of squares.452Multiplying by \(r_j\) and summing over \(j\) yields453\(I^{r}_{p,x}(v,v)\le I^{q}_{p,x}(v,v)\). Integrating yields the claim454for \(I^\mu\). At a fixed \(x\), equality in direction \(v\) holds455exactly when \(s_v\) is constant on the intermediate outcomes that456contribute to each reachable output.457\end{proof}458 459The parameter-independence hypothesis keeps the postprocessor from460introducing its own score. Formula~\eqref{stat:fisher} is the finite461instance of the parametrized-measure Fisher tensor462\cite{ay-jost-le-schwachhoefer-2015}.463 464\begin{example}[Gaussian observations of a manifold family]465Let \(m:P\times X\to\mathbb R^k\) be a smooth readout and let the466conditional outcome have distribution \(N(m(p,x),\Sigma(x))\), where467\(\Sigma(x)\) is positive definite and independent of \(p\). For an468observed outcome \(z\), differentiation of the Gaussian log density469at fixed \(x\) gives the score470\[471 d_pm(v)^\top\Sigma(x)^{-1}\bigl(z-m(p,x)\bigr).472\]473The expectation of the product of two scores uses the covariance474\(\Sigma(x)\) and gives475\[476 I_{p,x}(v,w)=d_pm(v)^\top\Sigma(x)^{-1}d_pm(w).477\]478Thus an immersed mean manifold at fixed \(x\) has a Riemannian479metric by pullback of the Euclidean metric weighted by480\(\Sigma(x)^{-1}\). If the mean map has a kernel in its differential,481the pullback has the same null directions. An invertible affine482coordinate change of the output with linear part \(A\) replaces483\(\Sigma\) by \(A\Sigma A^\top\) and \(d_pm\) by \(A\,d_pm\), leaving the484form invariant. For a general smooth output coordinate change,485one transports the entire conditional law. The transformed density486contains a Jacobian factor independent of \(p\), so its score is the487transport of the original score even when the law exits the Gaussian488family.489\end{example}490 491\subsection{What the Fisher form forgets}492 493Consider a two-feature decoder on \(S^1\), with angle \(t\) understood494modulo \(2\pi\) and parameters \(p=(a,b,c)\in\mathbb R^3\). Its logit495is \(L_p(t)=a\cos t+bc\sin t\) and its Bernoulli success probability496is \(s_p(t)=(1+e^{-L_p(t)})^{-1}\). For a tangent vector497\(v=(v_a,v_b,v_c)\), the score at label \(y\in\{0,1\}\) is498\((y-s_p(t))\,dL_p(t)(v)\). With any full-support design probability499measure \(\mu\) on the circle, the joint Fisher form is500\begin{align*}501 I^\mu_p(v,v)502  &=\int_{S^1}s_p(t)(1-s_p(t))\,(\delta_v(t))^2\,d\mu(t),\\