Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 343–442 of 1343. Source text is displayed literally.

343its two projections gives the displayed formula. Context maps344preserve the construction by naturality of strength.345A compatible system of monads on slices gives the same346construction over $B$ when their strengths and monad347operations commute with base change.348 349For instance, the finitely supported distribution monad on sets350gives finite channels by restricting the feature and output sets351to be finite. If $p,q$ are fixed, the composite has law352\[353 k_{p,q}(z\mid x)=\sum_y g_q(z\mid y) f_p(y\mid x).354\]355Dirac laws embed deterministic evaluations. Repeated sampling can356share an unknown parameter by pulling a product family back along357its diagonal. For a coin of bias $1/5$ or $4/5$ with equal prior358probability, shared-parameter sampling gives probability $17/50$359of two successes. Independently drawing the parameter for each360observation gives $1/4$. The single-observation law therefore361needs to be supplemented by a parameter sharing convention.362 363For smooth conditional models, the probability-valued evaluator364can be defined on a represented family of probability laws.365The next section uses finite outcome laws with positive366probabilities, where the open simplex is a manifold and the367Fisher tensor can be computed in ordinary coordinates.368\par369\section{Statistical and manifold models}370\label{stat:section}371 372A parametrized smooth map \(F:P\times X\to M\) is a manifold-valued373prediction, with a family of maps \(F_p:X\to M\). A deterministic network374may also retain intermediate feature bundles by factoring \(F\) through375smooth maps of total spaces. With an ordinary parameter manifold and no376prescribed equivalences of parameters, these are ordinary smooth families.377For stochastic predictions on a finite set \(Y\), a smooth readout is a map378\[379 q:P\times X\longrightarrow\Delta^\circ(Y),380 \qquad (p,x)\longmapsto(q_i(p,x))_{i\in Y},381\]382where \(\Delta^\circ(Y)\) is the simplex of strictly positive probability383vectors. Evaluation at \(p\) gives the conditional law of the outcome384given \(x\). Taking \(X=*\) recovers the usual smooth finite statistical385model. A deterministic prediction into a standard Borel space \(M\)386has the associated kernel \(x\mapsto\delta_{F(p,x)}\).387Dirac measures on a finite set with at least two elements lie on the388boundary of the simplex. Strict positivity permits differentiation of the finite log likelihood.389 390A stochastic decoder can be composed with a succeeding stochastic layer391by summing over the intermediate outcome. If \(K_{ji}(p,x)\) is the392conditional probability of output \(j\) given outcome \(i\), the new393readout is \(r_j=\sum_i K_{ji}q_i\). Associativity is the associativity394of finite summation, and deterministic relabelings are the special case395of Dirac kernels \cite{fritz-2020}. Smooth charts of \(P\) and \(X\)396present these models locally: if the predictions agree on the overlaps397after coordinate change, the coordinate functions \(q_i\) glue to a398unique smooth readout. For a manifold target \(M\), the same assertion399uses transition maps of \(M\) on overlapping target charts.400 401\subsection{The Fisher form and stochastic postprocessing}402 403At a point \(q\) of the positive simplex, write \(u,v\) for tangent404vectors with \(\sum_i u_i=\sum_i v_i=0\). The Fisher metric is405\[406 g_q(u,v)=\sum_{i\in Y}\frac{u_iv_i}{q_i}.407\]408At a fixed input \(x\), the score in direction \(v\in T_pP\) is409\(s_v(i)=d_p\log q_i(p,x)(v)\). Its mean is zero by differentiation of410\(\sum_i q_i=1\). Thus the covariance of the scores in directions411\(v,w\) is412\begin{equation}413 I_{p,x}(v,w)=414 \sum_{i\in Y}\frac{d_pq_i(v)\,d_pq_i(w)}{q_i(p,x)}.415 \label{stat:fisher}416\end{equation}417Equivalently, \(I_x=q_x^*g\), where \(q_x(p)=q(p,x)\).418For a parameter-independent Borel probability measure \(\mu\) on \(X\),419the joint experiment observes both the input and the label with law420\(d\mu(x)\,q_i(p,x)\). Its Fisher form is421\(I_p^\mu=\int_X I_{p,x}\,d\mu(x)\), assuming the scores are square422integrable and the experiment is differentiable under the integral.423These conditions hold for a smooth positive readout when \(X\) is424compact, locally in the parameter.425 426For a smooth reparametrization \(r:P'\to P\), the conditional Fisher427form of \(q\circ(r\times\operatorname{id}_X)\) is \(r^*I_x\)428by the chain rule. This holds for \(I^\mu\) as well. The identity gives429the change-of-coordinates rule \(I'=Dr^\top I\,Dr\) on charts.430On input charts the same joint form is computed by transporting431\(\mu\) with the coordinates. Thus the tensor descends with the432model and the chosen design measure. Its pointwise null space at433\(x\) is \(\ker d_pq_x\), because all \(q_i>0\). The joint form434vanishes on \(v\) precisely when \(d_pq_i(v)\) vanishes435\(\mu\)-almost everywhere for every \(i\).436 437\begin{proposition}[Finite Fisher contraction]438\label{stat:contraction}439A stochastic postprocessor \(K\), fixed with respect to \(p\), weakly440decreases both the conditional Fisher form and its design average.441\end{proposition}442