Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 901–1000 of 1343. Source text is displayed literally.

901$z\mapsto z\,\psi(|z|^2)$ for continuous $\psi$ is an902equivariant activation. A scalar real activation applied903separately to real and imaginary parts need not commute with904rotations. For instance, coordinatewise rectification fails905already at rotation by $\pi$. Nonlinearities can mix Fourier906modes even though the \emph{whole} model respects907\eqref{eq:equiv-eval}. The theorem describes the linear layers908and their allowed type transitions, rather than a Fourier909diagonalization of the nonlinear network.910\par911\section{A sign-symmetric decoder on the circle}912\label{gauge:circle}913 914A tied-weight scalar neural decoder gives a direct comparison of915observable predictions, parameter coordinates, and descent data. Let916\(G=\{1,-1\}\) act on a real weight by \(w\mapsto -w\). For input917\(x\in\mathbb R\), define the hidden feature and Bernoulli decoder by918\begin{align}919 h_w(x)&=\tanh(wx),&920 \ell_w(x)&=w h_w(x), \label{gauge:logit}\\921 K_w(1\mid x)&=\frac{1}{1+\exp(-\ell_w(x))},\notag\\922 K_w(0\mid x)&=1-K_w(1\mid x). \notag923\end{align}924Under a sign change the hidden feature changes sign and the tied925readout changes sign, so the logit and the two probabilities are926invariant. The statistical model is a Markov kernel with a finite927output space. Its local parameters encode a choice of frame for the928one-dimensional hidden feature.929 930\begin{proposition}[Observational equivalence of scalar weights]931\label{gauge:pointwise}932Two weights \(u,w\) give the same kernel for every real input \(x\)933if and only if \(u=\pm w\). The gauge stabilizer of a weight is \(G\) at \(w=0\)934and trivial elsewhere.935\end{proposition}936 937\begin{proof}938Equality of probabilities gives equality of logits by invertibility939of the logistic function. Differentiation in the input gives940\(\ell'_w(0)=w^2\), so equality implies \(u^2=w^2\).941The sign invariance of \eqref{gauge:logit} gives the converse.942A sign fixes \(w\) precisely when its product with \(w\) is \(w\).943\end{proof}944 945\subsection{Descent of weights and observables}946 947For a smooth base \(B\), a family in the quotient stack948\([\mathbb R/G]\) consists of a principal \(G\)-bundle \(P\to B\)949and a smooth equivariant function \(w:P\to\mathbb R\). Morphisms950over \(B\) are principal-bundle isomorphisms compatible with \(w\).951This is the torsor description of a quotient stack, using open952covers and local smooth trivializations in this setting953\cite[cf.\ the algebraic version in Tag 04UV]{stacks-quotients}.954Equivalently, choose local frames over an open cover \(U_i\) and955write the coordinates as \(w_i:U_i\to\mathbb R\). The locally956constant signs \(\varepsilon_{ij}\) satisfy957\[958 \varepsilon_{ij}\varepsilon_{jk}=\varepsilon_{ik},959 \qquad w_i=\varepsilon_{ij}w_j .960\]961Changing frames acts by a family of local signs962\(\alpha_i:U_i\to G\), replacing these data by963\(\alpha_i\varepsilon_{ij}\alpha_j^{-1}\) and \(\alpha_iw_i\).964The same signs glue the hidden features \(h_{w_i}\), and the965signs cancel in \(w_i h_{w_i}\). Hence the observable kernel is a966single smooth family \(K:B\times\mathbb R\to967\operatorname{Prob}(\{0,1\})\). The invariant function968\(r(b)=w(p)^2\), for \(p\in P_b\), can be recovered from \(K\)969by differentiating the logit in \(x\) at \(0\).970Both constructions commute with pullback along smooth base maps.971They give maps from the groupoid of families to the set of smooth972nonnegative functions and then to the set of observational laws.973For a fixed base, the second map is injective on the functions974obtained from families by Proposition~\ref{gauge:pointwise}.975The first map can identify nonisomorphic families, as the zero976example below shows.977 978\subsection{Monodromy in the zero fiber}979 980Over \(S^1=\mathbb R/(2\pi\mathbb Z)\), take the connected double981cover \(P_-\) with total space \(\mathbb R/(4\pi\mathbb Z)\) and982deck transformation \(\theta\mapsto\theta+2\pi\). The associated983sign line is the M\"obius bundle984\cite[\S 1.1]{hatcher-vbkt}. Transport once around the circle985switches sheets. For comparison, the trivial double cover986\(P_+=S^1\times G\) preserves each sheet. The connectedness of987\(P_-\) distinguishes it from \(P_+\). Both admit the zero988equivariant weight, and both resulting decoders output probability989\(1/2\) at every base point and input.990 991\begin{proposition}[Families over the zero observable]992\label{gauge:zero}993The groupoid of smooth families on \(S^1\) with constant994observable \(K(1\mid x)=1/2\) is equivalent to995\(BG\amalg BG\).996\end{proposition}997 998\begin{proof}999The recovery formula for \(r\) forces the equivariant function1000\(w\) to be zero, so the whole groupoid consists of \(G\)-torsors