Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 861–960 of 1343. Source text is displayed literally.

861Conversely the Hilbert sum of such blocks is bounded862with norm their supremum, and commutes with $G$ on each block.863\end{proof}864 865With $W$ trivial of dimension one this gives the usual866classification of a bounded intertwiner. A trivial parameter867representation of larger dimension adds multiplicity but868preserves the same restriction on which types are allowed.869A nontrivial $W$ changes that restriction. In a network with870two successive linear parameter families, the evaluation is871bilinear in their weights. It linearizes in $W_2\otimes W_1$,872with its diagonal action. A path of intermediate irreducible873types is allowed when each step obeys the tensor multiplicity874condition. Its actual linear map is the composition dictated875by the original layer maps. Not every block of a free876$W_2\otimes W_1$ model factors through a prescribed architecture.877 878\begin{example}[A learned Fourier shift]879Let $G=S^1$ act on $L^2(S^1)$ by translation880$(\rho(\theta)x)(t)=x(t-\theta)$. Let $W=\C$ transform by the881character $e^{im\theta}$ for a nonzero integer $m$. Then882\[883 (A(w)x)(t)=w\,e^{-imt}x(t)884\]885has norm $|w|$ and satisfies \eqref{eq:filter-covariance}.886For the Fourier vector $e_n(t)=e^{int}$, it maps $e_n$887to $w e_{n-m}$. Indeed, the representation character on888$w\otimes e_n$ is $e^{-i(n-m)\theta}$, which is the character889on $e_{n-m}$. This is the only possible output frequency890for input $e_n$ in any such family with scalar weight $m$.891More generally $A(w)e_n=w a_n e_{n-m}$ classifies these892families by bounded sequences $(a_n)_{n\in\Z}$.893If $w$ is frozen at a nonzero value without transforming,894the displayed operator fails to commute with rotations that895act nontrivially on that weight.896\end{example}897 898Pointwise nonlinearities must preserve equivariance at the899feature representations on which they operate. For a complex900line of a circle weight, the radial map901$z\mapsto z\,\psi(|z|^2)$ for continuous $\psi$ is an902equivariant activation. A scalar real activation applied903separately to real and imaginary parts need not commute with904rotations. For instance, coordinatewise rectification fails905already at rotation by $\pi$. Nonlinearities can mix Fourier906modes even though the \emph{whole} model respects907\eqref{eq:equiv-eval}. The theorem describes the linear layers908and their allowed type transitions, rather than a Fourier909diagonalization of the nonlinear network.910\par911\section{A sign-symmetric decoder on the circle}912\label{gauge:circle}913 914A tied-weight scalar neural decoder gives a direct comparison of915observable predictions, parameter coordinates, and descent data. Let916\(G=\{1,-1\}\) act on a real weight by \(w\mapsto -w\). For input917\(x\in\mathbb R\), define the hidden feature and Bernoulli decoder by918\begin{align}919 h_w(x)&=\tanh(wx),&920 \ell_w(x)&=w h_w(x), \label{gauge:logit}\\921 K_w(1\mid x)&=\frac{1}{1+\exp(-\ell_w(x))},\notag\\922 K_w(0\mid x)&=1-K_w(1\mid x). \notag923\end{align}924Under a sign change the hidden feature changes sign and the tied925readout changes sign, so the logit and the two probabilities are926invariant. The statistical model is a Markov kernel with a finite927output space. Its local parameters encode a choice of frame for the928one-dimensional hidden feature.929 930\begin{proposition}[Observational equivalence of scalar weights]931\label{gauge:pointwise}932Two weights \(u,w\) give the same kernel for every real input \(x\)933if and only if \(u=\pm w\). The gauge stabilizer of a weight is \(G\) at \(w=0\)934and trivial elsewhere.935\end{proposition}936 937\begin{proof}938Equality of probabilities gives equality of logits by invertibility939of the logistic function. Differentiation in the input gives940\(\ell'_w(0)=w^2\), so equality implies \(u^2=w^2\).941The sign invariance of \eqref{gauge:logit} gives the converse.942A sign fixes \(w\) precisely when its product with \(w\) is \(w\).943\end{proof}944 945\subsection{Descent of weights and observables}946 947For a smooth base \(B\), a family in the quotient stack948\([\mathbb R/G]\) consists of a principal \(G\)-bundle \(P\to B\)949and a smooth equivariant function \(w:P\to\mathbb R\). Morphisms950over \(B\) are principal-bundle isomorphisms compatible with \(w\).951This is the torsor description of a quotient stack, using open952covers and local smooth trivializations in this setting953\cite[cf.\ the algebraic version in Tag 04UV]{stacks-quotients}.954Equivalently, choose local frames over an open cover \(U_i\) and955write the coordinates as \(w_i:U_i\to\mathbb R\). The locally956constant signs \(\varepsilon_{ij}\) satisfy957\[958 \varepsilon_{ij}\varepsilon_{jk}=\varepsilon_{ik},959 \qquad w_i=\varepsilon_{ij}w_j .960\]