Indexed text attachment
representation-theory-of-parametrized-neural-models-on-spaces.tex
Source for Representation Theory of Parametrized Neural Models on Spaces
View paper · Download original file
Lines 781–880 of 1343. Source text is displayed literally.
781A moving parameter can carry the difference between the input782and output representation types. This section classifies that783effect for layers linear in the parameter and in the features.784We work with complex Hilbert spaces. The representation $W$785of the compact group $G$ is finite dimensional and unitary,786and $H_X,H_Y$ have strongly continuous unitary actions.787A linear filter family is a complex-linear map788$A:W\to\mathcal B(H_X,H_Y)$, with the equivariance condition789\begin{equation}790 A(gw)=\rho_Y(g) A(w)\rho_X(g)^{-1}.791 \label{eq:filter-covariance}792\end{equation}793Boundedness of the map into the operator norm is automatic794for finite-dimensional $W$.795 796Let $\widehat G$ index the irreducible unitary797representations $V_\pi$. Write $M_\pi^X$ for the Hilbert798multiplicity space of $V_\pi$ in $H_X$, and likewise $M_\pi^Y$.799Compact harmonic analysis gives the orthogonal decompositions800\[801 H_X\simeq\widehat{\bigoplus}_{\pi\in\widehat G}802 V_\pi\otimes M_\pi^X,\qquad803 H_Y\simeq\widehat{\bigoplus}_{\tau\in\widehat G}804 V_\tau\otimes M_\tau^Y .805\]806For intertwiners $S,T:V\to H$ with common irreducible source $V$,807we use the scalar inner product defined by808$S^*T=\langle S,T\rangle\id_V$.809 810\begin{theorem}[Classification of linear filter families]811\label{thm:harmonic}812For each $\tau\in\widehat G$, set813\[814 \begin{aligned}815 C_{\tau,\pi}816 &=\Hom_G(V_\tau,W\otimes V_\pi),\\817 N_\tau818 &=\widehat{\bigoplus}_{\pi\in\widehat G}819 C_{\tau,\pi}\otimes M_\pi^X .820 \end{aligned}821\]822The families satisfying \eqref{eq:filter-covariance} are in823bijection with families of bounded maps824$B_\tau:N_\tau\to M_\tau^Y$ satisfying825$\sup_\tau\|B_\tau\|<\infty$. The norm of the linearized operator826$W\otimes H_X\to H_Y$ equals that supremum. In particular the827transition from $\pi$ to $\tau$ can contribute only if828$V_\tau$ is a summand of $W\otimes V_\pi$.829\end{theorem}830 831\begin{proof}832Send $w\otimes x$ to $A(w)x$. For an orthonormal basis833$(w_j)_{j=1}^d$ of $W$, Cauchy--Schwarz gives834\[835 \left\|\sum_j A(w_j)x_j\right\|836 \le837 \left(\sum_j\|A(w_j)\|^2\right)^{1/2}838 \left(\sum_j\|x_j\|^2\right)^{1/2}.839\]840Thus linearization extends to the Hilbert tensor product.841Conversely, a bounded operator $L:W\otimes H_X\to H_Y$842gives $A(w)x=L(w\otimes x)$. Applying \eqref{eq:filter-covariance}843to a decomposable tensor is precisely the intertwining844condition for $L$ with the diagonal action on its source.845 846The compact isotypic decomposition follows from complete847reducibility of continuous unitary representations848\cite[Chs.~2--4]{reprbook}. Tensoring with $W$ and849decomposing each finite-dimensional $W\otimes V_\pi$ gives850\[851 W\otimes H_X\simeq852 \widehat{\bigoplus}_{\tau\in\widehat G} V_\tau\otimes N_\tau .853\]854Evaluation $V_\tau\otimes C_{\tau,\pi}\to W\otimes855V_\pi$ identifies the image with the $\tau$-isotypic part:856orthogonality of the summands is Schur's lemma and the857chosen Hilbert structure on $C_{\tau,\pi}$. Schur's lemma applied858using multiplicity coordinates then forces $L$ to have block859$\id_{V_\tau}\otimes B_\tau$ and zero blocks between distinct860types. The norm of each diagonal block is $\|B_\tau\|$.861Conversely the Hilbert sum of such blocks is bounded862with norm their supremum, and commutes with $G$ on each block.863\end{proof}864 865With $W$ trivial of dimension one this gives the usual866classification of a bounded intertwiner. A trivial parameter867representation of larger dimension adds multiplicity but868preserves the same restriction on which types are allowed.869A nontrivial $W$ changes that restriction. In a network with870two successive linear parameter families, the evaluation is871bilinear in their weights. It linearizes in $W_2\otimes W_1$,872with its diagonal action. A path of intermediate irreducible873types is allowed when each step obeys the tensor multiplicity874condition. Its actual linear map is the composition dictated875by the original layer maps. Not every block of a free876$W_2\otimes W_1$ model factors through a prescribed architecture.877 878\begin{example}[A learned Fourier shift]879Let $G=S^1$ act on $L^2(S^1)$ by translation880$(\rho(\theta)x)(t)=x(t-\theta)$. Let $W=\C$ transform by the