Independent scholarship · Open accessThe monthly digest

Indexed text attachment

representation-theory-of-parametrized-neural-models-on-spaces.tex

Source for Representation Theory of Parametrized Neural Models on Spaces

View paper · Download original file

Lines 803–902 of 1343. Source text is displayed literally.

803 H_Y\simeq\widehat{\bigoplus}_{\tau\in\widehat G}804         V_\tau\otimes M_\tau^Y .805\]806For intertwiners $S,T:V\to H$ with common irreducible source $V$,807we use the scalar inner product defined by808$S^*T=\langle S,T\rangle\id_V$.809 810\begin{theorem}[Classification of linear filter families]811\label{thm:harmonic}812For each $\tau\in\widehat G$, set813\[814 \begin{aligned}815 C_{\tau,\pi}816    &=\Hom_G(V_\tau,W\otimes V_\pi),\\817 N_\tau818    &=\widehat{\bigoplus}_{\pi\in\widehat G}819         C_{\tau,\pi}\otimes M_\pi^X .820 \end{aligned}821\]822The families satisfying \eqref{eq:filter-covariance} are in823bijection with families of bounded maps824$B_\tau:N_\tau\to M_\tau^Y$ satisfying825$\sup_\tau\|B_\tau\|<\infty$. The norm of the linearized operator826$W\otimes H_X\to H_Y$ equals that supremum. In particular the827transition from $\pi$ to $\tau$ can contribute only if828$V_\tau$ is a summand of $W\otimes V_\pi$.829\end{theorem}830 831\begin{proof}832Send $w\otimes x$ to $A(w)x$. For an orthonormal basis833$(w_j)_{j=1}^d$ of $W$, Cauchy--Schwarz gives834\[835 \left\|\sum_j A(w_j)x_j\right\|836 \le837 \left(\sum_j\|A(w_j)\|^2\right)^{1/2}838 \left(\sum_j\|x_j\|^2\right)^{1/2}.839\]840Thus linearization extends to the Hilbert tensor product.841Conversely, a bounded operator $L:W\otimes H_X\to H_Y$842gives $A(w)x=L(w\otimes x)$. Applying \eqref{eq:filter-covariance}843to a decomposable tensor is precisely the intertwining844condition for $L$ with the diagonal action on its source.845 846The compact isotypic decomposition follows from complete847reducibility of continuous unitary representations848\cite[Chs.~2--4]{reprbook}. Tensoring with $W$ and849decomposing each finite-dimensional $W\otimes V_\pi$ gives850\[851 W\otimes H_X\simeq852   \widehat{\bigoplus}_{\tau\in\widehat G} V_\tau\otimes N_\tau .853\]854Evaluation $V_\tau\otimes C_{\tau,\pi}\to W\otimes855V_\pi$ identifies the image with the $\tau$-isotypic part:856orthogonality of the summands is Schur's lemma and the857chosen Hilbert structure on $C_{\tau,\pi}$. Schur's lemma applied858using multiplicity coordinates then forces $L$ to have block859$\id_{V_\tau}\otimes B_\tau$ and zero blocks between distinct860types. The norm of each diagonal block is $\|B_\tau\|$.861Conversely the Hilbert sum of such blocks is bounded862with norm their supremum, and commutes with $G$ on each block.863\end{proof}864 865With $W$ trivial of dimension one this gives the usual866classification of a bounded intertwiner. A trivial parameter867representation of larger dimension adds multiplicity but868preserves the same restriction on which types are allowed.869A nontrivial $W$ changes that restriction. In a network with870two successive linear parameter families, the evaluation is871bilinear in their weights. It linearizes in $W_2\otimes W_1$,872with its diagonal action. A path of intermediate irreducible873types is allowed when each step obeys the tensor multiplicity874condition. Its actual linear map is the composition dictated875by the original layer maps. Not every block of a free876$W_2\otimes W_1$ model factors through a prescribed architecture.877 878\begin{example}[A learned Fourier shift]879Let $G=S^1$ act on $L^2(S^1)$ by translation880$(\rho(\theta)x)(t)=x(t-\theta)$. Let $W=\C$ transform by the881character $e^{im\theta}$ for a nonzero integer $m$. Then882\[883 (A(w)x)(t)=w\,e^{-imt}x(t)884\]885has norm $|w|$ and satisfies \eqref{eq:filter-covariance}.886For the Fourier vector $e_n(t)=e^{int}$, it maps $e_n$887to $w e_{n-m}$. Indeed, the representation character on888$w\otimes e_n$ is $e^{-i(n-m)\theta}$, which is the character889on $e_{n-m}$. This is the only possible output frequency890for input $e_n$ in any such family with scalar weight $m$.891More generally $A(w)e_n=w a_n e_{n-m}$ classifies these892families by bounded sequences $(a_n)_{n\in\Z}$.893If $w$ is frozen at a nonzero value without transforming,894the displayed operator fails to commute with rotations that895act nontrivially on that weight.896\end{example}897 898Pointwise nonlinearities must preserve equivariance at the899feature representations on which they operate. For a complex900line of a circle weight, the radial map901$z\mapsto z\,\psi(|z|^2)$ for continuous $\psi$ is an902equivariant activation. A scalar real activation applied