Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 641–740 of 2141. Source text is displayed literally.
641\begin{enumerate}[label=(\roman*)]642\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.643\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.644\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and645$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by646$\lambda^{-1}$.647\end{enumerate}648For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.649\end{proposition}650 651\section{One hidden layer}652 653Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with654$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.655 656\begin{definition}\label{def:irr}657The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and658$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.659\end{definition}660 661\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]662Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of663hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and664only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;665so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.666\end{theorem}667 668For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of669Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a670finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on671it.672 673\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob674The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is675injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then676\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }677Z'\bigr\}\times\R .\]678If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.679\end{proposition}680 681\begin{proof}682A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive683identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,684times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and685the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space686of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the687fundamental group of the base.688\end{proof}689 690So there are loops of networks, along which the function computed changes continuously and returns to691itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged692roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space693of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.694 695\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}696\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]697At a reducible parameter the fibre of the realization map has positive dimension:698\begin{enumerate}[label=(\roman*)]699\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;700\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;701\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates702$(a_j,b_j,c)$.703\end{enumerate}704\end{proposition}705 706Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of707parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union708of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At709these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is710therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower711dimension, onto which the reducible strata of $\Theta$ are collapsed.712 713\section{Homogeneous activations}714 715\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]716Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector717field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing718parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$719wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$720at every parameter.721\end{proposition}722 723\begin{proof}724$\xi_j$ is the generator of the one-parameter group of symmetries of \cref{prop:syms}(iii); differentiate725$F(g_\lambda\theta,x)=F(\theta,x)$ at $\lambda=1$. The $\xi_j$ for different units have disjoint supports in726the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and727the bound holds on the closure of that set by semicontinuity of rank.728\end{proof}729 730The script confirms both statements on small networks: for $\tanh$ the Jacobian has full rank $P$ at a731random parameter and loses rank at reducible ones; for $\relu$ its rank never exceeds $P-H$ and the fields732$\xi_j$ lie in its kernel.733 734For $\relu$ the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space:735the orbit of a unit with zero outgoing weight is not closed, since736$(\lambda w,\lambda b,0)\to(0,0,0)$ as $\lambda\to0$. The correct object is the quotient stack, in which737such points are remembered with their stabilizers. This is the same reason that vector bundles in738\DT{Ch.~5} formed a stack and not a space, and the loss is a function on it.739 740\section{Invariant functions}