Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 681–780 of 2141. Source text is displayed literally.
681\begin{proof}682A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive683identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,684times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and685the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space686of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the687fundamental group of the base.688\end{proof}689 690So there are loops of networks, along which the function computed changes continuously and returns to691itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged692roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space693of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.694 695\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}696\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]697At a reducible parameter the fibre of the realization map has positive dimension:698\begin{enumerate}[label=(\roman*)]699\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;700\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;701\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates702$(a_j,b_j,c)$.703\end{enumerate}704\end{proposition}705 706Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of707parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union708of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At709these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is710therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower711dimension, onto which the reducible strata of $\Theta$ are collapsed.712 713\section{Homogeneous activations}714 715\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]716Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector717field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing718parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$719wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$720at every parameter.721\end{proposition}722 723\begin{proof}724$\xi_j$ is the generator of the one-parameter group of symmetries of \cref{prop:syms}(iii); differentiate725$F(g_\lambda\theta,x)=F(\theta,x)$ at $\lambda=1$. The $\xi_j$ for different units have disjoint supports in726the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and727the bound holds on the closure of that set by semicontinuity of rank.728\end{proof}729 730The script confirms both statements on small networks: for $\tanh$ the Jacobian has full rank $P$ at a731random parameter and loses rank at reducible ones; for $\relu$ its rank never exceeds $P-H$ and the fields732$\xi_j$ lie in its kernel.733 734For $\relu$ the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space:735the orbit of a unit with zero outgoing weight is not closed, since736$(\lambda w,\lambda b,0)\to(0,0,0)$ as $\lambda\to0$. The correct object is the quotient stack, in which737such points are remembered with their stabilizers. This is the same reason that vector bundles in738\DT{Ch.~5} formed a stack and not a space, and the loss is a function on it.739 740\section{Invariant functions}741 742\begin{proposition}\label{prop:invariant}\Scl743Let $\loss\colon\Theta\to\R$ be invariant under a group $G$ of symmetries. Then the critical set of744$\loss$ is a union of $G$-orbits. If $G$ contains a one-parameter group with generator $\xi$, then745$d\loss(\xi)=0$ identically and, at every critical point, $\xi$ lies in the kernel of the Hessian. If746moreover $\nabla\lambda$ vanishes at $\ev(\theta)$ for a loss $\loss=\lambda\circ\ev$, then the whole fibre747of $\ev$ through $\theta$ is critical.748\end{proposition}749 750\begin{proof}751Differentiating $\loss(g_s\theta)=\loss(\theta)$ in $s$ gives $d\loss_\theta(\xi)=0$ for all $\theta$;752differentiating that identity in a direction $v$ at a critical point gives753$\nabla^2\loss(\xi,v)+d\loss(D_v\xi)=\nabla^2\loss(\xi,v)=0$. The last statement is754$\nabla\loss=J^{\mathsf T}\nabla\lambda$.755\end{proof}756 757\begin{corollary}\label{cor:notmorse}\Sob758For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the759form $\lambda\circ\ev$ is a Morse function on any open set containing a critical point at which some hidden760unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least761the number of such units.762\end{corollary}763 764So Morse theory in the form of \DT{Ch.~8} never applies directly to the loss of a $\relu$ network. It can765apply to the function induced on a slice transverse to the orbits, or in Morse--Bott form, with critical766manifolds in place of critical points. For $\tanh$ networks there is no continuous symmetry and no such767obstruction on $\Theta^{\mathrm{irr}}$; the degeneracies there come from the reducible strata and from768overparametrization (\cref{ch:loss}).769 770\section*{Notes}771 772Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann773and by Albertini and Sontag, and for deep $\tanh$ networks by Fefferman. The relation between reducibility774and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of775Watanabe's singular learning theory \cite{Wat09}. The use of permutation symmetry to compare trained776networks is discussed in \cref{ch:loss}.777 778\section*{Exercises}779 780\begin{exercise}\label{exer:verifysym}