Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 701–800 of 2141. Source text is displayed literally.
701\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates702$(a_j,b_j,c)$.703\end{enumerate}704\end{proposition}705 706Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of707parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union708of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At709these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is710therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower711dimension, onto which the reducible strata of $\Theta$ are collapsed.712 713\section{Homogeneous activations}714 715\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]716Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector717field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing718parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$719wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$720at every parameter.721\end{proposition}722 723\begin{proof}724$\xi_j$ is the generator of the one-parameter group of symmetries of \cref{prop:syms}(iii); differentiate725$F(g_\lambda\theta,x)=F(\theta,x)$ at $\lambda=1$. The $\xi_j$ for different units have disjoint supports in726the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and727the bound holds on the closure of that set by semicontinuity of rank.728\end{proof}729 730The script confirms both statements on small networks: for $\tanh$ the Jacobian has full rank $P$ at a731random parameter and loses rank at reducible ones; for $\relu$ its rank never exceeds $P-H$ and the fields732$\xi_j$ lie in its kernel.733 734For $\relu$ the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space:735the orbit of a unit with zero outgoing weight is not closed, since736$(\lambda w,\lambda b,0)\to(0,0,0)$ as $\lambda\to0$. The correct object is the quotient stack, in which737such points are remembered with their stabilizers. This is the same reason that vector bundles in738\DT{Ch.~5} formed a stack and not a space, and the loss is a function on it.739 740\section{Invariant functions}741 742\begin{proposition}\label{prop:invariant}\Scl743Let $\loss\colon\Theta\to\R$ be invariant under a group $G$ of symmetries. Then the critical set of744$\loss$ is a union of $G$-orbits. If $G$ contains a one-parameter group with generator $\xi$, then745$d\loss(\xi)=0$ identically and, at every critical point, $\xi$ lies in the kernel of the Hessian. If746moreover $\nabla\lambda$ vanishes at $\ev(\theta)$ for a loss $\loss=\lambda\circ\ev$, then the whole fibre747of $\ev$ through $\theta$ is critical.748\end{proposition}749 750\begin{proof}751Differentiating $\loss(g_s\theta)=\loss(\theta)$ in $s$ gives $d\loss_\theta(\xi)=0$ for all $\theta$;752differentiating that identity in a direction $v$ at a critical point gives753$\nabla^2\loss(\xi,v)+d\loss(D_v\xi)=\nabla^2\loss(\xi,v)=0$. The last statement is754$\nabla\loss=J^{\mathsf T}\nabla\lambda$.755\end{proof}756 757\begin{corollary}\label{cor:notmorse}\Sob758For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the759form $\lambda\circ\ev$ is a Morse function on any open set containing a critical point at which some hidden760unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least761the number of such units.762\end{corollary}763 764So Morse theory in the form of \DT{Ch.~8} never applies directly to the loss of a $\relu$ network. It can765apply to the function induced on a slice transverse to the orbits, or in Morse--Bott form, with critical766manifolds in place of critical points. For $\tanh$ networks there is no continuous symmetry and no such767obstruction on $\Theta^{\mathrm{irr}}$; the degeneracies there come from the reducible strata and from768overparametrization (\cref{ch:loss}).769 770\section*{Notes}771 772Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann773and by Albertini and Sontag, and for deep $\tanh$ networks by Fefferman. The relation between reducibility774and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of775Watanabe's singular learning theory \cite{Wat09}. The use of permutation symmetry to compare trained776networks is discussed in \cref{ch:loss}.777 778\section*{Exercises}779 780\begin{exercise}\label{exer:verifysym}781Verify the three symmetries of \cref{prop:syms} for a network with one hidden layer.782\end{exercise}783 784\begin{exercise}\label{exer:rankdrop}785For the one-hidden-layer $\tanh$ network, show that if $a_j=0$ then $\rk J(\theta)\le P-(d+1)$ for every set of786inputs, and if $(w_j,b_j)=(w_k,b_k)$ then $\rk J(\theta)\le P-1$.787\end{exercise}788 789\begin{exercise}\label{exer:reluscale}790For $F(\theta,x)=a\,\relu(wx+b)+c$ with $d=1$, write down $\xi$ and verify $D_\theta F\cdot\xi=0$ by hand.791What is the functional dimension?792\end{exercise}793 794\begin{exercise}\label{exer:order}795Compute the order of $G$ for the $\tanh$ architectures with widths $(2,3,1)$ and $(2,4,4,1)$.796\end{exercise}797%% ---- end ch/03-symmetry.tex ----798\part{Level Sets}799%% ---- begin ch/04-decision.tex ----800\chapter{Transversality and Decision Boundaries}\label{ch:decision}