Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 741–840 of 2141. Source text is displayed literally.
741 742\begin{proposition}\label{prop:invariant}\Scl743Let $\loss\colon\Theta\to\R$ be invariant under a group $G$ of symmetries. Then the critical set of744$\loss$ is a union of $G$-orbits. If $G$ contains a one-parameter group with generator $\xi$, then745$d\loss(\xi)=0$ identically and, at every critical point, $\xi$ lies in the kernel of the Hessian. If746moreover $\nabla\lambda$ vanishes at $\ev(\theta)$ for a loss $\loss=\lambda\circ\ev$, then the whole fibre747of $\ev$ through $\theta$ is critical.748\end{proposition}749 750\begin{proof}751Differentiating $\loss(g_s\theta)=\loss(\theta)$ in $s$ gives $d\loss_\theta(\xi)=0$ for all $\theta$;752differentiating that identity in a direction $v$ at a critical point gives753$\nabla^2\loss(\xi,v)+d\loss(D_v\xi)=\nabla^2\loss(\xi,v)=0$. The last statement is754$\nabla\loss=J^{\mathsf T}\nabla\lambda$.755\end{proof}756 757\begin{corollary}\label{cor:notmorse}\Sob758For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the759form $\lambda\circ\ev$ is a Morse function on any open set containing a critical point at which some hidden760unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least761the number of such units.762\end{corollary}763 764So Morse theory in the form of \DT{Ch.~8} never applies directly to the loss of a $\relu$ network. It can765apply to the function induced on a slice transverse to the orbits, or in Morse--Bott form, with critical766manifolds in place of critical points. For $\tanh$ networks there is no continuous symmetry and no such767obstruction on $\Theta^{\mathrm{irr}}$; the degeneracies there come from the reducible strata and from768overparametrization (\cref{ch:loss}).769 770\section*{Notes}771 772Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann773and by Albertini and Sontag, and for deep $\tanh$ networks by Fefferman. The relation between reducibility774and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of775Watanabe's singular learning theory \cite{Wat09}. The use of permutation symmetry to compare trained776networks is discussed in \cref{ch:loss}.777 778\section*{Exercises}779 780\begin{exercise}\label{exer:verifysym}781Verify the three symmetries of \cref{prop:syms} for a network with one hidden layer.782\end{exercise}783 784\begin{exercise}\label{exer:rankdrop}785For the one-hidden-layer $\tanh$ network, show that if $a_j=0$ then $\rk J(\theta)\le P-(d+1)$ for every set of786inputs, and if $(w_j,b_j)=(w_k,b_k)$ then $\rk J(\theta)\le P-1$.787\end{exercise}788 789\begin{exercise}\label{exer:reluscale}790For $F(\theta,x)=a\,\relu(wx+b)+c$ with $d=1$, write down $\xi$ and verify $D_\theta F\cdot\xi=0$ by hand.791What is the functional dimension?792\end{exercise}793 794\begin{exercise}\label{exer:order}795Compute the order of $G$ for the $\tanh$ architectures with widths $(2,3,1)$ and $(2,4,4,1)$.796\end{exercise}797%% ---- end ch/03-symmetry.tex ----798\part{Level Sets}799%% ---- begin ch/04-decision.tex ----800\chapter{Transversality and Decision Boundaries}\label{ch:decision}801 802A binary classifier is a function $f_\theta\colon X\to\R$ whose sign is the predicted class, and its803\term{decision boundary} is $B_\theta=f_\theta^{-1}(0)$. This chapter answers question (Q1) of804\cref{ch:families} for decision boundaries: for almost every parameter the boundary is a smooth hypersurface;805its isotopy type is constant on the components of the complement of a closed null set of parameters; and806every isotopy type of compact hypersurface occurs for wide enough networks.807 808\section{The boundary is generically a hypersurface}809 810\begin{theorem}[Generic decision boundaries]\label{thm:boundary}\Srf[\DT{Thm.~4.1, Thm.~1.12}]811Let $F\colon\Theta\times X\to\R$ be a family of the form $F((\theta',c),x)=G(\theta',x)+c$.812\begin{enumerate}[label=(\alph*)]813\item For every $\theta'$, the set of $c$ for which $0$ fails to be a regular value of $f_{(\theta',c)}$ has814measure zero.815\item For almost every $\theta$, the decision boundary $B_\theta$ is a smooth hypersurface in $X$, closed as816a subset and possibly empty, with trivial normal bundle oriented by $\nabla f_\theta$; and817$X\setminus B_\theta$ is the disjoint union of the open sets $\{f_\theta>0\}$ and $\{f_\theta<0\}$.818\item If $X$ is a compact manifold with boundary, then for almost every $\theta$ the boundary $B_\theta$ is819moreover a neat submanifold: it meets $\partial X$ transversally, in $\partial B_\theta$.820\end{enumerate}821\end{theorem}822 823\begin{proof}824(a) $0$ is a regular value of $G(\theta',\cdot)+c$ exactly when $-c$ is a regular value of825$G(\theta',\cdot)$, and the critical values of a smooth function have measure zero by \DT{Thm.~4.1}. (b)826follows from (a) by Fubini's theorem and \DT{Thm.~1.12}. (c) Apply (a) also to the restriction to827$\partial X$ and use \DT{Thm.~1.17}.828\end{proof}829 830Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever831smooth activation it uses. Part (a) is stronger than the conclusion of \cref{prin:rich}, since it holds for832each value of the other parameters separately; it says that a non-smooth decision boundary can always be833repaired by an arbitrarily small change of the output bias alone. The level sets $f_\theta^{-1}(s)$ for834other thresholds $s$ are covered by the same statement.835 836\begin{example}\label{ex:cross}837Let $f(x)=x_1^2-x_2^2+c$. For $c\ne0$ the boundary is a hyperbola, a $1$-manifold with two components. For838$c=0$ it is a pair of crossing lines, not a manifold. In the language of \DT{Ch.~4}, the boundary at $c=0$ is839a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.840\end{example}