Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 781–880 of 2141. Source text is displayed literally.
781Verify the three symmetries of \cref{prop:syms} for a network with one hidden layer.782\end{exercise}783 784\begin{exercise}\label{exer:rankdrop}785For the one-hidden-layer $\tanh$ network, show that if $a_j=0$ then $\rk J(\theta)\le P-(d+1)$ for every set of786inputs, and if $(w_j,b_j)=(w_k,b_k)$ then $\rk J(\theta)\le P-1$.787\end{exercise}788 789\begin{exercise}\label{exer:reluscale}790For $F(\theta,x)=a\,\relu(wx+b)+c$ with $d=1$, write down $\xi$ and verify $D_\theta F\cdot\xi=0$ by hand.791What is the functional dimension?792\end{exercise}793 794\begin{exercise}\label{exer:order}795Compute the order of $G$ for the $\tanh$ architectures with widths $(2,3,1)$ and $(2,4,4,1)$.796\end{exercise}797%% ---- end ch/03-symmetry.tex ----798\part{Level Sets}799%% ---- begin ch/04-decision.tex ----800\chapter{Transversality and Decision Boundaries}\label{ch:decision}801 802A binary classifier is a function $f_\theta\colon X\to\R$ whose sign is the predicted class, and its803\term{decision boundary} is $B_\theta=f_\theta^{-1}(0)$. This chapter answers question (Q1) of804\cref{ch:families} for decision boundaries: for almost every parameter the boundary is a smooth hypersurface;805its isotopy type is constant on the components of the complement of a closed null set of parameters; and806every isotopy type of compact hypersurface occurs for wide enough networks.807 808\section{The boundary is generically a hypersurface}809 810\begin{theorem}[Generic decision boundaries]\label{thm:boundary}\Srf[\DT{Thm.~4.1, Thm.~1.12}]811Let $F\colon\Theta\times X\to\R$ be a family of the form $F((\theta',c),x)=G(\theta',x)+c$.812\begin{enumerate}[label=(\alph*)]813\item For every $\theta'$, the set of $c$ for which $0$ fails to be a regular value of $f_{(\theta',c)}$ has814measure zero.815\item For almost every $\theta$, the decision boundary $B_\theta$ is a smooth hypersurface in $X$, closed as816a subset and possibly empty, with trivial normal bundle oriented by $\nabla f_\theta$; and817$X\setminus B_\theta$ is the disjoint union of the open sets $\{f_\theta>0\}$ and $\{f_\theta<0\}$.818\item If $X$ is a compact manifold with boundary, then for almost every $\theta$ the boundary $B_\theta$ is819moreover a neat submanifold: it meets $\partial X$ transversally, in $\partial B_\theta$.820\end{enumerate}821\end{theorem}822 823\begin{proof}824(a) $0$ is a regular value of $G(\theta',\cdot)+c$ exactly when $-c$ is a regular value of825$G(\theta',\cdot)$, and the critical values of a smooth function have measure zero by \DT{Thm.~4.1}. (b)826follows from (a) by Fubini's theorem and \DT{Thm.~1.12}. (c) Apply (a) also to the restriction to827$\partial X$ and use \DT{Thm.~1.17}.828\end{proof}829 830Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever831smooth activation it uses. Part (a) is stronger than the conclusion of \cref{prin:rich}, since it holds for832each value of the other parameters separately; it says that a non-smooth decision boundary can always be833repaired by an arbitrarily small change of the output bias alone. The level sets $f_\theta^{-1}(s)$ for834other thresholds $s$ are covered by the same statement.835 836\begin{example}\label{ex:cross}837Let $f(x)=x_1^2-x_2^2+c$. For $c\ne0$ the boundary is a hyperbola, a $1$-manifold with two components. For838$c=0$ it is a pair of crossing lines, not a manifold. In the language of \DT{Ch.~4}, the boundary at $c=0$ is839a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.840\end{example}841 842\section{Several classes}843 844A classifier with $k$ classes is a map $f_\theta\colon X\to\R^k$, and the predicted class is the index of the845largest coordinate. For a non-empty $I\subset\{1,\dots,k\}$ let $A_I\subset\R^k$ be the set of points whose846largest coordinate is attained exactly on $I$. Then $A_I$ is an open subset of the linear subspace847$\{y_i=y_j:i,j\in I\}$, a submanifold of codimension $|I|-1$, and the $A_I$ partition $\R^k$.848 849\begin{proposition}\label{prop:multiclass}\Srf[\DT{Thm.~4.5}]850Let $F\colon\Theta\times X\to\R^k$ be a family whose last layer is affine with a bias. For almost every851$\theta$, $f_\theta$ is transverse to every $A_I$. Then the decision region of class $i$ is the open set852$f_\theta^{-1}(A_{\{i\}})$; the set where exactly the classes in $I$ tie is a submanifold of codimension853$|I|-1$; and no more than $d+1$ classes tie anywhere.854\end{proposition}855 856\begin{proof}857$F$ is a submersion, so \DT{Thm.~4.5} applies to each of the finitely many $A_I$, and a finite intersection858of sets of full measure has full measure. A submanifold of codimension greater than $d$ is empty.859\end{proof}860 861In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time862at isolated points, like the cells of a foam, and four regions never meet at a point.863 864\section{Chambers}865 866Let $X$ be compact, and for simplicity without boundary. Define867\[\Sigma=\{(\theta,x):F(\theta,x)=0,\ d_xF(\theta,x)=0\}\subset\Theta\times X,868\qquad\Delta=\pi(\Sigma)\subset\Theta ,\]869the set of parameters for which $0$ is not a regular value: the \term{discriminant} of the family.870 871\begin{theorem}[Chambers]\label{thm:chambers}\Srf[\DT{Lem.~4.4, Thm.~8.5, Thm.~10.3}]872Let $F$ be as in \cref{thm:boundary} with $X$ compact. Then $\Delta$ is closed and of measure zero. If873$\gamma$ is a path in $\Theta\setminus\Delta$, there is a diffeotopy $\varphi_t$ of $X$ with874$\varphi_t(B_{\gamma(0)})=B_{\gamma(t)}$. In particular the diffeomorphism type of the pair $(X,B_\theta)$ is875constant on each connected component of $\Theta\setminus\Delta$.876\end{theorem}877 878\begin{proof}879$\Sigma$ is closed and $\pi|_\Sigma$ is proper because $X$ is compact, so $\Delta$ is closed; its complement880has full measure by \cref{thm:boundary}. Over $\Theta\setminus\Delta$ the set $W=F^{-1}(0)$ is a