Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 801–900 of 2141. Source text is displayed literally.

801 802A binary classifier is a function $f_\theta\colon X\to\R$ whose sign is the predicted class, and its803\term{decision boundary} is $B_\theta=f_\theta^{-1}(0)$. This chapter answers question (Q1) of804\cref{ch:families} for decision boundaries: for almost every parameter the boundary is a smooth hypersurface;805its isotopy type is constant on the components of the complement of a closed null set of parameters; and806every isotopy type of compact hypersurface occurs for wide enough networks.807 808\section{The boundary is generically a hypersurface}809 810\begin{theorem}[Generic decision boundaries]\label{thm:boundary}\Srf[\DT{Thm.~4.1, Thm.~1.12}]811Let $F\colon\Theta\times X\to\R$ be a family of the form $F((\theta',c),x)=G(\theta',x)+c$.812\begin{enumerate}[label=(\alph*)]813\item For every $\theta'$, the set of $c$ for which $0$ fails to be a regular value of $f_{(\theta',c)}$ has814measure zero.815\item For almost every $\theta$, the decision boundary $B_\theta$ is a smooth hypersurface in $X$, closed as816a subset and possibly empty, with trivial normal bundle oriented by $\nabla f_\theta$; and817$X\setminus B_\theta$ is the disjoint union of the open sets $\{f_\theta>0\}$ and $\{f_\theta<0\}$.818\item If $X$ is a compact manifold with boundary, then for almost every $\theta$ the boundary $B_\theta$ is819moreover a neat submanifold: it meets $\partial X$ transversally, in $\partial B_\theta$.820\end{enumerate}821\end{theorem}822 823\begin{proof}824(a) $0$ is a regular value of $G(\theta',\cdot)+c$ exactly when $-c$ is a regular value of825$G(\theta',\cdot)$, and the critical values of a smooth function have measure zero by \DT{Thm.~4.1}. (b)826follows from (a) by Fubini's theorem and \DT{Thm.~1.12}. (c) Apply (a) also to the restriction to827$\partial X$ and use \DT{Thm.~1.17}.828\end{proof}829 830Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever831smooth activation it uses. Part (a) is stronger than the conclusion of \cref{prin:rich}, since it holds for832each value of the other parameters separately; it says that a non-smooth decision boundary can always be833repaired by an arbitrarily small change of the output bias alone. The level sets $f_\theta^{-1}(s)$ for834other thresholds $s$ are covered by the same statement.835 836\begin{example}\label{ex:cross}837Let $f(x)=x_1^2-x_2^2+c$. For $c\ne0$ the boundary is a hyperbola, a $1$-manifold with two components. For838$c=0$ it is a pair of crossing lines, not a manifold. In the language of \DT{Ch.~4}, the boundary at $c=0$ is839a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.840\end{example}841 842\section{Several classes}843 844A classifier with $k$ classes is a map $f_\theta\colon X\to\R^k$, and the predicted class is the index of the845largest coordinate. For a non-empty $I\subset\{1,\dots,k\}$ let $A_I\subset\R^k$ be the set of points whose846largest coordinate is attained exactly on $I$. Then $A_I$ is an open subset of the linear subspace847$\{y_i=y_j:i,j\in I\}$, a submanifold of codimension $|I|-1$, and the $A_I$ partition $\R^k$.848 849\begin{proposition}\label{prop:multiclass}\Srf[\DT{Thm.~4.5}]850Let $F\colon\Theta\times X\to\R^k$ be a family whose last layer is affine with a bias. For almost every851$\theta$, $f_\theta$ is transverse to every $A_I$. Then the decision region of class $i$ is the open set852$f_\theta^{-1}(A_{\{i\}})$; the set where exactly the classes in $I$ tie is a submanifold of codimension853$|I|-1$; and no more than $d+1$ classes tie anywhere.854\end{proposition}855 856\begin{proof}857$F$ is a submersion, so \DT{Thm.~4.5} applies to each of the finitely many $A_I$, and a finite intersection858of sets of full measure has full measure. A submanifold of codimension greater than $d$ is empty.859\end{proof}860 861In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time862at isolated points, like the cells of a foam, and four regions never meet at a point.863 864\section{Chambers}865 866Let $X$ be compact, and for simplicity without boundary. Define867\[\Sigma=\{(\theta,x):F(\theta,x)=0,\ d_xF(\theta,x)=0\}\subset\Theta\times X,868\qquad\Delta=\pi(\Sigma)\subset\Theta ,\]869the set of parameters for which $0$ is not a regular value: the \term{discriminant} of the family.870 871\begin{theorem}[Chambers]\label{thm:chambers}\Srf[\DT{Lem.~4.4, Thm.~8.5, Thm.~10.3}]872Let $F$ be as in \cref{thm:boundary} with $X$ compact. Then $\Delta$ is closed and of measure zero. If873$\gamma$ is a path in $\Theta\setminus\Delta$, there is a diffeotopy $\varphi_t$ of $X$ with874$\varphi_t(B_{\gamma(0)})=B_{\gamma(t)}$. In particular the diffeomorphism type of the pair $(X,B_\theta)$ is875constant on each connected component of $\Theta\setminus\Delta$.876\end{theorem}877 878\begin{proof}879$\Sigma$ is closed and $\pi|_\Sigma$ is proper because $X$ is compact, so $\Delta$ is closed; its complement880has full measure by \cref{thm:boundary}. Over $\Theta\setminus\Delta$ the set $W=F^{-1}(0)$ is a881submanifold of $\Theta\times X$, since $d_xF\ne0$ there, and by \DT{Lem.~4.4} the projection882$W\to\Theta\setminus\Delta$ is a submersion; it is proper. By Ehresmann's theorem \DT{Thm.~8.5} it is a fibre883bundle. Pulled back along $\gamma$ it is trivial, which gives an isotopy of embeddings884$B_{\gamma(0)}\to X$ with image $B_{\gamma(t)}$ at time $t$, and the isotopy extends to a diffeotopy of $X$ by885\DT{Thm.~10.3}.886\end{proof}887 888The components of $\Theta\setminus\Delta$ are the \term{chambers} of the architecture. The theorem says that889a classifier has a well-defined topology (the number of components of its decision boundary and of its890decision regions, their genera, how they are nested) which does not change under small changes of the891parameters, and can change only when the parameter crosses $\Delta$. What happens at a crossing is the892subject of the next chapter. The symmetry group of \cref{ch:symmetry} permutes the chambers.893 894For manifolds with boundary the same holds with $\Delta$ enlarged by the parameters for which $B_\theta$ is895tangent to $\partial X$.896 897\begin{proposition}[Walls]\label{prop:walls}\Sob898Consider the one-hidden-layer architecture899$F(\theta,x)=\sum_{j\le H}a_j\sigma(\langle w_j,x\rangle+b_j)+c$ on $\R^d$ with $\sigma'>0$. At every point900where the matrix $W$ with rows $w_j$ has rank $d$, the map