Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 821–920 of 2141. Source text is displayed literally.

821\end{theorem}822 823\begin{proof}824(a) $0$ is a regular value of $G(\theta',\cdot)+c$ exactly when $-c$ is a regular value of825$G(\theta',\cdot)$, and the critical values of a smooth function have measure zero by \DT{Thm.~4.1}. (b)826follows from (a) by Fubini's theorem and \DT{Thm.~1.12}. (c) Apply (a) also to the restriction to827$\partial X$ and use \DT{Thm.~1.17}.828\end{proof}829 830Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever831smooth activation it uses. Part (a) is stronger than the conclusion of \cref{prin:rich}, since it holds for832each value of the other parameters separately; it says that a non-smooth decision boundary can always be833repaired by an arbitrarily small change of the output bias alone. The level sets $f_\theta^{-1}(s)$ for834other thresholds $s$ are covered by the same statement.835 836\begin{example}\label{ex:cross}837Let $f(x)=x_1^2-x_2^2+c$. For $c\ne0$ the boundary is a hyperbola, a $1$-manifold with two components. For838$c=0$ it is a pair of crossing lines, not a manifold. In the language of \DT{Ch.~4}, the boundary at $c=0$ is839a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.840\end{example}841 842\section{Several classes}843 844A classifier with $k$ classes is a map $f_\theta\colon X\to\R^k$, and the predicted class is the index of the845largest coordinate. For a non-empty $I\subset\{1,\dots,k\}$ let $A_I\subset\R^k$ be the set of points whose846largest coordinate is attained exactly on $I$. Then $A_I$ is an open subset of the linear subspace847$\{y_i=y_j:i,j\in I\}$, a submanifold of codimension $|I|-1$, and the $A_I$ partition $\R^k$.848 849\begin{proposition}\label{prop:multiclass}\Srf[\DT{Thm.~4.5}]850Let $F\colon\Theta\times X\to\R^k$ be a family whose last layer is affine with a bias. For almost every851$\theta$, $f_\theta$ is transverse to every $A_I$. Then the decision region of class $i$ is the open set852$f_\theta^{-1}(A_{\{i\}})$; the set where exactly the classes in $I$ tie is a submanifold of codimension853$|I|-1$; and no more than $d+1$ classes tie anywhere.854\end{proposition}855 856\begin{proof}857$F$ is a submersion, so \DT{Thm.~4.5} applies to each of the finitely many $A_I$, and a finite intersection858of sets of full measure has full measure. A submanifold of codimension greater than $d$ is empty.859\end{proof}860 861In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time862at isolated points, like the cells of a foam, and four regions never meet at a point.863 864\section{Chambers}865 866Let $X$ be compact, and for simplicity without boundary. Define867\[\Sigma=\{(\theta,x):F(\theta,x)=0,\ d_xF(\theta,x)=0\}\subset\Theta\times X,868\qquad\Delta=\pi(\Sigma)\subset\Theta ,\]869the set of parameters for which $0$ is not a regular value: the \term{discriminant} of the family.870 871\begin{theorem}[Chambers]\label{thm:chambers}\Srf[\DT{Lem.~4.4, Thm.~8.5, Thm.~10.3}]872Let $F$ be as in \cref{thm:boundary} with $X$ compact. Then $\Delta$ is closed and of measure zero. If873$\gamma$ is a path in $\Theta\setminus\Delta$, there is a diffeotopy $\varphi_t$ of $X$ with874$\varphi_t(B_{\gamma(0)})=B_{\gamma(t)}$. In particular the diffeomorphism type of the pair $(X,B_\theta)$ is875constant on each connected component of $\Theta\setminus\Delta$.876\end{theorem}877 878\begin{proof}879$\Sigma$ is closed and $\pi|_\Sigma$ is proper because $X$ is compact, so $\Delta$ is closed; its complement880has full measure by \cref{thm:boundary}. Over $\Theta\setminus\Delta$ the set $W=F^{-1}(0)$ is a881submanifold of $\Theta\times X$, since $d_xF\ne0$ there, and by \DT{Lem.~4.4} the projection882$W\to\Theta\setminus\Delta$ is a submersion; it is proper. By Ehresmann's theorem \DT{Thm.~8.5} it is a fibre883bundle. Pulled back along $\gamma$ it is trivial, which gives an isotopy of embeddings884$B_{\gamma(0)}\to X$ with image $B_{\gamma(t)}$ at time $t$, and the isotopy extends to a diffeotopy of $X$ by885\DT{Thm.~10.3}.886\end{proof}887 888The components of $\Theta\setminus\Delta$ are the \term{chambers} of the architecture. The theorem says that889a classifier has a well-defined topology (the number of components of its decision boundary and of its890decision regions, their genera, how they are nested) which does not change under small changes of the891parameters, and can change only when the parameter crosses $\Delta$. What happens at a crossing is the892subject of the next chapter. The symmetry group of \cref{ch:symmetry} permutes the chambers.893 894For manifolds with boundary the same holds with $\Delta$ enlarged by the parameters for which $B_\theta$ is895tangent to $\partial X$.896 897\begin{proposition}[Walls]\label{prop:walls}\Sob898Consider the one-hidden-layer architecture899$F(\theta,x)=\sum_{j\le H}a_j\sigma(\langle w_j,x\rangle+b_j)+c$ on $\R^d$ with $\sigma'>0$. At every point900where the matrix $W$ with rows $w_j$ has rank $d$, the map901$\Phi(\theta,x)=(F,d_xF)\in\R\times\R^d$ is a submersion. Hence, over the open set of such parameters,902$\Sigma$ is a submanifold of dimension $P-1$, and $\Delta$ is the image of a manifold of dimension $P-1$.903\end{proposition}904 905\begin{proof}906$d_xF=W^{\mathsf T}(a\odot\sigma'(z))$ with $z=Wx+b$. A variation $\delta a$ changes $d_xF$ by907$W^{\mathsf T}(\delta a\odot\sigma'(z))$, and $\delta a\mapsto\delta a\odot\sigma'(z)$ is a linear908isomorphism since $\sigma'>0$; so if $W^{\mathsf T}$ is onto, every variation of $d_xF$ is produced. The909resulting change in $F$ is compensated by $\delta c$.910\end{proof}911 912So the discriminant is no larger than a hypersurface, and the picture of parameter space is that of open913chambers separated by walls of codimension one. The proposition is stated for the simplest architecture;914the argument needs only that the parameters can move the value and the input-gradient of the network915independently at each point, which holds for deeper networks at parameters where the layer Jacobians have916full rank.917 918\section{Every hypersurface is a decision boundary}919 920\begin{theorem}[Realization]\label{thm:realize}\Srf[\DT{Thm.~6.2}; Hornik \cite{Hor91}]