Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1021–1120 of 2141. Source text is displayed literally.
1021Let $\gamma$ be a path for which $W_\gamma$ is a manifold and $\tau$ is a Morse function with distinct1022critical values $t_1<\dots<t_r$ in $(0,1)$. Then:1023\begin{enumerate}[label=(\alph*)]1024\item On each interval of $[0,1]\setminus\{t_1,\dots,t_r\}$ the boundary $B_t$ moves by an ambient isotopy of1025$X$.1026\item At each $t_i$ there is exactly one point $x_i\in B_{t_i}$ at which $B_{t_i}$ fails to be a manifold. It1027is a non-degenerate critical point of $f_{t_i}$ with $\partial_t\Phi_\gamma(t_i,x_i)\ne0$, and near1028$(t_i,x_i)$ there are coordinates on $X$ in which1029\[\Phi_\gamma(t,x)=\pm(t-t_i)-x_1^2-\dots-x_k^2+x_{k+1}^2+\dots+x_d^2 ,\]1030with $k$ the index of $x_i$ as a critical point of $f_{t_i}$.1031\item $B_{t_i+\varepsilon}$ is obtained from $B_{t_i-\varepsilon}$ by a surgery: a copy of1032$S^{\lambda-1}\times D^{d-\lambda}$ is removed and $D^\lambda\times S^{d-\lambda-1}$ is glued in, where1033$\lambda$ is the index of $\tau$ at $(t_i,x_i)$; $\lambda=k$ or $d-k$ according to the sign in (b).1034Correspondingly the region $\{f_t\le0\}$ gains or loses a handle of index $k$.1035\end{enumerate}1036\end{theorem}1037 1038\begin{proof}1039(a) is \cref{thm:chambers}. For (b): at a critical point of $\tau$ on $W_\gamma$ the tangent space of1040$W_\gamma$ is $\{dt=0\}$, so $d_x\Phi_\gamma=0$ and, $W_\gamma$ being a manifold, $\partial_t\Phi_\gamma\ne0$.1041Solving $\Phi_\gamma=0$ for $t$ as a function of $x$ near the point exhibits $W_\gamma$ as a graph over $X$1042on which $\tau$ is the function $t(x)$, whose Hessian at $x_i$ is1043$-\nabla^2f_{t_i}(x_i)/\partial_t\Phi_\gamma$. So $\tau$ is non-degenerate there exactly when $x_i$ is a1044non-degenerate critical point of $f_{t_i}$, and the normal form follows from the Morse lemma with parameters.1045(c) is \DT{Thm.~8.6} for $\tau$ on $W_\gamma$: the level sets of $\tau$ are the $B_t$, and passing a critical1046point of index $\lambda$ changes the level set by the stated surgery.1047\end{proof}1048 1049In words: the topology of a decision boundary changes exactly when a critical point of the network function1050crosses the threshold, and the change is read off from the index of the critical point and the direction of1051crossing. \Cref{tab:surgery} lists the cases for inputs of dimension two and three.1052 1053\begin{table}[ht]\centering\small1054\begin{tabular}{@{}llp{0.58\textwidth}@{}}1055\toprule1056$d$ & index $k$ of $x_i$ & effect on the region $\{f_t\le0\}$ when the critical value becomes negative\\1057\midrule1058$2$ & $0$ (minimum) & a new disc appears; the boundary gains a circle\\1059$2$ & $1$ (saddle) & two parts of the region join, or one part meets itself; two boundary arcs1060reconnect, so the number of boundary circles changes by $\pm1$\\1061$2$ & $2$ (maximum) & a hole in the region is filled; a boundary circle disappears\\1062$3$ & $0$ & a new ball appears; the boundary gains a sphere\\1063$3$ & $1$ & a tube joins two parts of the region, or adds a handle to one\\1064$3$ & $2$ & a tunnel through the region is closed by a disc, or a cavity is split in two\\1065$3$ & $3$ & a cavity is filled; a boundary sphere disappears\\1066\bottomrule1067\end{tabular}1068\caption{Elementary changes of topology of a decision region, by the index of the critical point crossing1069zero from above. Crossing in the other direction reverses each.}\label{tab:surgery}1070\end{table}1071 1072\begin{proposition}\label{prop:chi}\Sob1073In the situation of \cref{thm:surgery}(c), $\chi(B_{t_i+\varepsilon})-\chi(B_{t_i-\varepsilon})$ equals1074$2(-1)^\lambda$ if $d$ is odd and $0$ if $d$ is even, and the Euler characteristic of the region1075$\{f_t\le0\}$ changes by $\pm(-1)^k$.1076\end{proposition}1077 1078\begin{proof}1079The change is $\chi(D^\lambda\times S^{d-\lambda-1})-\chi(S^{\lambda-1}\times D^{d-\lambda})1080=\chi(S^{d-\lambda-1})-\chi(S^{\lambda-1})=(-1)^{d-\lambda-1}-(-1)^{\lambda-1}$. The second statement is1081additivity of $\chi$ under attaching a $k$-cell.1082\end{proof}1083 1084\begin{example}\label{ex:twoblobs}1085Let $F(t,x)=(x_1^2-1)^2+x_2^2-t$ on the plane. The function $(x_1^2-1)^2+x_2^2$ has minima at $(\pm1,0)$ with1086value $0$ and a saddle at the origin with value $1$. For $t<0$ the region $\{F\le0\}$ is empty. At $t=0$ two1087discs are born (two critical points cross at the same time, which is not generic; a small asymmetry1088separates them). At $t=1$ the saddle crosses and the two discs merge. The numbers of components are $0$,1089$2$, $1$, and the Euler characteristics of the region $0$, $2$, $1$. The script counts components on a1090grid.1091\end{example}1092 1093\section{Genericity of paths}1094 1095Are the hypotheses of \cref{thm:surgery} generic? For paths in the space of all functions, yes: this is the1096content of \DT{Thm.~8.12}. For paths in a given architecture one needs enough parameters to move critical1097points and critical values independently.1098 1099\begin{proposition}\label{prop:genericpath}\Srf[\DT{Thm.~4.6, Thm.~8.12}]1100For the architecture of \cref{prop:walls}, with $H\ge d$, let $\gamma$ be a path with endpoints in1101$\Theta\setminus\Delta$ along which $W$ has rank $d$. Then arbitrarily close to $\gamma$, with the same1102endpoints, there is a path satisfying the hypotheses of \cref{thm:surgery}.1103\end{proposition}1104 1105\begin{proof}[Sketch]1106By \cref{prop:walls} the map $(\theta,x)\mapsto(F,d_xF)$ is a submersion along the path, so the family of1107perturbations of $\gamma$ by compactly supported variations of $(a,c)$ induces a family of maps1108$(t,x)\mapsto(\Phi,d_x\Phi)$ transverse to the zero section; by \DT{Thm.~4.5} a generic member has $\Sigma$1109pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at1110those points, and distinctness of the times, are further conditions of positive codimension on the1111$2$-jet, which a variation of $W$ and $b$ moves freely; \DT{Thm.~4.6} applies.1112\end{proof}1113 1114A training path produced by gradient descent is one particular path, and nothing guarantees that it is1115generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and1116that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the1117typical exception (\cref{ex:twoblobs}): a symmetric network crossing a wall has several critical points1118crossing together.1119 1120\section{What cobordism does and does not see}