Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1041–1140 of 2141. Source text is displayed literally.

1041Solving $\Phi_\gamma=0$ for $t$ as a function of $x$ near the point exhibits $W_\gamma$ as a graph over $X$1042on which $\tau$ is the function $t(x)$, whose Hessian at $x_i$ is1043$-\nabla^2f_{t_i}(x_i)/\partial_t\Phi_\gamma$. So $\tau$ is non-degenerate there exactly when $x_i$ is a1044non-degenerate critical point of $f_{t_i}$, and the normal form follows from the Morse lemma with parameters.1045(c) is \DT{Thm.~8.6} for $\tau$ on $W_\gamma$: the level sets of $\tau$ are the $B_t$, and passing a critical1046point of index $\lambda$ changes the level set by the stated surgery.1047\end{proof}1048 1049In words: the topology of a decision boundary changes exactly when a critical point of the network function1050crosses the threshold, and the change is read off from the index of the critical point and the direction of1051crossing. \Cref{tab:surgery} lists the cases for inputs of dimension two and three.1052 1053\begin{table}[ht]\centering\small1054\begin{tabular}{@{}llp{0.58\textwidth}@{}}1055\toprule1056$d$ & index $k$ of $x_i$ & effect on the region $\{f_t\le0\}$ when the critical value becomes negative\\1057\midrule1058$2$ & $0$ (minimum) & a new disc appears; the boundary gains a circle\\1059$2$ & $1$ (saddle) & two parts of the region join, or one part meets itself; two boundary arcs1060reconnect, so the number of boundary circles changes by $\pm1$\\1061$2$ & $2$ (maximum) & a hole in the region is filled; a boundary circle disappears\\1062$3$ & $0$ & a new ball appears; the boundary gains a sphere\\1063$3$ & $1$ & a tube joins two parts of the region, or adds a handle to one\\1064$3$ & $2$ & a tunnel through the region is closed by a disc, or a cavity is split in two\\1065$3$ & $3$ & a cavity is filled; a boundary sphere disappears\\1066\bottomrule1067\end{tabular}1068\caption{Elementary changes of topology of a decision region, by the index of the critical point crossing1069zero from above. Crossing in the other direction reverses each.}\label{tab:surgery}1070\end{table}1071 1072\begin{proposition}\label{prop:chi}\Sob1073In the situation of \cref{thm:surgery}(c), $\chi(B_{t_i+\varepsilon})-\chi(B_{t_i-\varepsilon})$ equals1074$2(-1)^\lambda$ if $d$ is odd and $0$ if $d$ is even, and the Euler characteristic of the region1075$\{f_t\le0\}$ changes by $\pm(-1)^k$.1076\end{proposition}1077 1078\begin{proof}1079The change is $\chi(D^\lambda\times S^{d-\lambda-1})-\chi(S^{\lambda-1}\times D^{d-\lambda})1080=\chi(S^{d-\lambda-1})-\chi(S^{\lambda-1})=(-1)^{d-\lambda-1}-(-1)^{\lambda-1}$. The second statement is1081additivity of $\chi$ under attaching a $k$-cell.1082\end{proof}1083 1084\begin{example}\label{ex:twoblobs}1085Let $F(t,x)=(x_1^2-1)^2+x_2^2-t$ on the plane. The function $(x_1^2-1)^2+x_2^2$ has minima at $(\pm1,0)$ with1086value $0$ and a saddle at the origin with value $1$. For $t<0$ the region $\{F\le0\}$ is empty. At $t=0$ two1087discs are born (two critical points cross at the same time, which is not generic; a small asymmetry1088separates them). At $t=1$ the saddle crosses and the two discs merge. The numbers of components are $0$,1089$2$, $1$, and the Euler characteristics of the region $0$, $2$, $1$. The script counts components on a1090grid.1091\end{example}1092 1093\section{Genericity of paths}1094 1095Are the hypotheses of \cref{thm:surgery} generic? For paths in the space of all functions, yes: this is the1096content of \DT{Thm.~8.12}. For paths in a given architecture one needs enough parameters to move critical1097points and critical values independently.1098 1099\begin{proposition}\label{prop:genericpath}\Srf[\DT{Thm.~4.6, Thm.~8.12}]1100For the architecture of \cref{prop:walls}, with $H\ge d$, let $\gamma$ be a path with endpoints in1101$\Theta\setminus\Delta$ along which $W$ has rank $d$. Then arbitrarily close to $\gamma$, with the same1102endpoints, there is a path satisfying the hypotheses of \cref{thm:surgery}.1103\end{proposition}1104 1105\begin{proof}[Sketch]1106By \cref{prop:walls} the map $(\theta,x)\mapsto(F,d_xF)$ is a submersion along the path, so the family of1107perturbations of $\gamma$ by compactly supported variations of $(a,c)$ induces a family of maps1108$(t,x)\mapsto(\Phi,d_x\Phi)$ transverse to the zero section; by \DT{Thm.~4.5} a generic member has $\Sigma$1109pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at1110those points, and distinctness of the times, are further conditions of positive codimension on the1111$2$-jet, which a variation of $W$ and $b$ moves freely; \DT{Thm.~4.6} applies.1112\end{proof}1113 1114A training path produced by gradient descent is one particular path, and nothing guarantees that it is1115generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and1116that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the1117typical exception (\cref{ex:twoblobs}): a symmetric network crossing a wall has several critical points1118crossing together.1119 1120\section{What cobordism does and does not see}1121 1122\begin{proposition}\label{prop:constraints}\Scl1123Let $d\ge2$ and let $B=f^{-1}(0)\subset\R^d$ be compact with $0$ a regular value. Then $B$ is the boundary of a1124compact region of $\R^d$; it is orientable, its normal bundle is trivial, and $TB\oplus\R$ is trivial. In1125particular $B$ is null-cobordant, all its Stiefel--Whitney numbers vanish, and no closed non-orientable1126surface is a decision boundary in $\R^3$.1127\end{proposition}1128 1129\begin{proof}1130Choose a closed ball $D$ containing $B$ in its interior. Its boundary sphere is connected and $f$ does not1131vanish on it, so $f$ has a constant sign there, say positive. Then $\{f\le0\}\cap D$ is a compact manifold1132with boundary $B$. The normal bundle is trivialized by $\nabla f$, and $TB\oplus\nu=T\R^d|_B$ is trivial.1133\end{proof}1134 1135So the cobordism class of a decision boundary carries no information: it is always zero, as1136\DT{Thm.~4.16} predicts for a fibre of a map to a contractible space. What carries information is1137finer: the diffeomorphism type of the pair $(X,B_t)$ in each chamber, and the sequence of surgeries along a1138path.1139 1140That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is