Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1081–1180 of 2141. Source text is displayed literally.

1081additivity of $\chi$ under attaching a $k$-cell.1082\end{proof}1083 1084\begin{example}\label{ex:twoblobs}1085Let $F(t,x)=(x_1^2-1)^2+x_2^2-t$ on the plane. The function $(x_1^2-1)^2+x_2^2$ has minima at $(\pm1,0)$ with1086value $0$ and a saddle at the origin with value $1$. For $t<0$ the region $\{F\le0\}$ is empty. At $t=0$ two1087discs are born (two critical points cross at the same time, which is not generic; a small asymmetry1088separates them). At $t=1$ the saddle crosses and the two discs merge. The numbers of components are $0$,1089$2$, $1$, and the Euler characteristics of the region $0$, $2$, $1$. The script counts components on a1090grid.1091\end{example}1092 1093\section{Genericity of paths}1094 1095Are the hypotheses of \cref{thm:surgery} generic? For paths in the space of all functions, yes: this is the1096content of \DT{Thm.~8.12}. For paths in a given architecture one needs enough parameters to move critical1097points and critical values independently.1098 1099\begin{proposition}\label{prop:genericpath}\Srf[\DT{Thm.~4.6, Thm.~8.12}]1100For the architecture of \cref{prop:walls}, with $H\ge d$, let $\gamma$ be a path with endpoints in1101$\Theta\setminus\Delta$ along which $W$ has rank $d$. Then arbitrarily close to $\gamma$, with the same1102endpoints, there is a path satisfying the hypotheses of \cref{thm:surgery}.1103\end{proposition}1104 1105\begin{proof}[Sketch]1106By \cref{prop:walls} the map $(\theta,x)\mapsto(F,d_xF)$ is a submersion along the path, so the family of1107perturbations of $\gamma$ by compactly supported variations of $(a,c)$ induces a family of maps1108$(t,x)\mapsto(\Phi,d_x\Phi)$ transverse to the zero section; by \DT{Thm.~4.5} a generic member has $\Sigma$1109pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at1110those points, and distinctness of the times, are further conditions of positive codimension on the1111$2$-jet, which a variation of $W$ and $b$ moves freely; \DT{Thm.~4.6} applies.1112\end{proof}1113 1114A training path produced by gradient descent is one particular path, and nothing guarantees that it is1115generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and1116that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the1117typical exception (\cref{ex:twoblobs}): a symmetric network crossing a wall has several critical points1118crossing together.1119 1120\section{What cobordism does and does not see}1121 1122\begin{proposition}\label{prop:constraints}\Scl1123Let $d\ge2$ and let $B=f^{-1}(0)\subset\R^d$ be compact with $0$ a regular value. Then $B$ is the boundary of a1124compact region of $\R^d$; it is orientable, its normal bundle is trivial, and $TB\oplus\R$ is trivial. In1125particular $B$ is null-cobordant, all its Stiefel--Whitney numbers vanish, and no closed non-orientable1126surface is a decision boundary in $\R^3$.1127\end{proposition}1128 1129\begin{proof}1130Choose a closed ball $D$ containing $B$ in its interior. Its boundary sphere is connected and $f$ does not1131vanish on it, so $f$ has a constant sign there, say positive. Then $\{f\le0\}\cap D$ is a compact manifold1132with boundary $B$. The normal bundle is trivialized by $\nabla f$, and $TB\oplus\nu=T\R^d|_B$ is trivial.1133\end{proof}1134 1135So the cobordism class of a decision boundary carries no information: it is always zero, as1136\DT{Thm.~4.16} predicts for a fibre of a map to a contractible space. What carries information is1137finer: the diffeomorphism type of the pair $(X,B_t)$ in each chamber, and the sequence of surgeries along a1138path.1139 1140That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is1141contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between1142generic paths meets a short list of codimension-two events.1143 1144\begin{proposition}[Moves]\label{prop:moves}\Srf[Cerf; \DT{Thm.~8.12}]1145Let $\gamma_0,\gamma_1$ be generic paths with the same endpoints in an architecture rich enough for jet1146transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three1147kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or1148deleting a pair of consecutive surgeries of indices $\lambda,\lambda+1$ that cancel (a birth--death point of1149$f_t$ on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that1150touches zero and returns).1151\end{proposition}1152 1153The proposition is the two-parameter analogue of \cref{thm:surgery}, and is the form Cerf's theorem takes1154for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and1155that anything invariant under them, for instance the parity of the number of surgeries of each index1156modulo the cancellation of adjacent pairs, depends only on the endpoints.1157 1158\section*{Notes}1159 1160The description of how level sets change in one-parameter families is classical Morse and Cerf theory;1161see Milnor's lectures on the h-cobordism theorem. Its use for decision boundaries during training appears1162in empirical form in studies that track Betti numbers of decision regions and of data representations,1163such as \cite{NZL20}. Persistent homology of the family $\{f_t\le s\}$ in the two parameters $(t,s)$ is the1164natural computational tool, and the events of \cref{prop:moves} are the generic singularities of its1165bifiltration.1166 1167\section*{Exercises}1168 1169\begin{exercise}\label{exer:blobs}1170Verify \cref{ex:twoblobs}: find the critical points and indices of $(x_1^2-1)^2+x_2^2$ and the topology of1171the sublevel sets.1172\end{exercise}1173 1174\begin{exercise}\label{exer:d1path}1175For $d=1$, show that along a generic path the points of the decision boundary are born and die in pairs, and1176that the parity of their number is constant if $X$ is a circle.1177\end{exercise}1178 1179\begin{exercise}\label{exer:chi3}1180In $d=3$, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries,