Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1121–1220 of 2141. Source text is displayed literally.
1121 1122\begin{proposition}\label{prop:constraints}\Scl1123Let $d\ge2$ and let $B=f^{-1}(0)\subset\R^d$ be compact with $0$ a regular value. Then $B$ is the boundary of a1124compact region of $\R^d$; it is orientable, its normal bundle is trivial, and $TB\oplus\R$ is trivial. In1125particular $B$ is null-cobordant, all its Stiefel--Whitney numbers vanish, and no closed non-orientable1126surface is a decision boundary in $\R^3$.1127\end{proposition}1128 1129\begin{proof}1130Choose a closed ball $D$ containing $B$ in its interior. Its boundary sphere is connected and $f$ does not1131vanish on it, so $f$ has a constant sign there, say positive. Then $\{f\le0\}\cap D$ is a compact manifold1132with boundary $B$. The normal bundle is trivialized by $\nabla f$, and $TB\oplus\nu=T\R^d|_B$ is trivial.1133\end{proof}1134 1135So the cobordism class of a decision boundary carries no information: it is always zero, as1136\DT{Thm.~4.16} predicts for a fibre of a map to a contractible space. What carries information is1137finer: the diffeomorphism type of the pair $(X,B_t)$ in each chamber, and the sequence of surgeries along a1138path.1139 1140That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is1141contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between1142generic paths meets a short list of codimension-two events.1143 1144\begin{proposition}[Moves]\label{prop:moves}\Srf[Cerf; \DT{Thm.~8.12}]1145Let $\gamma_0,\gamma_1$ be generic paths with the same endpoints in an architecture rich enough for jet1146transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three1147kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or1148deleting a pair of consecutive surgeries of indices $\lambda,\lambda+1$ that cancel (a birth--death point of1149$f_t$ on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that1150touches zero and returns).1151\end{proposition}1152 1153The proposition is the two-parameter analogue of \cref{thm:surgery}, and is the form Cerf's theorem takes1154for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and1155that anything invariant under them, for instance the parity of the number of surgeries of each index1156modulo the cancellation of adjacent pairs, depends only on the endpoints.1157 1158\section*{Notes}1159 1160The description of how level sets change in one-parameter families is classical Morse and Cerf theory;1161see Milnor's lectures on the h-cobordism theorem. Its use for decision boundaries during training appears1162in empirical form in studies that track Betti numbers of decision regions and of data representations,1163such as \cite{NZL20}. Persistent homology of the family $\{f_t\le s\}$ in the two parameters $(t,s)$ is the1164natural computational tool, and the events of \cref{prop:moves} are the generic singularities of its1165bifiltration.1166 1167\section*{Exercises}1168 1169\begin{exercise}\label{exer:blobs}1170Verify \cref{ex:twoblobs}: find the critical points and indices of $(x_1^2-1)^2+x_2^2$ and the topology of1171the sublevel sets.1172\end{exercise}1173 1174\begin{exercise}\label{exer:d1path}1175For $d=1$, show that along a generic path the points of the decision boundary are born and die in pairs, and1176that the parity of their number is constant if $X$ is a circle.1177\end{exercise}1178 1179\begin{exercise}\label{exer:chi3}1180In $d=3$, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries,1181and what are their indices?1182\end{exercise}1183 1184\begin{exercise}\label{exer:hessgraph}1185Prove the formula for the Hessian of $\tau$ used in the proof of \cref{thm:surgery}(b).1186\end{exercise}1187%% ---- end ch/05-paths.tex ----1188%% ---- begin ch/06-loss.tex ----1189\chapter{The Loss Landscape}\label{ch:loss}1190 1191Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the1192loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the1193landscape is organized by the sublevel sets of the loss, to which the Morse theory of \DT{Ch.~8} applies1194with two caveats established in \cref{ch:symmetry} and below: the loss is rarely a Morse function, and1195parameter space is not compact.1196 1197\section{The interpolation manifold}1198 1199Let $\ev\colon\Theta\to\R^N$ be the evaluation map on $n$ data points, $N=nm$, and1200$\loss=\tfrac12|\ev-y|^2$. Write $M_y=\ev^{-1}(y)=\loss^{-1}(0)$.1201 1202\begin{theorem}[Interpolating parameters]\label{thm:interp}1203\Srf[\DT{Thm.~1.12, Thm.~4.1}; Cooper \cite{Coo21}]1204\begin{enumerate}[label=(\alph*)]1205\item For almost every $y\in\R^N$, $M_y$ is a submanifold of $\Theta$ of dimension $P-N$, possibly empty. If1206$P<N$ it is empty for almost every $y$.1207\item If $y$ is a regular value and $\theta\in M_y$, then $T_\theta M_y=\ker J(\theta)$ and1208$\nabla^2\loss(\theta)=J^{\mathsf T}J$ has rank $N$: it vanishes on $T_\theta M_y$ and is positive definite on1209its orthogonal complement.1210\item $M_y$ is non-empty for $y$ in the image of $\ev$, and that image has non-empty interior if and only if1211$J$ has rank $N$ somewhere.1212\end{enumerate}1213\end{theorem}1214 1215\begin{proof}1216(a) By \DT{Thm.~4.1} almost every $y$ is a regular value of $\ev$, and then \DT{Thm.~1.12} applies; if $P<N$1217every point of the image is a critical value. (b) $T_\theta M_y=\ker D\ev_\theta$ by \DT{Thm.~1.12}, and the1218Hessian is given by \cref{prop:hess}. (c) If $J$ has rank $N$ at $\theta$ then $\ev$ is open near $\theta$;1219conversely if the rank is everywhere less than $N$ the image has measure zero.1220\end{proof}