Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1141–1240 of 2141. Source text is displayed literally.

1141contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between1142generic paths meets a short list of codimension-two events.1143 1144\begin{proposition}[Moves]\label{prop:moves}\Srf[Cerf; \DT{Thm.~8.12}]1145Let $\gamma_0,\gamma_1$ be generic paths with the same endpoints in an architecture rich enough for jet1146transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three1147kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or1148deleting a pair of consecutive surgeries of indices $\lambda,\lambda+1$ that cancel (a birth--death point of1149$f_t$ on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that1150touches zero and returns).1151\end{proposition}1152 1153The proposition is the two-parameter analogue of \cref{thm:surgery}, and is the form Cerf's theorem takes1154for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and1155that anything invariant under them, for instance the parity of the number of surgeries of each index1156modulo the cancellation of adjacent pairs, depends only on the endpoints.1157 1158\section*{Notes}1159 1160The description of how level sets change in one-parameter families is classical Morse and Cerf theory;1161see Milnor's lectures on the h-cobordism theorem. Its use for decision boundaries during training appears1162in empirical form in studies that track Betti numbers of decision regions and of data representations,1163such as \cite{NZL20}. Persistent homology of the family $\{f_t\le s\}$ in the two parameters $(t,s)$ is the1164natural computational tool, and the events of \cref{prop:moves} are the generic singularities of its1165bifiltration.1166 1167\section*{Exercises}1168 1169\begin{exercise}\label{exer:blobs}1170Verify \cref{ex:twoblobs}: find the critical points and indices of $(x_1^2-1)^2+x_2^2$ and the topology of1171the sublevel sets.1172\end{exercise}1173 1174\begin{exercise}\label{exer:d1path}1175For $d=1$, show that along a generic path the points of the decision boundary are born and die in pairs, and1176that the parity of their number is constant if $X$ is a circle.1177\end{exercise}1178 1179\begin{exercise}\label{exer:chi3}1180In $d=3$, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries,1181and what are their indices?1182\end{exercise}1183 1184\begin{exercise}\label{exer:hessgraph}1185Prove the formula for the Hessian of $\tau$ used in the proof of \cref{thm:surgery}(b).1186\end{exercise}1187%% ---- end ch/05-paths.tex ----1188%% ---- begin ch/06-loss.tex ----1189\chapter{The Loss Landscape}\label{ch:loss}1190 1191Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the1192loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the1193landscape is organized by the sublevel sets of the loss, to which the Morse theory of \DT{Ch.~8} applies1194with two caveats established in \cref{ch:symmetry} and below: the loss is rarely a Morse function, and1195parameter space is not compact.1196 1197\section{The interpolation manifold}1198 1199Let $\ev\colon\Theta\to\R^N$ be the evaluation map on $n$ data points, $N=nm$, and1200$\loss=\tfrac12|\ev-y|^2$. Write $M_y=\ev^{-1}(y)=\loss^{-1}(0)$.1201 1202\begin{theorem}[Interpolating parameters]\label{thm:interp}1203\Srf[\DT{Thm.~1.12, Thm.~4.1}; Cooper \cite{Coo21}]1204\begin{enumerate}[label=(\alph*)]1205\item For almost every $y\in\R^N$, $M_y$ is a submanifold of $\Theta$ of dimension $P-N$, possibly empty. If1206$P<N$ it is empty for almost every $y$.1207\item If $y$ is a regular value and $\theta\in M_y$, then $T_\theta M_y=\ker J(\theta)$ and1208$\nabla^2\loss(\theta)=J^{\mathsf T}J$ has rank $N$: it vanishes on $T_\theta M_y$ and is positive definite on1209its orthogonal complement.1210\item $M_y$ is non-empty for $y$ in the image of $\ev$, and that image has non-empty interior if and only if1211$J$ has rank $N$ somewhere.1212\end{enumerate}1213\end{theorem}1214 1215\begin{proof}1216(a) By \DT{Thm.~4.1} almost every $y$ is a regular value of $\ev$, and then \DT{Thm.~1.12} applies; if $P<N$1217every point of the image is a critical value. (b) $T_\theta M_y=\ker D\ev_\theta$ by \DT{Thm.~1.12}, and the1218Hessian is given by \cref{prop:hess}. (c) If $J$ has rank $N$ at $\theta$ then $\ev$ is open near $\theta$;1219conversely if the rank is everywhere less than $N$ the image has measure zero.1220\end{proof}1221 1222Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in1223addition that for sufficiently wide networks $M_y$ is non-empty for generic data. In the overparametrized1224regime $P>N$, then, the global minima of the loss are not isolated: they form a manifold whose dimension is1225the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it.1226In the terminology of Morse theory the loss is a Morse--Bott function near $M_y$. By \cref{prop:hess} its1227normal curvature is the spectrum of the tangent kernel.1228 1229\begin{proposition}[Convergence near the interpolation manifold]\label{prop:PL}1230\Sli[Liu--Zhu--Belkin \cite{LZB22}]1231Let $U\subset\Theta$ be an open set on which the smallest eigenvalue of the tangent kernel $K=JJ^{\mathsf T}$1232is at least $\mu>0$. Then $|\nabla\loss|^2\ge2\mu\,\loss$ on $U$. A gradient flow line starting at1233$\theta_0$ satisfies $\loss(\gamma(t))\le\loss(\theta_0)e^{-2\mu t}$ and has length at most1234$\sqrt{2\loss(\theta_0)/\mu}$ for as long as it stays in $U$; if the ball of that radius about $\theta_0$1235lies in $U$, the flow line converges to a point of $M_y$.1236\end{proposition}1237 1238\begin{proof}1239$|\nabla\loss|^2=(\ev-y)^{\mathsf T}JJ^{\mathsf T}(\ev-y)\ge\mu|\ev-y|^2=2\mu\loss$. Then1240$\frac{d}{dt}\loss=-|\nabla\loss|^2\le-2\mu\loss$, and