Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1161–1260 of 2141. Source text is displayed literally.
1161see Milnor's lectures on the h-cobordism theorem. Its use for decision boundaries during training appears1162in empirical form in studies that track Betti numbers of decision regions and of data representations,1163such as \cite{NZL20}. Persistent homology of the family $\{f_t\le s\}$ in the two parameters $(t,s)$ is the1164natural computational tool, and the events of \cref{prop:moves} are the generic singularities of its1165bifiltration.1166 1167\section*{Exercises}1168 1169\begin{exercise}\label{exer:blobs}1170Verify \cref{ex:twoblobs}: find the critical points and indices of $(x_1^2-1)^2+x_2^2$ and the topology of1171the sublevel sets.1172\end{exercise}1173 1174\begin{exercise}\label{exer:d1path}1175For $d=1$, show that along a generic path the points of the decision boundary are born and die in pairs, and1176that the parity of their number is constant if $X$ is a circle.1177\end{exercise}1178 1179\begin{exercise}\label{exer:chi3}1180In $d=3$, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries,1181and what are their indices?1182\end{exercise}1183 1184\begin{exercise}\label{exer:hessgraph}1185Prove the formula for the Hessian of $\tau$ used in the proof of \cref{thm:surgery}(b).1186\end{exercise}1187%% ---- end ch/05-paths.tex ----1188%% ---- begin ch/06-loss.tex ----1189\chapter{The Loss Landscape}\label{ch:loss}1190 1191Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the1192loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the1193landscape is organized by the sublevel sets of the loss, to which the Morse theory of \DT{Ch.~8} applies1194with two caveats established in \cref{ch:symmetry} and below: the loss is rarely a Morse function, and1195parameter space is not compact.1196 1197\section{The interpolation manifold}1198 1199Let $\ev\colon\Theta\to\R^N$ be the evaluation map on $n$ data points, $N=nm$, and1200$\loss=\tfrac12|\ev-y|^2$. Write $M_y=\ev^{-1}(y)=\loss^{-1}(0)$.1201 1202\begin{theorem}[Interpolating parameters]\label{thm:interp}1203\Srf[\DT{Thm.~1.12, Thm.~4.1}; Cooper \cite{Coo21}]1204\begin{enumerate}[label=(\alph*)]1205\item For almost every $y\in\R^N$, $M_y$ is a submanifold of $\Theta$ of dimension $P-N$, possibly empty. If1206$P<N$ it is empty for almost every $y$.1207\item If $y$ is a regular value and $\theta\in M_y$, then $T_\theta M_y=\ker J(\theta)$ and1208$\nabla^2\loss(\theta)=J^{\mathsf T}J$ has rank $N$: it vanishes on $T_\theta M_y$ and is positive definite on1209its orthogonal complement.1210\item $M_y$ is non-empty for $y$ in the image of $\ev$, and that image has non-empty interior if and only if1211$J$ has rank $N$ somewhere.1212\end{enumerate}1213\end{theorem}1214 1215\begin{proof}1216(a) By \DT{Thm.~4.1} almost every $y$ is a regular value of $\ev$, and then \DT{Thm.~1.12} applies; if $P<N$1217every point of the image is a critical value. (b) $T_\theta M_y=\ker D\ev_\theta$ by \DT{Thm.~1.12}, and the1218Hessian is given by \cref{prop:hess}. (c) If $J$ has rank $N$ at $\theta$ then $\ev$ is open near $\theta$;1219conversely if the rank is everywhere less than $N$ the image has measure zero.1220\end{proof}1221 1222Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in1223addition that for sufficiently wide networks $M_y$ is non-empty for generic data. In the overparametrized1224regime $P>N$, then, the global minima of the loss are not isolated: they form a manifold whose dimension is1225the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it.1226In the terminology of Morse theory the loss is a Morse--Bott function near $M_y$. By \cref{prop:hess} its1227normal curvature is the spectrum of the tangent kernel.1228 1229\begin{proposition}[Convergence near the interpolation manifold]\label{prop:PL}1230\Sli[Liu--Zhu--Belkin \cite{LZB22}]1231Let $U\subset\Theta$ be an open set on which the smallest eigenvalue of the tangent kernel $K=JJ^{\mathsf T}$1232is at least $\mu>0$. Then $|\nabla\loss|^2\ge2\mu\,\loss$ on $U$. A gradient flow line starting at1233$\theta_0$ satisfies $\loss(\gamma(t))\le\loss(\theta_0)e^{-2\mu t}$ and has length at most1234$\sqrt{2\loss(\theta_0)/\mu}$ for as long as it stays in $U$; if the ball of that radius about $\theta_0$1235lies in $U$, the flow line converges to a point of $M_y$.1236\end{proposition}1237 1238\begin{proof}1239$|\nabla\loss|^2=(\ev-y)^{\mathsf T}JJ^{\mathsf T}(\ev-y)\ge\mu|\ev-y|^2=2\mu\loss$. Then1240$\frac{d}{dt}\loss=-|\nabla\loss|^2\le-2\mu\loss$, and1241$\frac{d}{dt}\sqrt{\loss}=-|\nabla\loss|^2/(2\sqrt\loss)\le-\sqrt{\mu/2}\,|\nabla\loss|$, which integrates to1242the bound on the length $\int|\dot\gamma|=\int|\nabla\loss|$.1243\end{proof}1244 1245The inequality is the quantitative form of transversality: $\mu>0$ says that $\ev$ is a submersion on $U$1246with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the1247qualitative theory hands over to analysis.1248 1249\section{Sublevel sets and barriers}1250 1251For $c\in\R$ let $\Theta^c=\{\loss\le c\}$. Two parameters are \term{connected at level $c$} if they lie in1252the same component of $\Theta^c$, and the \term{barrier} between them is the infimum of such $c$. This is the1253notion studied empirically under the name of mode connectivity \cite{GIP18,DVSH18}.1254 1255\begin{proposition}[Barriers]\label{prop:barrier}\Srf[\DT{Cor.~8.7, Prop.~8.10}]1256Let $\loss\colon\Theta\to\R$ be a Morse function with compact sublevel sets. Then the components of1257$\Theta^c$ change only at critical values: a component is born at each local minimum, and two components1258merge at a critical point of index $1$ whose two descending directions lead to different components. The1259barrier between two local minima is the level at which their components merge; it is attained at a1260critical point of index $1$, and equals the minimum over paths joining them of the maximum of $\loss$ along