Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1181–1280 of 2141. Source text is displayed literally.
1181and what are their indices?1182\end{exercise}1183 1184\begin{exercise}\label{exer:hessgraph}1185Prove the formula for the Hessian of $\tau$ used in the proof of \cref{thm:surgery}(b).1186\end{exercise}1187%% ---- end ch/05-paths.tex ----1188%% ---- begin ch/06-loss.tex ----1189\chapter{The Loss Landscape}\label{ch:loss}1190 1191Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the1192loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the1193landscape is organized by the sublevel sets of the loss, to which the Morse theory of \DT{Ch.~8} applies1194with two caveats established in \cref{ch:symmetry} and below: the loss is rarely a Morse function, and1195parameter space is not compact.1196 1197\section{The interpolation manifold}1198 1199Let $\ev\colon\Theta\to\R^N$ be the evaluation map on $n$ data points, $N=nm$, and1200$\loss=\tfrac12|\ev-y|^2$. Write $M_y=\ev^{-1}(y)=\loss^{-1}(0)$.1201 1202\begin{theorem}[Interpolating parameters]\label{thm:interp}1203\Srf[\DT{Thm.~1.12, Thm.~4.1}; Cooper \cite{Coo21}]1204\begin{enumerate}[label=(\alph*)]1205\item For almost every $y\in\R^N$, $M_y$ is a submanifold of $\Theta$ of dimension $P-N$, possibly empty. If1206$P<N$ it is empty for almost every $y$.1207\item If $y$ is a regular value and $\theta\in M_y$, then $T_\theta M_y=\ker J(\theta)$ and1208$\nabla^2\loss(\theta)=J^{\mathsf T}J$ has rank $N$: it vanishes on $T_\theta M_y$ and is positive definite on1209its orthogonal complement.1210\item $M_y$ is non-empty for $y$ in the image of $\ev$, and that image has non-empty interior if and only if1211$J$ has rank $N$ somewhere.1212\end{enumerate}1213\end{theorem}1214 1215\begin{proof}1216(a) By \DT{Thm.~4.1} almost every $y$ is a regular value of $\ev$, and then \DT{Thm.~1.12} applies; if $P<N$1217every point of the image is a critical value. (b) $T_\theta M_y=\ker D\ev_\theta$ by \DT{Thm.~1.12}, and the1218Hessian is given by \cref{prop:hess}. (c) If $J$ has rank $N$ at $\theta$ then $\ev$ is open near $\theta$;1219conversely if the rank is everywhere less than $N$ the image has measure zero.1220\end{proof}1221 1222Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in1223addition that for sufficiently wide networks $M_y$ is non-empty for generic data. In the overparametrized1224regime $P>N$, then, the global minima of the loss are not isolated: they form a manifold whose dimension is1225the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it.1226In the terminology of Morse theory the loss is a Morse--Bott function near $M_y$. By \cref{prop:hess} its1227normal curvature is the spectrum of the tangent kernel.1228 1229\begin{proposition}[Convergence near the interpolation manifold]\label{prop:PL}1230\Sli[Liu--Zhu--Belkin \cite{LZB22}]1231Let $U\subset\Theta$ be an open set on which the smallest eigenvalue of the tangent kernel $K=JJ^{\mathsf T}$1232is at least $\mu>0$. Then $|\nabla\loss|^2\ge2\mu\,\loss$ on $U$. A gradient flow line starting at1233$\theta_0$ satisfies $\loss(\gamma(t))\le\loss(\theta_0)e^{-2\mu t}$ and has length at most1234$\sqrt{2\loss(\theta_0)/\mu}$ for as long as it stays in $U$; if the ball of that radius about $\theta_0$1235lies in $U$, the flow line converges to a point of $M_y$.1236\end{proposition}1237 1238\begin{proof}1239$|\nabla\loss|^2=(\ev-y)^{\mathsf T}JJ^{\mathsf T}(\ev-y)\ge\mu|\ev-y|^2=2\mu\loss$. Then1240$\frac{d}{dt}\loss=-|\nabla\loss|^2\le-2\mu\loss$, and1241$\frac{d}{dt}\sqrt{\loss}=-|\nabla\loss|^2/(2\sqrt\loss)\le-\sqrt{\mu/2}\,|\nabla\loss|$, which integrates to1242the bound on the length $\int|\dot\gamma|=\int|\nabla\loss|$.1243\end{proof}1244 1245The inequality is the quantitative form of transversality: $\mu>0$ says that $\ev$ is a submersion on $U$1246with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the1247qualitative theory hands over to analysis.1248 1249\section{Sublevel sets and barriers}1250 1251For $c\in\R$ let $\Theta^c=\{\loss\le c\}$. Two parameters are \term{connected at level $c$} if they lie in1252the same component of $\Theta^c$, and the \term{barrier} between them is the infimum of such $c$. This is the1253notion studied empirically under the name of mode connectivity \cite{GIP18,DVSH18}.1254 1255\begin{proposition}[Barriers]\label{prop:barrier}\Srf[\DT{Cor.~8.7, Prop.~8.10}]1256Let $\loss\colon\Theta\to\R$ be a Morse function with compact sublevel sets. Then the components of1257$\Theta^c$ change only at critical values: a component is born at each local minimum, and two components1258merge at a critical point of index $1$ whose two descending directions lead to different components. The1259barrier between two local minima is the level at which their components merge; it is attained at a1260critical point of index $1$, and equals the minimum over paths joining them of the maximum of $\loss$ along1261the path.1262\end{proposition}1263 1264So barriers are the death times in the degree-zero barcode of the sublevel functor \DT{Prop.~8.10}, and the1265barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a1266generic analytic loss with weight decay, $\loss+\tfrac\varepsilon2|\theta|^2$, on an architecture without1267continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail1268(\cref{prop:notproper}).1269 1270\begin{example}[Two parameters]\label{ex:hyperbola}1271Let $F(a,b;x)=abx$ with one data point $(x,y)=(1,1)$, so $\loss=\tfrac12(ab-1)^2$. The interpolation manifold1272is the hyperbola $ab=1$, of dimension $P-N=1$, with two components exchanged by the symmetry1273$(a,b)\mapsto(-a,-b)$. The only other critical point is the origin, a saddle with $\loss=\tfrac12$ and1274Hessian eigenvalues $\pm1$. For $c<\tfrac12$ the sublevel set has two components; at $c=\tfrac12$ they merge.1275So the barrier between $(1,1)$ and $(-1,-1)$ is $\tfrac12$, attained on the straight segment between them,1276and no path does better. After applying the symmetry to one of the two points they coincide. The script1277checks the component counts on a grid.1278\end{example}1279 1280The example is the smallest instance of a general phenomenon. By \cref{prop:invariant} minima come in orbits