Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1221–1320 of 2141. Source text is displayed literally.

1221 1222Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in1223addition that for sufficiently wide networks $M_y$ is non-empty for generic data. In the overparametrized1224regime $P>N$, then, the global minima of the loss are not isolated: they form a manifold whose dimension is1225the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it.1226In the terminology of Morse theory the loss is a Morse--Bott function near $M_y$. By \cref{prop:hess} its1227normal curvature is the spectrum of the tangent kernel.1228 1229\begin{proposition}[Convergence near the interpolation manifold]\label{prop:PL}1230\Sli[Liu--Zhu--Belkin \cite{LZB22}]1231Let $U\subset\Theta$ be an open set on which the smallest eigenvalue of the tangent kernel $K=JJ^{\mathsf T}$1232is at least $\mu>0$. Then $|\nabla\loss|^2\ge2\mu\,\loss$ on $U$. A gradient flow line starting at1233$\theta_0$ satisfies $\loss(\gamma(t))\le\loss(\theta_0)e^{-2\mu t}$ and has length at most1234$\sqrt{2\loss(\theta_0)/\mu}$ for as long as it stays in $U$; if the ball of that radius about $\theta_0$1235lies in $U$, the flow line converges to a point of $M_y$.1236\end{proposition}1237 1238\begin{proof}1239$|\nabla\loss|^2=(\ev-y)^{\mathsf T}JJ^{\mathsf T}(\ev-y)\ge\mu|\ev-y|^2=2\mu\loss$. Then1240$\frac{d}{dt}\loss=-|\nabla\loss|^2\le-2\mu\loss$, and1241$\frac{d}{dt}\sqrt{\loss}=-|\nabla\loss|^2/(2\sqrt\loss)\le-\sqrt{\mu/2}\,|\nabla\loss|$, which integrates to1242the bound on the length $\int|\dot\gamma|=\int|\nabla\loss|$.1243\end{proof}1244 1245The inequality is the quantitative form of transversality: $\mu>0$ says that $\ev$ is a submersion on $U$1246with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the1247qualitative theory hands over to analysis.1248 1249\section{Sublevel sets and barriers}1250 1251For $c\in\R$ let $\Theta^c=\{\loss\le c\}$. Two parameters are \term{connected at level $c$} if they lie in1252the same component of $\Theta^c$, and the \term{barrier} between them is the infimum of such $c$. This is the1253notion studied empirically under the name of mode connectivity \cite{GIP18,DVSH18}.1254 1255\begin{proposition}[Barriers]\label{prop:barrier}\Srf[\DT{Cor.~8.7, Prop.~8.10}]1256Let $\loss\colon\Theta\to\R$ be a Morse function with compact sublevel sets. Then the components of1257$\Theta^c$ change only at critical values: a component is born at each local minimum, and two components1258merge at a critical point of index $1$ whose two descending directions lead to different components. The1259barrier between two local minima is the level at which their components merge; it is attained at a1260critical point of index $1$, and equals the minimum over paths joining them of the maximum of $\loss$ along1261the path.1262\end{proposition}1263 1264So barriers are the death times in the degree-zero barcode of the sublevel functor \DT{Prop.~8.10}, and the1265barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a1266generic analytic loss with weight decay, $\loss+\tfrac\varepsilon2|\theta|^2$, on an architecture without1267continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail1268(\cref{prop:notproper}).1269 1270\begin{example}[Two parameters]\label{ex:hyperbola}1271Let $F(a,b;x)=abx$ with one data point $(x,y)=(1,1)$, so $\loss=\tfrac12(ab-1)^2$. The interpolation manifold1272is the hyperbola $ab=1$, of dimension $P-N=1$, with two components exchanged by the symmetry1273$(a,b)\mapsto(-a,-b)$. The only other critical point is the origin, a saddle with $\loss=\tfrac12$ and1274Hessian eigenvalues $\pm1$. For $c<\tfrac12$ the sublevel set has two components; at $c=\tfrac12$ they merge.1275So the barrier between $(1,1)$ and $(-1,-1)$ is $\tfrac12$, attained on the straight segment between them,1276and no path does better. After applying the symmetry to one of the two points they coincide. The script1277checks the component counts on a grid.1278\end{example}1279 1280The example is the smallest instance of a general phenomenon. By \cref{prop:invariant} minima come in orbits1281of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a1282barrier. The barrier between two trained networks should therefore be measured after choosing1283representatives of their orbits, which is the content of the conjecture of Entezari et al.\ that barriers1284largely disappear once permutations are accounted for \cite{EKSN22}. In the language of \cref{ch:symmetry},1285the relevant sublevel sets are those of the function induced on the quotient $\Theta/G$.1286 1287Two results from the literature describe the overparametrized case. For a class of deep networks with1288piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen1289proved that every sublevel set of the loss is connected \cite{Ngu19}. And \c{S}im\c{s}ek et al.\ describe the1290critical manifolds that the reducible strata of \cref{prop:fibres} contribute, and show how adding hidden1291units connects the $|G|$ copies of a minimum \cite{SGJ21}.1292 1293\section{Why the loss is rarely a Morse function}1294 1295Three mechanisms make the loss degenerate, and they are independent.1296 1297\begin{enumerate}[label=(\roman*)]1298\item \emph{Overparametrization.} If $P>N$ the zero-loss minima form the manifold $M_y$ of positive1299dimension (\cref{thm:interp}).1300\item \emph{Continuous symmetry.} For homogeneous activations every critical point has a Hessian of nullity1301at least $H$ (\cref{cor:notmorse}).1302\item \emph{Reducible strata.} A critical point of a smaller network gives a positive-dimensional family of1303critical points of a larger one, along the fibres of \cref{prop:fibres}; these are typically saddles1304\cite{FA00}.1305\end{enumerate}1306 1307In each case the degeneracy is of Morse--Bott type on an open dense part of the critical set, and the handle1308theorem \DT{Thm.~8.6} has a Morse--Bott version in which a critical manifold contributes a bundle of cells1309over itself. But the density of Morse functions \DT{Cor.~4.8} is of no direct use: a generic perturbation of1310the loss as a function on $\Theta$ is not the loss of any data set, and perturbing the data preserves all1311three mechanisms. This is the point of \cref{prin:rich} again. Genericity must be established within the1312family one actually has.1313 1314\section{Critical points at infinity}1315 1316\begin{proposition}\label{prop:notproper}\Sob1317For architectures with a bounded activation the loss is not proper, and its infimum need not be attained.1318For $F(w;x)=\tanh(wx)$ with the single data point $(1,1)$, $\loss(w)=\tfrac12(\tanh w-1)^2$ has no critical1319point, its infimum $0$ is not attained, and every sublevel set $\Theta^c$ with $0<c<2$ is a half-line.1320\end{proposition}