Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1241–1340 of 2141. Source text is displayed literally.

1241$\frac{d}{dt}\sqrt{\loss}=-|\nabla\loss|^2/(2\sqrt\loss)\le-\sqrt{\mu/2}\,|\nabla\loss|$, which integrates to1242the bound on the length $\int|\dot\gamma|=\int|\nabla\loss|$.1243\end{proof}1244 1245The inequality is the quantitative form of transversality: $\mu>0$ says that $\ev$ is a submersion on $U$1246with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the1247qualitative theory hands over to analysis.1248 1249\section{Sublevel sets and barriers}1250 1251For $c\in\R$ let $\Theta^c=\{\loss\le c\}$. Two parameters are \term{connected at level $c$} if they lie in1252the same component of $\Theta^c$, and the \term{barrier} between them is the infimum of such $c$. This is the1253notion studied empirically under the name of mode connectivity \cite{GIP18,DVSH18}.1254 1255\begin{proposition}[Barriers]\label{prop:barrier}\Srf[\DT{Cor.~8.7, Prop.~8.10}]1256Let $\loss\colon\Theta\to\R$ be a Morse function with compact sublevel sets. Then the components of1257$\Theta^c$ change only at critical values: a component is born at each local minimum, and two components1258merge at a critical point of index $1$ whose two descending directions lead to different components. The1259barrier between two local minima is the level at which their components merge; it is attained at a1260critical point of index $1$, and equals the minimum over paths joining them of the maximum of $\loss$ along1261the path.1262\end{proposition}1263 1264So barriers are the death times in the degree-zero barcode of the sublevel functor \DT{Prop.~8.10}, and the1265barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a1266generic analytic loss with weight decay, $\loss+\tfrac\varepsilon2|\theta|^2$, on an architecture without1267continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail1268(\cref{prop:notproper}).1269 1270\begin{example}[Two parameters]\label{ex:hyperbola}1271Let $F(a,b;x)=abx$ with one data point $(x,y)=(1,1)$, so $\loss=\tfrac12(ab-1)^2$. The interpolation manifold1272is the hyperbola $ab=1$, of dimension $P-N=1$, with two components exchanged by the symmetry1273$(a,b)\mapsto(-a,-b)$. The only other critical point is the origin, a saddle with $\loss=\tfrac12$ and1274Hessian eigenvalues $\pm1$. For $c<\tfrac12$ the sublevel set has two components; at $c=\tfrac12$ they merge.1275So the barrier between $(1,1)$ and $(-1,-1)$ is $\tfrac12$, attained on the straight segment between them,1276and no path does better. After applying the symmetry to one of the two points they coincide. The script1277checks the component counts on a grid.1278\end{example}1279 1280The example is the smallest instance of a general phenomenon. By \cref{prop:invariant} minima come in orbits1281of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a1282barrier. The barrier between two trained networks should therefore be measured after choosing1283representatives of their orbits, which is the content of the conjecture of Entezari et al.\ that barriers1284largely disappear once permutations are accounted for \cite{EKSN22}. In the language of \cref{ch:symmetry},1285the relevant sublevel sets are those of the function induced on the quotient $\Theta/G$.1286 1287Two results from the literature describe the overparametrized case. For a class of deep networks with1288piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen1289proved that every sublevel set of the loss is connected \cite{Ngu19}. And \c{S}im\c{s}ek et al.\ describe the1290critical manifolds that the reducible strata of \cref{prop:fibres} contribute, and show how adding hidden1291units connects the $|G|$ copies of a minimum \cite{SGJ21}.1292 1293\section{Why the loss is rarely a Morse function}1294 1295Three mechanisms make the loss degenerate, and they are independent.1296 1297\begin{enumerate}[label=(\roman*)]1298\item \emph{Overparametrization.} If $P>N$ the zero-loss minima form the manifold $M_y$ of positive1299dimension (\cref{thm:interp}).1300\item \emph{Continuous symmetry.} For homogeneous activations every critical point has a Hessian of nullity1301at least $H$ (\cref{cor:notmorse}).1302\item \emph{Reducible strata.} A critical point of a smaller network gives a positive-dimensional family of1303critical points of a larger one, along the fibres of \cref{prop:fibres}; these are typically saddles1304\cite{FA00}.1305\end{enumerate}1306 1307In each case the degeneracy is of Morse--Bott type on an open dense part of the critical set, and the handle1308theorem \DT{Thm.~8.6} has a Morse--Bott version in which a critical manifold contributes a bundle of cells1309over itself. But the density of Morse functions \DT{Cor.~4.8} is of no direct use: a generic perturbation of1310the loss as a function on $\Theta$ is not the loss of any data set, and perturbing the data preserves all1311three mechanisms. This is the point of \cref{prin:rich} again. Genericity must be established within the1312family one actually has.1313 1314\section{Critical points at infinity}1315 1316\begin{proposition}\label{prop:notproper}\Sob1317For architectures with a bounded activation the loss is not proper, and its infimum need not be attained.1318For $F(w;x)=\tanh(wx)$ with the single data point $(1,1)$, $\loss(w)=\tfrac12(\tanh w-1)^2$ has no critical1319point, its infimum $0$ is not attained, and every sublevel set $\Theta^c$ with $0<c<2$ is a half-line.1320\end{proposition}1321 1322\begin{proof}1323$\loss'(w)=(\tanh w-1)\operatorname{sech}^2w<0$ for all $w$.1324\end{proof}1325 1326So the topology of sublevel sets can change with no critical point to account for it, the change being1327carried off to infinity, and the regular interval theorem \DT{Thm.~8.4}, which needs compactness, does not1328apply. Any use of Morse theory for losses must either restore compactness (weight decay, a bounded1329parameter domain) or account for these ends explicitly, as one does for the Palais--Smale condition in1330infinite dimensions.1331 1332\section*{Notes}1333 1334The manifold of global minima was described by Cooper \cite{Coo21}; the Polyak--\L{}ojasiewicz condition in1335this setting is due to Liu, Zhu and Belkin. Mode connectivity was observed by Garipov et al.\ and Draxler et1336al., and analysed for wide networks by Freeman and Bruna and by Nguyen; permutation alignment is studied in1337\cite{EKSN22}. For the singular structure of the landscape at reducible parameters see \cite{FA00,SGJ21} and1338Watanabe's theory \cite{Wat09}.1339 1340\section*{Exercises}