Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1281–1380 of 2141. Source text is displayed literally.

1281of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a1282barrier. The barrier between two trained networks should therefore be measured after choosing1283representatives of their orbits, which is the content of the conjecture of Entezari et al.\ that barriers1284largely disappear once permutations are accounted for \cite{EKSN22}. In the language of \cref{ch:symmetry},1285the relevant sublevel sets are those of the function induced on the quotient $\Theta/G$.1286 1287Two results from the literature describe the overparametrized case. For a class of deep networks with1288piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen1289proved that every sublevel set of the loss is connected \cite{Ngu19}. And \c{S}im\c{s}ek et al.\ describe the1290critical manifolds that the reducible strata of \cref{prop:fibres} contribute, and show how adding hidden1291units connects the $|G|$ copies of a minimum \cite{SGJ21}.1292 1293\section{Why the loss is rarely a Morse function}1294 1295Three mechanisms make the loss degenerate, and they are independent.1296 1297\begin{enumerate}[label=(\roman*)]1298\item \emph{Overparametrization.} If $P>N$ the zero-loss minima form the manifold $M_y$ of positive1299dimension (\cref{thm:interp}).1300\item \emph{Continuous symmetry.} For homogeneous activations every critical point has a Hessian of nullity1301at least $H$ (\cref{cor:notmorse}).1302\item \emph{Reducible strata.} A critical point of a smaller network gives a positive-dimensional family of1303critical points of a larger one, along the fibres of \cref{prop:fibres}; these are typically saddles1304\cite{FA00}.1305\end{enumerate}1306 1307In each case the degeneracy is of Morse--Bott type on an open dense part of the critical set, and the handle1308theorem \DT{Thm.~8.6} has a Morse--Bott version in which a critical manifold contributes a bundle of cells1309over itself. But the density of Morse functions \DT{Cor.~4.8} is of no direct use: a generic perturbation of1310the loss as a function on $\Theta$ is not the loss of any data set, and perturbing the data preserves all1311three mechanisms. This is the point of \cref{prin:rich} again. Genericity must be established within the1312family one actually has.1313 1314\section{Critical points at infinity}1315 1316\begin{proposition}\label{prop:notproper}\Sob1317For architectures with a bounded activation the loss is not proper, and its infimum need not be attained.1318For $F(w;x)=\tanh(wx)$ with the single data point $(1,1)$, $\loss(w)=\tfrac12(\tanh w-1)^2$ has no critical1319point, its infimum $0$ is not attained, and every sublevel set $\Theta^c$ with $0<c<2$ is a half-line.1320\end{proposition}1321 1322\begin{proof}1323$\loss'(w)=(\tanh w-1)\operatorname{sech}^2w<0$ for all $w$.1324\end{proof}1325 1326So the topology of sublevel sets can change with no critical point to account for it, the change being1327carried off to infinity, and the regular interval theorem \DT{Thm.~8.4}, which needs compactness, does not1328apply. Any use of Morse theory for losses must either restore compactness (weight decay, a bounded1329parameter domain) or account for these ends explicitly, as one does for the Palais--Smale condition in1330infinite dimensions.1331 1332\section*{Notes}1333 1334The manifold of global minima was described by Cooper \cite{Coo21}; the Polyak--\L{}ojasiewicz condition in1335this setting is due to Liu, Zhu and Belkin. Mode connectivity was observed by Garipov et al.\ and Draxler et1336al., and analysed for wide networks by Freeman and Bruna and by Nguyen; permutation alignment is studied in1337\cite{EKSN22}. For the singular structure of the landscape at reducible parameters see \cite{FA00,SGJ21} and1338Watanabe's theory \cite{Wat09}.1339 1340\section*{Exercises}1341 1342\begin{exercise}\label{exer:linreg}1343For the linear family of \cref{exer:linear} with $P>n$ and features in general position, describe $M_y$ and1344the Hessian of the loss.1345\end{exercise}1346 1347\begin{exercise}\label{exer:hyperbola}1348Verify \cref{ex:hyperbola}: find all critical points of $\tfrac12(ab-1)^2$, their Hessians, and the1349components of the sublevel sets.1350\end{exercise}1351 1352\begin{exercise}\label{exer:PL}1353In the setting of \cref{prop:PL}, show that the limit point of the flow line lies within1354$\sqrt{2\loss(\theta_0)/\mu}$ of $\theta_0$.1355\end{exercise}1356 1357\begin{exercise}\label{exer:deeplin}1358For $F(a_1,\dots,a_k;x)=a_1\cdots a_kx$ with data $(1,1)$, show that $M_y$ has $2^{k-1}$ components and1359dimension $k-1$, and that the origin is a degenerate critical point for $k\ge3$.1360\end{exercise}1361%% ---- end ch/06-loss.tex ----1362\part{Maps}1363%% ---- begin ch/07-isotopy.tex ----1364\chapter{Embeddings, Isotopy, and Depth}\label{ch:isotopy}1365 1366The previous three chapters studied level sets. This one studies the maps themselves, and in particular1367what a network does to a data set regarded as a subset of input space. The standing assumption, usually1368called the manifold hypothesis \cite{FMN16}, is that the data lie on or near a compact submanifold1369$K\subset\R^D$ of dimension $m$ much smaller than $D$, with the classes occupying disjoint closed subsets1370$K_+,K_-$. A hidden layer of width $k$ is a map $h_\theta\colon\R^D\to\R^k$, and the question is what1371$h_\theta(K)$ can be.1372 1373\section{The first layer generically embeds}1374 1375\begin{proposition}\label{prop:firstlayer}\Srf[Whitney; \DT{Thm.~4.5, Cor.~4.8, Prop.~3.12}]1376Let $K\subset\R^D$ be a compact submanifold of dimension $m$, let $k\ge2m+1$, and let $\sigma$ be a1377diffeomorphism of $\R$ onto an open interval. Then for almost every $W\in\R^{k\times D}$ and every $b$, the1378map $x\mapsto\sigma(Wx+b)$ restricted to $K$ is an embedding. The set of such $(W,b)$ is open.1379\end{proposition}1380