Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1681–1780 of 2141. Source text is displayed literally.

1681%% ---- begin ch/09-coda.tex ----1682\chapter{What Was Used, and What Is Missing}\label{ch:coda}1683 1684\section{The inputs}1685 1686\Cref{tab:used} lists the main statements of this book with the theorems of the companion volume from which1687they follow. Every entry in the right-hand column was known by the mid-1970s.1688 1689\begin{table}[ht]\centering\small1690\begin{tabular}{@{}p{0.47\textwidth}p{0.43\textwidth}@{}}1691\toprule1692statement here & from \cite{DTHC}\\1693\midrule1694backpropagation composes (\cref{thm:backprop}) & reverse chain rule, Prop.~2.11\\1695generic decision boundaries are hypersurfaces (\cref{thm:boundary}) & Sard, Thm.~4.1; regular values, Thm.~1.12\\1696chambers and walls (\cref{thm:chambers,prop:walls}) & fibrewise lemma 4.4; Ehresmann, Thm.~8.5; isotopy extension, Thm.~10.3\\1697every hypersurface is realized (\cref{thm:realize}) & tubular neighbourhoods, Thm.~6.2; openness, Thm.~3.2\\1698surgery along paths (\cref{thm:trace,thm:surgery}) & neat preimages, Thm.~1.17; handles, Thm.~8.6\\1699moves between paths (\cref{prop:moves}) & Cerf, Thm.~8.12\\1700interpolation manifold (\cref{thm:interp}) & regular values, Thm.~1.12; Sard, Thm.~4.1\\1701barriers (\cref{prop:barrier}) & sublevel functor, Cor.~8.7, Prop.~8.10\\1702first layer embeds (\cref{prop:firstlayer}) & parametric transversality, Thm.~4.5\\1703neural ODEs are diffeotopies (\cref{prop:node}) & flows, Thm.~10.2\\1704augmentation (\cref{thm:augment}) & isotopy in the stable range, Thm.~10.9; Thm.~10.3\\1705odd number of equilibria (\cref{thm:deq}) & degree, Thm.~7.2; Thm.~4.5\\1706\bottomrule1707\end{tabular}1708\caption{What is used.}\label{tab:used}1709\end{table}1710 1711\section{What is and is not new}1712 1713Nothing in this book is a new theorem of differential topology, and most of its statements about networks1714have appeared in the machine learning literature in some special case, with the references given in the1715text: generic smoothness of decision boundaries, the manifold of global minima, the obstruction to neural1716ODEs and its removal by augmentation, the mismatch of supports for flows. What the book adds is the uniform1717derivation, with the hypothesis that makes each statement true stated as a transversality property of the1718architecture (\cref{prin:rich}), and a small number of statements tagged as observations that we have not1719found in this form: the description of the moduli of irreducible one-hidden-layer networks as a1720configuration space, with its monodromy (\cref{prop:moduli}); the wall structure of the discriminant1721(\cref{prop:walls}); the surgery description of training with its Euler characteristic count1722(\cref{thm:surgery,prop:chi}) and its moves (\cref{prop:moves}); the dimension count $2m+2$ for1723augmentation in terms of the data (\cref{thm:augment,cor:separate}); and the parity of the number of1724equilibria (\cref{thm:deq}). Each is a short corollary and is offered as such.1725 1726The higher-categorical layer was used as language more than as a tool. The bicategory $\Para$ and the lens1727functor organize \cref{ch:families,ch:backprop}; the quotient by symmetries is a stack and not a space for1728homogeneous activations (\cref{ch:symmetry}); and a degenerate level set is a derived manifold with a1729cobordism class (\cref{ex:cross}). The theorems that do the work are those of \cref{tab:used}.1730 1731\section{Limits}1732 1733\emph{Smoothness.} The standing assumption excludes $\relu$. For piecewise-linear networks the analogue of1734\cref{thm:boundary} is the theorem of Grigsby and Lindsey \cite{GL22}; analogues of1735\cref{thm:chambers,thm:surgery} would require stratified Morse theory, and are not proved here.1736 1737\emph{Genericity within the family.} ``Almost every parameter'' is with respect to Lebesgue measure on1738$\Theta$, so the statements apply to parameters drawn from a distribution with a density, as at a random1739initialization. They do not apply to distinguished parameters. In particular, if $g$ is an orthogonal1740symmetry of the architecture, the gradient of an invariant loss is equivariant, and so the fixed-point set1741of $g$ is invariant under gradient flow: a network initialized with two identical hidden units keeps them1742identical for ever, and stays in a reducible stratum of measure zero where none of the generic statements1743need hold.1744 1745\emph{Paths.} \Cref{thm:surgery} describes generic paths. A training trajectory is a particular path, the1746solution of a differential equation, and we have proved nothing about whether it is generic. The honest1747statement is \cref{prop:genericpath}: arbitrarily close to it there is a path with the stated behaviour.1748 1749\emph{Compactness.} Input domains are assumed compact for the isotopy statements, and parameter space is1750never compact, which restricts the use of Morse theory for losses (\cref{prop:notproper}).1751 1752\emph{No rates.} With the single exception of \cref{prop:PL}, the statements are qualitative. They say that1753a set has measure zero and not how small its neighbourhoods are; that an isotopy exists and not how large1754its velocity must be; that a boundary of a given topology is realizable and not at what width. Each of these1755has a quantitative counterpart in the literature that the present methods do not reach.1756 1757\emph{No learning theory.} Nothing here concerns generalization.1758 1759\section{Directions}1760 1761\begin{problem}[Piecewise-linear version]\label{prob:PL}\Spg1762Prove the analogues of \cref{thm:chambers,thm:surgery} for $\relu$ networks that are generic and transversal1763in the sense of \cite{GL22}, with handle attachments replaced by the elementary moves of stratified Morse1764theory on the canonical polyhedral complex.1765\end{problem}1766 1767\begin{problem}[Quantitative discriminant]\label{prob:quant}\Spg1768For a given architecture and initialization distribution, estimate the measure of the1769$\varepsilon$-neighbourhood of the discriminant $\Delta$ of \cref{ch:decision}, and hence the expected number1770of surgeries along a path of given length. \Cref{prop:walls} suggests that $\Delta$ is a hypersurface and1771that the answer is governed by its volume.1772\end{problem}1773 1774\begin{problem}[The stack of networks]\label{prob:stack}\Spg1775For homogeneous activations, describe the quotient stack $[\Theta/G]$ of \cref{ch:symmetry} and the critical1776locus of a loss on it as a derived object, so that \cref{thm:interp} and the degeneracies listed in1777\cref{ch:loss} become statements about a single space with a virtual dimension.1778\end{problem}1779 1780Together with \cref{prob:calculus}, these are the places where the methods of the companion volume, and in