Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1721–1820 of 2141. Source text is displayed literally.
1721(\cref{prop:walls}); the surgery description of training with its Euler characteristic count1722(\cref{thm:surgery,prop:chi}) and its moves (\cref{prop:moves}); the dimension count $2m+2$ for1723augmentation in terms of the data (\cref{thm:augment,cor:separate}); and the parity of the number of1724equilibria (\cref{thm:deq}). Each is a short corollary and is offered as such.1725 1726The higher-categorical layer was used as language more than as a tool. The bicategory $\Para$ and the lens1727functor organize \cref{ch:families,ch:backprop}; the quotient by symmetries is a stack and not a space for1728homogeneous activations (\cref{ch:symmetry}); and a degenerate level set is a derived manifold with a1729cobordism class (\cref{ex:cross}). The theorems that do the work are those of \cref{tab:used}.1730 1731\section{Limits}1732 1733\emph{Smoothness.} The standing assumption excludes $\relu$. For piecewise-linear networks the analogue of1734\cref{thm:boundary} is the theorem of Grigsby and Lindsey \cite{GL22}; analogues of1735\cref{thm:chambers,thm:surgery} would require stratified Morse theory, and are not proved here.1736 1737\emph{Genericity within the family.} ``Almost every parameter'' is with respect to Lebesgue measure on1738$\Theta$, so the statements apply to parameters drawn from a distribution with a density, as at a random1739initialization. They do not apply to distinguished parameters. In particular, if $g$ is an orthogonal1740symmetry of the architecture, the gradient of an invariant loss is equivariant, and so the fixed-point set1741of $g$ is invariant under gradient flow: a network initialized with two identical hidden units keeps them1742identical for ever, and stays in a reducible stratum of measure zero where none of the generic statements1743need hold.1744 1745\emph{Paths.} \Cref{thm:surgery} describes generic paths. A training trajectory is a particular path, the1746solution of a differential equation, and we have proved nothing about whether it is generic. The honest1747statement is \cref{prop:genericpath}: arbitrarily close to it there is a path with the stated behaviour.1748 1749\emph{Compactness.} Input domains are assumed compact for the isotopy statements, and parameter space is1750never compact, which restricts the use of Morse theory for losses (\cref{prop:notproper}).1751 1752\emph{No rates.} With the single exception of \cref{prop:PL}, the statements are qualitative. They say that1753a set has measure zero and not how small its neighbourhoods are; that an isotopy exists and not how large1754its velocity must be; that a boundary of a given topology is realizable and not at what width. Each of these1755has a quantitative counterpart in the literature that the present methods do not reach.1756 1757\emph{No learning theory.} Nothing here concerns generalization.1758 1759\section{Directions}1760 1761\begin{problem}[Piecewise-linear version]\label{prob:PL}\Spg1762Prove the analogues of \cref{thm:chambers,thm:surgery} for $\relu$ networks that are generic and transversal1763in the sense of \cite{GL22}, with handle attachments replaced by the elementary moves of stratified Morse1764theory on the canonical polyhedral complex.1765\end{problem}1766 1767\begin{problem}[Quantitative discriminant]\label{prob:quant}\Spg1768For a given architecture and initialization distribution, estimate the measure of the1769$\varepsilon$-neighbourhood of the discriminant $\Delta$ of \cref{ch:decision}, and hence the expected number1770of surgeries along a path of given length. \Cref{prop:walls} suggests that $\Delta$ is a hypersurface and1771that the answer is governed by its volume.1772\end{problem}1773 1774\begin{problem}[The stack of networks]\label{prob:stack}\Spg1775For homogeneous activations, describe the quotient stack $[\Theta/G]$ of \cref{ch:symmetry} and the critical1776locus of a loss on it as a derived object, so that \cref{thm:interp} and the degeneracies listed in1777\cref{ch:loss} become statements about a single space with a virtual dimension.1778\end{problem}1779 1780Together with \cref{prob:calculus}, these are the places where the methods of the companion volume, and in1781particular its higher-categorical parts (derived manifolds, stacks, the calculus of embeddings), would be1782doing more than providing a vocabulary.1783%% ---- end ch/09-coda.tex ----1784\appendix1785%% ---- begin ch/A-dictionary.tex ----1786\chapter{Dictionary}\label{app:dictionary}1787 1788{\small1789\begin{longtable}{@{}p{0.25\textwidth}p{0.36\textwidth}p{0.29\textwidth}@{}}1790\toprule1791Network notion & Differential topology & Where\\1792\midrule1793\endhead1794architecture $F\colon\Theta\times X\to Y$ & plot of the mapping object $[X,Y]$; morphism of $\Para$ & \cref{def:family,prop:plot}\\1795composition of layers & composition of plots & \cref{def:para}\\1796weight sharing, freezing & $2$-cells of $\Para$ & \cref{exer:sharing}\\1797forward-mode derivative & tangent functor & \DT{Prop.~2.1}\\1798backpropagation & reverse derivative as a functor to lenses & \cref{thm:backprop}\\1799tangent kernel & Gram matrix $JJ^{\mathsf T}$ of the evaluation map & \cref{prop:ntk}\\1800functional dimension & rank of the realization map & \cref{def:fdim,prop:rank}\\1801permutation, sign, scaling symmetry & group of invertible $2$-cells & \cref{prop:syms}\\1802redundant hidden unit & reducible parameter; positive-dimensional fibre & \cref{prop:fibres}\\1803space of networks & quotient manifold, or stack & \cref{prop:moduli}\\1804decision boundary & preimage of a regular value & \cref{thm:boundary}\\1805multi-class boundaries & preimage of a stratification & \cref{prop:multiclass}\\1806robust topology of a classifier & chamber of the discriminant complement & \cref{thm:chambers}\\1807change of boundary topology & surgery; critical point of the trace & \cref{thm:surgery}\\1808training run, seen in input space & embedded cobordism with a Morse function & \cref{thm:trace}\\1809two runs with the same endpoints & Cerf moves & \cref{prop:moves}\\1810interpolating parameters & fibre of the evaluation map over a regular value & \cref{thm:interp}\\1811sharpness at a minimum & normal Hessian $J^{\mathsf T}J$ & \cref{prop:hess}\\1812loss barrier, mode connectivity & death level in the degree-zero barcode & \cref{prop:barrier}\\1813saturation, diverging weights & failure of properness & \cref{prop:notproper}\\1814hidden representation & immersion or embedding of the data manifold & \cref{prop:firstlayer}\\1815minimal width & invariance of domain & \cref{thm:obstruction}\\1816neural ODE & diffeotopy & \cref{prop:node}\\1817augmented neural ODE & isotopy of embeddings in the stable range & \cref{thm:augment}\\1818residual block with small branch & diffeomorphism isotopic to the identity & \cref{prop:iresnet}\\1819normalizing flow & diffeomorphism; pushforward of measures & \cref{prop:mismatch}\\1820equilibrium layer & transverse fixed point; Lefschetz count & \cref{thm:deq}\\