Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 261–360 of 2141. Source text is displayed literally.

261 262\medskip263\noindent Open access. Rights remain with the author.264}265\clearpage266 267\tableofcontents268 269\chapter{Preface}270 271A neural network architecture is a smooth map $F\colon\Theta\times X\to Y$: a family of maps from inputs to272outputs, parametrized by weights. Differential topology, in the form Hirsch gave it, is the study of what is273true of a generic member of a family of maps and of what happens along a generic path in one. The fit is274close enough that most of the qualitative questions asked about networks have answers that are corollaries275of theorems proved before 1976, once the questions are phrased as questions about families.276 277Is the decision boundary of a classifier a manifold? For almost every value of the parameters, yes, by the278parametric transversality theorem, provided the last layer has a bias. How does the boundary change during279training? By ambient isotopy, except at isolated times, at each of which it undergoes one surgery whose type280is the index of a critical point; this is Morse theory on the trace of the boundary. What is the set of281parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the282number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary283differential equation separate two linked circles? Not in dimension three, because its time-one map is284ambient isotopic to the identity; in dimension four always, because there any two embeddings of a285one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the286companion volume, and this book supplies the paragraphs.287 288The higher-categorical layer of the companion volume enters in three places. A parametrized network is a289plot of the mapping object $[X,Y]$ in smooth sets, and composition of layers is composition in a category of290parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an291architecture make the parameter space an atlas for a stack whose points are networks up to relabelling,292and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional293parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is294what one sees of them from nearby.295 296The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a297rate of convergence or a generalization error. Activation functions are assumed smooth except where stated,298and the piecewise-linear case, which needs a different transversality theory, is treated only by reference.299What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks300the examples.301 302\chapter{Introduction}303 304\section*{What is assumed}305 306The companion volume \cite{DTHC}, at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of307Hirsch's book. From machine learning, only the definition of a feedforward network and of gradient descent.308 309\section*{How the book is organized}310 311Part~I sets up families, their derivatives and their symmetries. Part~II studies level sets: of the network312in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss313in parameter space (the interpolation manifold). Part~III studies the maps themselves: when they are314embeddings, when they are isotopic, when they are invertible. Part~IV is an assessment.315\Cref{app:dictionary} gives the dictionary between the two subjects in one table.316 317\section*{Conventions}318 319Manifolds and maps are smooth. A \emph{family} is a smooth map $F\colon\Theta\times X\to Y$; we write320$f_\theta=F(\theta,-)$. $P=\dim\Theta$ is the number of parameters, $d=\dim X$ the input dimension, $n$ the321number of data points. ``Almost every'' refers to Lebesgue measure on a Euclidean parameter space, and322``generic'' means belonging to a residual set whose complement has measure zero. An activation function323$\sigma\colon\R\to\R$ is smooth unless it is explicitly $\relu(s)=\max(s,0)$. Citations of the form324\DT{4.5} refer to numbered statements of the companion volume.325%% ---- end front.tex ----326\mainmatter327\part{Families}328%% ---- begin ch/01-families.tex ----329\chapter{Families of Maps}\label{ch:families}330 331\section{Parametrized maps}332 333\begin{definition}\label{def:family}334A \term{parametrized map} from $X$ to $Y$ is a pair $(\Theta,F)$ of a manifold $\Theta$ and a smooth map335$F\colon\Theta\times X\to Y$. Its \term{members} are the maps $f_\theta=F(\theta,-)$, and its336\term{realization map} is $\theta\mapsto f_\theta$, from $\Theta$ to $C^\infty(X,Y)$.337\end{definition}338 339By \DT{Thm.~1.6} a parametrized map is exactly a plot of the mapping object $[X,Y]$ in smooth sets, with340domain $\Theta$; the realization map is the function it induces on points. Everything in this book is a341statement about plots of mapping objects. The topologies on $C^\infty(X,Y)$ play no direct role, since342parameter spaces are finite-dimensional; what is used is the relation between properties of the total map343$F$ and properties of its members.344 345\begin{definition}\label{def:para}346$\Para$ is the bicategory whose objects are manifolds, whose morphisms $X\to Y$ are parametrized maps, and347whose $2$-cells $(\Theta',F')\Rightarrow(\Theta,F)$ are smooth maps $r\colon\Theta'\to\Theta$ with348$F'=F\circ(r\times\id_X)$. The composite of $(\Theta,F)\colon X\to Y$ and $(\Theta',G)\colon Y\to Z$ is349\[\bigl(\Theta\times\Theta',\ (\theta,\theta',x)\mapsto G(\theta',F(\theta,x))\bigr).\]350\end{definition}351 352Composition is associative up to the canonical reassociation of products, and the identity of $X$ is353parametrized by a point. The construction is due to Fong, Spivak and Tuy\'eras \cite{FST19} and is developed354in \cite{CGGWZ22}. A $2$-cell is a reparametrization: sharing weights between layers is the $2$-cell induced355by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.356 357\begin{proposition}\label{prop:plot}\Srf[\DT{Thm.~1.6}; \cite{FST19}]358Composition in $\Para$ is the internal composition359$[X,Y]\times[Y,Z]\to[X,Z]$ of smooth sets, evaluated on plots: the composite of two parametrized maps is the360plot $\Theta\times\Theta'\to[X,Y]\times[Y,Z]\to[X,Z]$. Hence $\Para$ is the bicategory of plots of the