Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 281–380 of 2141. Source text is displayed literally.
281parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the282number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary283differential equation separate two linked circles? Not in dimension three, because its time-one map is284ambient isotopic to the identity; in dimension four always, because there any two embeddings of a285one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the286companion volume, and this book supplies the paragraphs.287 288The higher-categorical layer of the companion volume enters in three places. A parametrized network is a289plot of the mapping object $[X,Y]$ in smooth sets, and composition of layers is composition in a category of290parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an291architecture make the parameter space an atlas for a stack whose points are networks up to relabelling,292and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional293parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is294what one sees of them from nearby.295 296The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a297rate of convergence or a generalization error. Activation functions are assumed smooth except where stated,298and the piecewise-linear case, which needs a different transversality theory, is treated only by reference.299What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks300the examples.301 302\chapter{Introduction}303 304\section*{What is assumed}305 306The companion volume \cite{DTHC}, at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of307Hirsch's book. From machine learning, only the definition of a feedforward network and of gradient descent.308 309\section*{How the book is organized}310 311Part~I sets up families, their derivatives and their symmetries. Part~II studies level sets: of the network312in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss313in parameter space (the interpolation manifold). Part~III studies the maps themselves: when they are314embeddings, when they are isotopic, when they are invertible. Part~IV is an assessment.315\Cref{app:dictionary} gives the dictionary between the two subjects in one table.316 317\section*{Conventions}318 319Manifolds and maps are smooth. A \emph{family} is a smooth map $F\colon\Theta\times X\to Y$; we write320$f_\theta=F(\theta,-)$. $P=\dim\Theta$ is the number of parameters, $d=\dim X$ the input dimension, $n$ the321number of data points. ``Almost every'' refers to Lebesgue measure on a Euclidean parameter space, and322``generic'' means belonging to a residual set whose complement has measure zero. An activation function323$\sigma\colon\R\to\R$ is smooth unless it is explicitly $\relu(s)=\max(s,0)$. Citations of the form324\DT{4.5} refer to numbered statements of the companion volume.325%% ---- end front.tex ----326\mainmatter327\part{Families}328%% ---- begin ch/01-families.tex ----329\chapter{Families of Maps}\label{ch:families}330 331\section{Parametrized maps}332 333\begin{definition}\label{def:family}334A \term{parametrized map} from $X$ to $Y$ is a pair $(\Theta,F)$ of a manifold $\Theta$ and a smooth map335$F\colon\Theta\times X\to Y$. Its \term{members} are the maps $f_\theta=F(\theta,-)$, and its336\term{realization map} is $\theta\mapsto f_\theta$, from $\Theta$ to $C^\infty(X,Y)$.337\end{definition}338 339By \DT{Thm.~1.6} a parametrized map is exactly a plot of the mapping object $[X,Y]$ in smooth sets, with340domain $\Theta$; the realization map is the function it induces on points. Everything in this book is a341statement about plots of mapping objects. The topologies on $C^\infty(X,Y)$ play no direct role, since342parameter spaces are finite-dimensional; what is used is the relation between properties of the total map343$F$ and properties of its members.344 345\begin{definition}\label{def:para}346$\Para$ is the bicategory whose objects are manifolds, whose morphisms $X\to Y$ are parametrized maps, and347whose $2$-cells $(\Theta',F')\Rightarrow(\Theta,F)$ are smooth maps $r\colon\Theta'\to\Theta$ with348$F'=F\circ(r\times\id_X)$. The composite of $(\Theta,F)\colon X\to Y$ and $(\Theta',G)\colon Y\to Z$ is349\[\bigl(\Theta\times\Theta',\ (\theta,\theta',x)\mapsto G(\theta',F(\theta,x))\bigr).\]350\end{definition}351 352Composition is associative up to the canonical reassociation of products, and the identity of $X$ is353parametrized by a point. The construction is due to Fong, Spivak and Tuy\'eras \cite{FST19} and is developed354in \cite{CGGWZ22}. A $2$-cell is a reparametrization: sharing weights between layers is the $2$-cell induced355by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.356 357\begin{proposition}\label{prop:plot}\Srf[\DT{Thm.~1.6}; \cite{FST19}]358Composition in $\Para$ is the internal composition359$[X,Y]\times[Y,Z]\to[X,Z]$ of smooth sets, evaluated on plots: the composite of two parametrized maps is the360plot $\Theta\times\Theta'\to[X,Y]\times[Y,Z]\to[X,Z]$. Hence $\Para$ is the bicategory of plots of the361category of manifolds enriched in smooth sets.362\end{proposition}363 364\begin{proof}365The internal composition is the map of sheaves which on plots with domain $U$ sends366$(F,G)$ to $(u,x)\mapsto G(u,F(u,x))$. Precomposing with the product plot, whose components depend on367separate factors of $U=\Theta\times\Theta'$, gives the formula of \cref{def:para}.368\end{proof}369 370\section{Architectures}371 372\begin{definition}\label{def:arch}373A \term{feedforward architecture} with widths $(n_0,\dots,n_L)$ and activation $\sigma$ is the composite in374$\Para$ of the layers375\begin{align*}376\ell_i&\colon\R^{n_{i-1}}\to\R^{n_i}, & \ell_i(W_i,b_i;x)&=\sigma(W_ix+b_i)\qquad(i<L),\\377\ell_L&\colon\R^{n_{L-1}}\to\R^{n_L}, & \ell_L(W_L,b_L;x)&=W_Lx+b_L,378\end{align*}379with $\sigma$ applied to each coordinate. Its parameter space is380$\Theta=\prod_i(\R^{n_i\times n_{i-1}}\times\R^{n_i})$, of dimension $P=\sum_in_i(n_{i-1}+1)$. The