Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 321–420 of 2141. Source text is displayed literally.
321number of data points. ``Almost every'' refers to Lebesgue measure on a Euclidean parameter space, and322``generic'' means belonging to a residual set whose complement has measure zero. An activation function323$\sigma\colon\R\to\R$ is smooth unless it is explicitly $\relu(s)=\max(s,0)$. Citations of the form324\DT{4.5} refer to numbered statements of the companion volume.325%% ---- end front.tex ----326\mainmatter327\part{Families}328%% ---- begin ch/01-families.tex ----329\chapter{Families of Maps}\label{ch:families}330 331\section{Parametrized maps}332 333\begin{definition}\label{def:family}334A \term{parametrized map} from $X$ to $Y$ is a pair $(\Theta,F)$ of a manifold $\Theta$ and a smooth map335$F\colon\Theta\times X\to Y$. Its \term{members} are the maps $f_\theta=F(\theta,-)$, and its336\term{realization map} is $\theta\mapsto f_\theta$, from $\Theta$ to $C^\infty(X,Y)$.337\end{definition}338 339By \DT{Thm.~1.6} a parametrized map is exactly a plot of the mapping object $[X,Y]$ in smooth sets, with340domain $\Theta$; the realization map is the function it induces on points. Everything in this book is a341statement about plots of mapping objects. The topologies on $C^\infty(X,Y)$ play no direct role, since342parameter spaces are finite-dimensional; what is used is the relation between properties of the total map343$F$ and properties of its members.344 345\begin{definition}\label{def:para}346$\Para$ is the bicategory whose objects are manifolds, whose morphisms $X\to Y$ are parametrized maps, and347whose $2$-cells $(\Theta',F')\Rightarrow(\Theta,F)$ are smooth maps $r\colon\Theta'\to\Theta$ with348$F'=F\circ(r\times\id_X)$. The composite of $(\Theta,F)\colon X\to Y$ and $(\Theta',G)\colon Y\to Z$ is349\[\bigl(\Theta\times\Theta',\ (\theta,\theta',x)\mapsto G(\theta',F(\theta,x))\bigr).\]350\end{definition}351 352Composition is associative up to the canonical reassociation of products, and the identity of $X$ is353parametrized by a point. The construction is due to Fong, Spivak and Tuy\'eras \cite{FST19} and is developed354in \cite{CGGWZ22}. A $2$-cell is a reparametrization: sharing weights between layers is the $2$-cell induced355by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.356 357\begin{proposition}\label{prop:plot}\Srf[\DT{Thm.~1.6}; \cite{FST19}]358Composition in $\Para$ is the internal composition359$[X,Y]\times[Y,Z]\to[X,Z]$ of smooth sets, evaluated on plots: the composite of two parametrized maps is the360plot $\Theta\times\Theta'\to[X,Y]\times[Y,Z]\to[X,Z]$. Hence $\Para$ is the bicategory of plots of the361category of manifolds enriched in smooth sets.362\end{proposition}363 364\begin{proof}365The internal composition is the map of sheaves which on plots with domain $U$ sends366$(F,G)$ to $(u,x)\mapsto G(u,F(u,x))$. Precomposing with the product plot, whose components depend on367separate factors of $U=\Theta\times\Theta'$, gives the formula of \cref{def:para}.368\end{proof}369 370\section{Architectures}371 372\begin{definition}\label{def:arch}373A \term{feedforward architecture} with widths $(n_0,\dots,n_L)$ and activation $\sigma$ is the composite in374$\Para$ of the layers375\begin{align*}376\ell_i&\colon\R^{n_{i-1}}\to\R^{n_i}, & \ell_i(W_i,b_i;x)&=\sigma(W_ix+b_i)\qquad(i<L),\\377\ell_L&\colon\R^{n_{L-1}}\to\R^{n_L}, & \ell_L(W_L,b_L;x)&=W_Lx+b_L,378\end{align*}379with $\sigma$ applied to each coordinate. Its parameter space is380$\Theta=\prod_i(\R^{n_i\times n_{i-1}}\times\R^{n_i})$, of dimension $P=\sum_in_i(n_{i-1}+1)$. The381number of hidden units is $H=n_1+\dots+n_{L-1}$.382\end{definition}383 384If $\sigma$ is smooth the family is smooth, and real-analytic if $\sigma$ is (as $\tanh$, the logistic385function and softplus are). For $\sigma=\relu$ the family is continuous and piecewise polynomial in386$(\theta,x)$, and it is not a morphism of $\Para$ as defined; statements about it are made separately.387Convolutional and attention layers are parametrized maps in the same sense, with $2$-cells expressing weight388sharing. Nothing below depends on the feedforward form except where an explicit hypothesis on the last or389first layer is made.390 391The image of the realization map, the set $\mathcal F=\{f_\theta\}$ of functions the architecture can392compute, is a subset of $C^\infty(X,Y)$. With the plots that factor locally through $\Theta$ it is a393diffeological space, the quotient of $\Theta$ by the relation $f_\theta=f_{\theta'}$. It is in general not a394manifold; \cref{ch:symmetry} describes its singularities.395 396\section{Data, loss, and training}397 398Fix inputs $x_1,\dots,x_n\in X$ and targets $y_1,\dots,y_n\in Y=\R^m$. The \term{evaluation map} is399\[\ev\colon\Theta\to Y^n=\R^{nm},\qquad\ev(\theta)=\bigl(f_\theta(x_1),\dots,f_\theta(x_n)\bigr),\]400and the squared loss is $\loss(\theta)=\tfrac12|\ev(\theta)-y|^2$. More generally a loss is401$\loss=\lambda\circ\ev$ for a smooth $\lambda\colon\R^{nm}\to\R$. A \term{training path} is a smooth curve402$\gamma\colon[0,T]\to\Theta$; gradient flow is the curve with $\dot\gamma=-\nabla\loss(\gamma)$, for the403Euclidean metric on $\Theta$. A path in $\Theta$ is a plot $[0,T]\to[X,Y]$: a one-parameter family of404maps. When every member is an embedding of a submanifold $K\subset X$ it is an isotopy of $K$ in the sense of405\DT{Def.~10.1}.406 407Three kinds of question organize the rest of the book.408 409\begin{enumerate}[label=(Q\arabic*)]410\item What does a generic member $f_\theta$ look like? (\cref{ch:decision,ch:isotopy,ch:degree})411\item What happens to the members along a generic path? (\cref{ch:paths})412\item What does the set of parameters satisfying a condition, such as $\loss(\theta)=0$, look like?413(\cref{ch:symmetry,ch:loss})414\end{enumerate}415 416In each case the answer comes from the same mechanism. A condition on a member, or on a parameter, is the417condition that a point lies in the preimage of a submanifold under a map built from $F$; the preimage is a418manifold when the map is transverse to the submanifold; and transversality of the map built from $F$ can be419checked on the architecture, usually by exhibiting one direction in parameter space, a bias, in which the420map moves freely.