Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 381–480 of 2141. Source text is displayed literally.

381number of hidden units is $H=n_1+\dots+n_{L-1}$.382\end{definition}383 384If $\sigma$ is smooth the family is smooth, and real-analytic if $\sigma$ is (as $\tanh$, the logistic385function and softplus are). For $\sigma=\relu$ the family is continuous and piecewise polynomial in386$(\theta,x)$, and it is not a morphism of $\Para$ as defined; statements about it are made separately.387Convolutional and attention layers are parametrized maps in the same sense, with $2$-cells expressing weight388sharing. Nothing below depends on the feedforward form except where an explicit hypothesis on the last or389first layer is made.390 391The image of the realization map, the set $\mathcal F=\{f_\theta\}$ of functions the architecture can392compute, is a subset of $C^\infty(X,Y)$. With the plots that factor locally through $\Theta$ it is a393diffeological space, the quotient of $\Theta$ by the relation $f_\theta=f_{\theta'}$. It is in general not a394manifold; \cref{ch:symmetry} describes its singularities.395 396\section{Data, loss, and training}397 398Fix inputs $x_1,\dots,x_n\in X$ and targets $y_1,\dots,y_n\in Y=\R^m$. The \term{evaluation map} is399\[\ev\colon\Theta\to Y^n=\R^{nm},\qquad\ev(\theta)=\bigl(f_\theta(x_1),\dots,f_\theta(x_n)\bigr),\]400and the squared loss is $\loss(\theta)=\tfrac12|\ev(\theta)-y|^2$. More generally a loss is401$\loss=\lambda\circ\ev$ for a smooth $\lambda\colon\R^{nm}\to\R$. A \term{training path} is a smooth curve402$\gamma\colon[0,T]\to\Theta$; gradient flow is the curve with $\dot\gamma=-\nabla\loss(\gamma)$, for the403Euclidean metric on $\Theta$. A path in $\Theta$ is a plot $[0,T]\to[X,Y]$: a one-parameter family of404maps. When every member is an embedding of a submanifold $K\subset X$ it is an isotopy of $K$ in the sense of405\DT{Def.~10.1}.406 407Three kinds of question organize the rest of the book.408 409\begin{enumerate}[label=(Q\arabic*)]410\item What does a generic member $f_\theta$ look like? (\cref{ch:decision,ch:isotopy,ch:degree})411\item What happens to the members along a generic path? (\cref{ch:paths})412\item What does the set of parameters satisfying a condition, such as $\loss(\theta)=0$, look like?413(\cref{ch:symmetry,ch:loss})414\end{enumerate}415 416In each case the answer comes from the same mechanism. A condition on a member, or on a parameter, is the417condition that a point lies in the preimage of a submanifold under a map built from $F$; the preimage is a418manifold when the map is transverse to the submanifold; and transversality of the map built from $F$ can be419checked on the architecture, usually by exhibiting one direction in parameter space, a bias, in which the420map moves freely.421 422\begin{principle}[Richness]\label{prin:rich}\Srf[\DT{Thm.~4.5}]423Let $F\colon\Theta\times X\to Y$ be a family and $A\subset Y$ a submanifold. If $F$ is transverse to $A$, in424particular if $F$ is a submersion, then $f_\theta$ is transverse to $A$ for almost every $\theta$. A425feedforward architecture is a submersion onto $Y=\R^{n_L}$ because of the last-layer bias:426$\partial F/\partial b_L=\id$.427\end{principle}428 429This is \DT{Thm.~4.5} with its hypothesis verified. It is worth stating separately because the hypothesis is430about the family and not about function space. A statement that a property is generic among all smooth maps431(for instance, that Morse functions are dense) says nothing about a given architecture, whose realization is432a thin subset of all maps. Genericity transfers to an architecture only through a transversality property of433$F$ itself, and each use of \cref{prin:rich} below names the parameters that provide it.434 435\section{The dictionary in brief}436 437\begin{center}\small438\begin{tabular}{@{}p{0.34\textwidth}p{0.42\textwidth}l@{}}439\toprule440network notion & object of differential topology & Ch.\\441\midrule442architecture & plot of a mapping object; morphism of $\Para$ & \ref{ch:families}\\443backpropagation & functoriality of the reverse derivative & \ref{ch:backprop}\\444weight-space symmetry & group action with invariant realization & \ref{ch:symmetry}\\445decision boundary & preimage of a regular value & \ref{ch:decision}\\446change of topology in training & handle attachment on the trace & \ref{ch:paths}\\447interpolating parameters & fibre of the evaluation map & \ref{ch:loss}\\448barrier between minima & death level in the sublevel barcode & \ref{ch:loss}\\449hidden representation of data & immersion or embedding of a manifold & \ref{ch:isotopy}\\450neural ODE, residual network & diffeotopy; isotopy of embeddings & \ref{ch:isotopy}\\451invertible network & diffeomorphism of degree one & \ref{ch:degree}\\452equilibrium of an implicit layer & transverse fixed point & \ref{ch:degree}\\453\bottomrule454\end{tabular}455\end{center}456 457\section*{Notes}458 459The categorical treatment of learners as parametrized maps with update and request functions begins with460\cite{FST19}; the bicategory $\Para$ and its combination with lenses and reverse derivatives is in461\cite{CGGWZ22}. The point of view that a network is a family in the sense of singularity theory, to which462transversality applies, is implicit in much work on loss landscapes, and explicit for piecewise-linear463networks in \cite{GL22}.464 465\section*{Exercises}466 467\begin{exercise}\label{exer:count}468Compute $P$ and $H$ for the architectures with widths $(2,3,1)$, $(2,4,4,1)$ and $(d,k,1)$.469\end{exercise}470 471\begin{exercise}\label{exer:paracomp}472Verify that composition in $\Para$ is associative up to the reassociation473$(\Theta\times\Theta')\times\Theta''\cong\Theta\times(\Theta'\times\Theta'')$, and that this isomorphism474is a $2$-cell in both directions.475\end{exercise}476 477\begin{exercise}\label{exer:sharing}478Let $(\Theta,F)\colon X\to X$ be a layer. Express the weight-tied network $f_\theta\circ f_\theta$ as the479source of a $2$-cell into the composite $(\Theta\times\Theta,F\circ F)$, and show that its derivative with480respect to $\theta$ is the sum of the two partial derivatives of the untied network on the diagonal.