Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 421–520 of 2141. Source text is displayed literally.
421 422\begin{principle}[Richness]\label{prin:rich}\Srf[\DT{Thm.~4.5}]423Let $F\colon\Theta\times X\to Y$ be a family and $A\subset Y$ a submanifold. If $F$ is transverse to $A$, in424particular if $F$ is a submersion, then $f_\theta$ is transverse to $A$ for almost every $\theta$. A425feedforward architecture is a submersion onto $Y=\R^{n_L}$ because of the last-layer bias:426$\partial F/\partial b_L=\id$.427\end{principle}428 429This is \DT{Thm.~4.5} with its hypothesis verified. It is worth stating separately because the hypothesis is430about the family and not about function space. A statement that a property is generic among all smooth maps431(for instance, that Morse functions are dense) says nothing about a given architecture, whose realization is432a thin subset of all maps. Genericity transfers to an architecture only through a transversality property of433$F$ itself, and each use of \cref{prin:rich} below names the parameters that provide it.434 435\section{The dictionary in brief}436 437\begin{center}\small438\begin{tabular}{@{}p{0.34\textwidth}p{0.42\textwidth}l@{}}439\toprule440network notion & object of differential topology & Ch.\\441\midrule442architecture & plot of a mapping object; morphism of $\Para$ & \ref{ch:families}\\443backpropagation & functoriality of the reverse derivative & \ref{ch:backprop}\\444weight-space symmetry & group action with invariant realization & \ref{ch:symmetry}\\445decision boundary & preimage of a regular value & \ref{ch:decision}\\446change of topology in training & handle attachment on the trace & \ref{ch:paths}\\447interpolating parameters & fibre of the evaluation map & \ref{ch:loss}\\448barrier between minima & death level in the sublevel barcode & \ref{ch:loss}\\449hidden representation of data & immersion or embedding of a manifold & \ref{ch:isotopy}\\450neural ODE, residual network & diffeotopy; isotopy of embeddings & \ref{ch:isotopy}\\451invertible network & diffeomorphism of degree one & \ref{ch:degree}\\452equilibrium of an implicit layer & transverse fixed point & \ref{ch:degree}\\453\bottomrule454\end{tabular}455\end{center}456 457\section*{Notes}458 459The categorical treatment of learners as parametrized maps with update and request functions begins with460\cite{FST19}; the bicategory $\Para$ and its combination with lenses and reverse derivatives is in461\cite{CGGWZ22}. The point of view that a network is a family in the sense of singularity theory, to which462transversality applies, is implicit in much work on loss landscapes, and explicit for piecewise-linear463networks in \cite{GL22}.464 465\section*{Exercises}466 467\begin{exercise}\label{exer:count}468Compute $P$ and $H$ for the architectures with widths $(2,3,1)$, $(2,4,4,1)$ and $(d,k,1)$.469\end{exercise}470 471\begin{exercise}\label{exer:paracomp}472Verify that composition in $\Para$ is associative up to the reassociation473$(\Theta\times\Theta')\times\Theta''\cong\Theta\times(\Theta'\times\Theta'')$, and that this isomorphism474is a $2$-cell in both directions.475\end{exercise}476 477\begin{exercise}\label{exer:sharing}478Let $(\Theta,F)\colon X\to X$ be a layer. Express the weight-tied network $f_\theta\circ f_\theta$ as the479source of a $2$-cell into the composite $(\Theta\times\Theta,F\circ F)$, and show that its derivative with480respect to $\theta$ is the sum of the two partial derivatives of the untied network on the diagonal.481\end{exercise}482 483\begin{exercise}\label{exer:notrich}484Give an example of a family $F\colon\R\times\R\to\R$ no member of which is a Morse function, although Morse485functions are dense in $C^\infty(\R,\R)$. Which hypothesis of \cref{prin:rich} fails for the relevant jet486condition?487\end{exercise}488%% ---- end ch/01-families.tex ----489%% ---- begin ch/02-backprop.tex ----490\chapter{Derivatives}\label{ch:backprop}491 492Two derivatives of a family matter: with respect to the input, which governs the geometry of each member, and493with respect to the parameters, which governs training. Both are instances of the tangent functor of494\DT{Ch.~2}, and the algorithm that computes the second efficiently is the functoriality of its transpose.495 496\section{Backpropagation}497 498Recall from \DT{Def.~2.10} the category $\cat{Lens}$ and from \DT{Prop.~2.11} the functor499$f\mapsto(f,R[f])$, $R[f](x,w)=(Df_x)^{\mathsf T}w$, from Euclidean spaces to lenses. For a parametrized map500$F\colon\Theta\times X\to Y$ the reverse derivative has two components,501\[R[F](\theta,x,w)=\bigl(R_\Theta[F](\theta,x,w),\ R_X[F](\theta,x,w)\bigr)502=\bigl((D_\theta F)^{\mathsf T}w,\ (D_xF)^{\mathsf T}w\bigr),\]503the first being the update direction for the parameters and the second the covector passed back to the504previous layer.505 506\begin{theorem}[Backpropagation is a functor]\label{thm:backprop}507\Sli[Fong--Spivak--Tuy\'eras \cite{FST19}; Cruttwell et al.\ \cite{CGGWZ22}]508The assignment $(\Theta,F)\mapsto(\Theta,(F,R[F]))$ is a functor of bicategories from $\Para$ over Euclidean509spaces to parametrized lenses. In particular, for a composite of layers $F=\ell_L\circ\dots\circ\ell_1$ the510component $R_{\Theta_i}[F]$ is obtained by composing the lenses of the layers: a forward pass computing the511activations $x_i=\ell_i(\theta_i,x_{i-1})$, and a backward pass computing $w_{i-1}=R_X[\ell_i](\theta_i,512x_{i-1},w_i)$ and emitting $R_\Theta[\ell_i](\theta_i,x_{i-1},w_i)$.513\end{theorem}514 515\begin{proof}516Composition in $\Para$ is composition of maps after taking products with identities, and $R$ is a517product-preserving functor to lenses by \DT{Prop.~2.11}.518\end{proof}519 520The gradient of a loss $\loss=\lambda\circ\ev$ is $R[\ev](\theta,\nabla\lambda)$. The theorem says the