Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 481–580 of 2141. Source text is displayed literally.

481\end{exercise}482 483\begin{exercise}\label{exer:notrich}484Give an example of a family $F\colon\R\times\R\to\R$ no member of which is a Morse function, although Morse485functions are dense in $C^\infty(\R,\R)$. Which hypothesis of \cref{prin:rich} fails for the relevant jet486condition?487\end{exercise}488%% ---- end ch/01-families.tex ----489%% ---- begin ch/02-backprop.tex ----490\chapter{Derivatives}\label{ch:backprop}491 492Two derivatives of a family matter: with respect to the input, which governs the geometry of each member, and493with respect to the parameters, which governs training. Both are instances of the tangent functor of494\DT{Ch.~2}, and the algorithm that computes the second efficiently is the functoriality of its transpose.495 496\section{Backpropagation}497 498Recall from \DT{Def.~2.10} the category $\cat{Lens}$ and from \DT{Prop.~2.11} the functor499$f\mapsto(f,R[f])$, $R[f](x,w)=(Df_x)^{\mathsf T}w$, from Euclidean spaces to lenses. For a parametrized map500$F\colon\Theta\times X\to Y$ the reverse derivative has two components,501\[R[F](\theta,x,w)=\bigl(R_\Theta[F](\theta,x,w),\ R_X[F](\theta,x,w)\bigr)502=\bigl((D_\theta F)^{\mathsf T}w,\ (D_xF)^{\mathsf T}w\bigr),\]503the first being the update direction for the parameters and the second the covector passed back to the504previous layer.505 506\begin{theorem}[Backpropagation is a functor]\label{thm:backprop}507\Sli[Fong--Spivak--Tuy\'eras \cite{FST19}; Cruttwell et al.\ \cite{CGGWZ22}]508The assignment $(\Theta,F)\mapsto(\Theta,(F,R[F]))$ is a functor of bicategories from $\Para$ over Euclidean509spaces to parametrized lenses. In particular, for a composite of layers $F=\ell_L\circ\dots\circ\ell_1$ the510component $R_{\Theta_i}[F]$ is obtained by composing the lenses of the layers: a forward pass computing the511activations $x_i=\ell_i(\theta_i,x_{i-1})$, and a backward pass computing $w_{i-1}=R_X[\ell_i](\theta_i,512x_{i-1},w_i)$ and emitting $R_\Theta[\ell_i](\theta_i,x_{i-1},w_i)$.513\end{theorem}514 515\begin{proof}516Composition in $\Para$ is composition of maps after taking products with identities, and $R$ is a517product-preserving functor to lenses by \DT{Prop.~2.11}.518\end{proof}519 520The gradient of a loss $\loss=\lambda\circ\ev$ is $R[\ev](\theta,\nabla\lambda)$. The theorem says the521cost of the gradient is that of one forward and one backward pass, whatever the number of parameters, which522is the practical content of reverse-mode differentiation. The script implements both modes for a small523network and checks them against each other and against finite differences.524 525\section{The Jacobian of the evaluation map}526 527\begin{definition}\label{def:jac}528For data $x_1,\dots,x_n$ the \term{parameter Jacobian} is $J(\theta)=D\ev_\theta$, an $nm\times P$ matrix.529The \term{tangent kernel} is the $nm\times nm$ matrix $K(\theta)=J(\theta)J(\theta)^{\mathsf T}$.530\end{definition}531 532\begin{proposition}\label{prop:ntk}\Scl[Jacot--Gabriel--Hongler \cite{JGH18}]533$K(\theta)$ is symmetric positive semidefinite of rank $\rk J(\theta)$. Along gradient flow of534$\loss=\lambda\circ\ev$ the outputs $u(t)=\ev(\gamma(t))$ satisfy $\dot u=-K(\gamma)\nabla\lambda(u)$.535\end{proposition}536 537\begin{proof}538$\dot u=J\dot\gamma=-J\nabla\loss=-JJ^{\mathsf T}\nabla\lambda$.539\end{proof}540 541So the tangent kernel is the Riemannian metric that the Euclidean metric of $\Theta$ induces, by pushing542forward along $\ev$, on the cotangent directions of output space; where $J$ is onto, $K$ is invertible and543the outputs follow a gradient flow for the metric $K^{-1}$.544 545\section{Rank}546 547The rank of $J$ is the first invariant of a parameter. It is lower semicontinuous, and for analytic548architectures its generic value is attained almost everywhere.549 550\begin{proposition}\label{prop:rank}\Scl551Let $F$ be real-analytic in $\theta$ on a connected $\Theta$, and let $r$ be the maximum of $\rk J(\theta)$.552Then $\{\theta:\rk J(\theta)=r\}$ is open and dense and its complement has measure zero. On it the553fibres of $\ev$ are submanifolds of dimension $P-r$, and $\ev$ is locally a submersion onto an554$r$-dimensional submanifold of $\R^{nm}$.555\end{proposition}556 557\begin{proof}558The complement is the common zero set of the $r\times r$ minors of $J$, which are analytic and not all559identically zero. The zero set of a non-zero analytic function on a connected domain is closed with empty560interior and measure zero. The rest is the constant rank theorem.561\end{proof}562 563\begin{definition}\label{def:fdim}564The \term{functional dimension} of the architecture at $\theta$ is the supremum over all finite sets of565inputs of $\rk J(\theta)$: the rank of the differential of the realization map \cite{GLMW22}.566\end{definition}567 568The functional dimension is at most $P$, and \cref{ch:symmetry} shows when it is smaller. The strata569$\{\rk J=s\}$ for $s<r$ are where the fibres of $\ev$ jump in dimension. They contain the parameters at570which hidden units are redundant, and they are responsible for most of what is singular in the loss571landscape (\cref{ch:loss}).572 573\section{Second derivatives}574 575By the composition law for $2$-jets \DT{Prop.~2.9}, the Hessian of a composite $\lambda\circ\ev$ is576\[\nabla^2\loss(\theta)=J^{\mathsf T}\,\nabla^2\lambda\,J+\sum_k\frac{\partial\lambda}{\partial u_k}\,577\nabla^2\ev_k(\theta),\]578a Gauss--Newton term and a term weighted by the gradient of $\lambda$ in output space.579 580\begin{proposition}\label{prop:hess}\Scl