Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 541–640 of 2141. Source text is displayed literally.
541So the tangent kernel is the Riemannian metric that the Euclidean metric of $\Theta$ induces, by pushing542forward along $\ev$, on the cotangent directions of output space; where $J$ is onto, $K$ is invertible and543the outputs follow a gradient flow for the metric $K^{-1}$.544 545\section{Rank}546 547The rank of $J$ is the first invariant of a parameter. It is lower semicontinuous, and for analytic548architectures its generic value is attained almost everywhere.549 550\begin{proposition}\label{prop:rank}\Scl551Let $F$ be real-analytic in $\theta$ on a connected $\Theta$, and let $r$ be the maximum of $\rk J(\theta)$.552Then $\{\theta:\rk J(\theta)=r\}$ is open and dense and its complement has measure zero. On it the553fibres of $\ev$ are submanifolds of dimension $P-r$, and $\ev$ is locally a submersion onto an554$r$-dimensional submanifold of $\R^{nm}$.555\end{proposition}556 557\begin{proof}558The complement is the common zero set of the $r\times r$ minors of $J$, which are analytic and not all559identically zero. The zero set of a non-zero analytic function on a connected domain is closed with empty560interior and measure zero. The rest is the constant rank theorem.561\end{proof}562 563\begin{definition}\label{def:fdim}564The \term{functional dimension} of the architecture at $\theta$ is the supremum over all finite sets of565inputs of $\rk J(\theta)$: the rank of the differential of the realization map \cite{GLMW22}.566\end{definition}567 568The functional dimension is at most $P$, and \cref{ch:symmetry} shows when it is smaller. The strata569$\{\rk J=s\}$ for $s<r$ are where the fibres of $\ev$ jump in dimension. They contain the parameters at570which hidden units are redundant, and they are responsible for most of what is singular in the loss571landscape (\cref{ch:loss}).572 573\section{Second derivatives}574 575By the composition law for $2$-jets \DT{Prop.~2.9}, the Hessian of a composite $\lambda\circ\ev$ is576\[\nabla^2\loss(\theta)=J^{\mathsf T}\,\nabla^2\lambda\,J+\sum_k\frac{\partial\lambda}{\partial u_k}\,577\nabla^2\ev_k(\theta),\]578a Gauss--Newton term and a term weighted by the gradient of $\lambda$ in output space.579 580\begin{proposition}\label{prop:hess}\Scl581At a parameter where $\nabla\lambda(\ev(\theta))=0$, in particular at a zero of the squared loss,582$\nabla^2\loss=J^{\mathsf T}\nabla^2\lambda\,J$. For the squared loss this is $J^{\mathsf T}J$, positive583semidefinite of rank $\rk J$, with kernel $\ker J$.584\end{proposition}585 586The non-zero eigenvalues of $J^{\mathsf T}J$ and of the tangent kernel $JJ^{\mathsf T}$ coincide. So at an587interpolating parameter the curvature of the loss in the directions normal to the solution set is the588spectrum of the tangent kernel.589 590\section*{Notes}591 592Reverse-mode differentiation has been rediscovered several times; Griewank's history is the standard593reference. The compositional formulation is from \cite{FST19} and \cite{CGGWZ22}, and the axiomatics of594reverse derivatives from \cite{CCG20}. The tangent kernel was introduced in \cite{JGH18}. Functional595dimension for $\relu$ networks is studied in \cite{GLMW22}.596 597\section*{Exercises}598 599\begin{exercise}\label{exer:bp}600Write out the lens of a layer $\ell(W,b;x)=\sigma(Wx+b)$, and compose two of them to obtain the gradient of601$\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2$ with respect to $(W,b,a,c)$.602\end{exercise}603 604\begin{exercise}\label{exer:ntk}605Show that $\rk K(\theta)=\rk J(\theta)$ and that, for $n$ data points and scalar output, $K(\theta)$ is606invertible if and only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, restricted607to the data, span $\R^n$.608\end{exercise}609 610\begin{exercise}\label{exer:hess}611Derive the formula for $\nabla^2(\lambda\circ\ev)$ from the composition of $2$-jets.612\end{exercise}613 614\begin{exercise}\label{exer:linear}615For the linear family $F(\theta,x)=\langle\theta,\phi(x)\rangle$ with a fixed feature map616$\phi\colon X\to\R^P$, compute $J$, $K$, the fibres of $\ev$, and the functional dimension.617\end{exercise}618%% ---- end ch/02-backprop.tex ----619%% ---- begin ch/03-symmetry.tex ----620\chapter{Symmetries and the Moduli of Networks}\label{ch:symmetry}621 622The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand623about an architecture, since every function on parameter space that is defined in terms of the network's624input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for625networks with one hidden layer, where the theory is complete, and draws the consequences that hold in626general.627 628\section{Symmetries}629 630\begin{definition}\label{def:sym}631A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with632$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.633\end{definition}634 635For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the636$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of637$W_{i+1}$.638 639\begin{proposition}\label{prop:syms}\Scl640Every feedforward architecture has the following symmetries.