Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 561–660 of 2141. Source text is displayed literally.

561\end{proof}562 563\begin{definition}\label{def:fdim}564The \term{functional dimension} of the architecture at $\theta$ is the supremum over all finite sets of565inputs of $\rk J(\theta)$: the rank of the differential of the realization map \cite{GLMW22}.566\end{definition}567 568The functional dimension is at most $P$, and \cref{ch:symmetry} shows when it is smaller. The strata569$\{\rk J=s\}$ for $s<r$ are where the fibres of $\ev$ jump in dimension. They contain the parameters at570which hidden units are redundant, and they are responsible for most of what is singular in the loss571landscape (\cref{ch:loss}).572 573\section{Second derivatives}574 575By the composition law for $2$-jets \DT{Prop.~2.9}, the Hessian of a composite $\lambda\circ\ev$ is576\[\nabla^2\loss(\theta)=J^{\mathsf T}\,\nabla^2\lambda\,J+\sum_k\frac{\partial\lambda}{\partial u_k}\,577\nabla^2\ev_k(\theta),\]578a Gauss--Newton term and a term weighted by the gradient of $\lambda$ in output space.579 580\begin{proposition}\label{prop:hess}\Scl581At a parameter where $\nabla\lambda(\ev(\theta))=0$, in particular at a zero of the squared loss,582$\nabla^2\loss=J^{\mathsf T}\nabla^2\lambda\,J$. For the squared loss this is $J^{\mathsf T}J$, positive583semidefinite of rank $\rk J$, with kernel $\ker J$.584\end{proposition}585 586The non-zero eigenvalues of $J^{\mathsf T}J$ and of the tangent kernel $JJ^{\mathsf T}$ coincide. So at an587interpolating parameter the curvature of the loss in the directions normal to the solution set is the588spectrum of the tangent kernel.589 590\section*{Notes}591 592Reverse-mode differentiation has been rediscovered several times; Griewank's history is the standard593reference. The compositional formulation is from \cite{FST19} and \cite{CGGWZ22}, and the axiomatics of594reverse derivatives from \cite{CCG20}. The tangent kernel was introduced in \cite{JGH18}. Functional595dimension for $\relu$ networks is studied in \cite{GLMW22}.596 597\section*{Exercises}598 599\begin{exercise}\label{exer:bp}600Write out the lens of a layer $\ell(W,b;x)=\sigma(Wx+b)$, and compose two of them to obtain the gradient of601$\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2$ with respect to $(W,b,a,c)$.602\end{exercise}603 604\begin{exercise}\label{exer:ntk}605Show that $\rk K(\theta)=\rk J(\theta)$ and that, for $n$ data points and scalar output, $K(\theta)$ is606invertible if and only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, restricted607to the data, span $\R^n$.608\end{exercise}609 610\begin{exercise}\label{exer:hess}611Derive the formula for $\nabla^2(\lambda\circ\ev)$ from the composition of $2$-jets.612\end{exercise}613 614\begin{exercise}\label{exer:linear}615For the linear family $F(\theta,x)=\langle\theta,\phi(x)\rangle$ with a fixed feature map616$\phi\colon X\to\R^P$, compute $J$, $K$, the fibres of $\ev$, and the functional dimension.617\end{exercise}618%% ---- end ch/02-backprop.tex ----619%% ---- begin ch/03-symmetry.tex ----620\chapter{Symmetries and the Moduli of Networks}\label{ch:symmetry}621 622The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand623about an architecture, since every function on parameter space that is defined in terms of the network's624input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for625networks with one hidden layer, where the theory is complete, and draws the consequences that hold in626general.627 628\section{Symmetries}629 630\begin{definition}\label{def:sym}631A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with632$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.633\end{definition}634 635For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the636$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of637$W_{i+1}$.638 639\begin{proposition}\label{prop:syms}\Scl640Every feedforward architecture has the following symmetries.641\begin{enumerate}[label=(\roman*)]642\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.643\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.644\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and645$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by646$\lambda^{-1}$.647\end{enumerate}648For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.649\end{proposition}650 651\section{One hidden layer}652 653Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with654$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.655 656\begin{definition}\label{def:irr}657The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and658$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.659\end{definition}660