Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 581–680 of 2141. Source text is displayed literally.
581At a parameter where $\nabla\lambda(\ev(\theta))=0$, in particular at a zero of the squared loss,582$\nabla^2\loss=J^{\mathsf T}\nabla^2\lambda\,J$. For the squared loss this is $J^{\mathsf T}J$, positive583semidefinite of rank $\rk J$, with kernel $\ker J$.584\end{proposition}585 586The non-zero eigenvalues of $J^{\mathsf T}J$ and of the tangent kernel $JJ^{\mathsf T}$ coincide. So at an587interpolating parameter the curvature of the loss in the directions normal to the solution set is the588spectrum of the tangent kernel.589 590\section*{Notes}591 592Reverse-mode differentiation has been rediscovered several times; Griewank's history is the standard593reference. The compositional formulation is from \cite{FST19} and \cite{CGGWZ22}, and the axiomatics of594reverse derivatives from \cite{CCG20}. The tangent kernel was introduced in \cite{JGH18}. Functional595dimension for $\relu$ networks is studied in \cite{GLMW22}.596 597\section*{Exercises}598 599\begin{exercise}\label{exer:bp}600Write out the lens of a layer $\ell(W,b;x)=\sigma(Wx+b)$, and compose two of them to obtain the gradient of601$\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2$ with respect to $(W,b,a,c)$.602\end{exercise}603 604\begin{exercise}\label{exer:ntk}605Show that $\rk K(\theta)=\rk J(\theta)$ and that, for $n$ data points and scalar output, $K(\theta)$ is606invertible if and only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, restricted607to the data, span $\R^n$.608\end{exercise}609 610\begin{exercise}\label{exer:hess}611Derive the formula for $\nabla^2(\lambda\circ\ev)$ from the composition of $2$-jets.612\end{exercise}613 614\begin{exercise}\label{exer:linear}615For the linear family $F(\theta,x)=\langle\theta,\phi(x)\rangle$ with a fixed feature map616$\phi\colon X\to\R^P$, compute $J$, $K$, the fibres of $\ev$, and the functional dimension.617\end{exercise}618%% ---- end ch/02-backprop.tex ----619%% ---- begin ch/03-symmetry.tex ----620\chapter{Symmetries and the Moduli of Networks}\label{ch:symmetry}621 622The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand623about an architecture, since every function on parameter space that is defined in terms of the network's624input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for625networks with one hidden layer, where the theory is complete, and draws the consequences that hold in626general.627 628\section{Symmetries}629 630\begin{definition}\label{def:sym}631A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with632$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.633\end{definition}634 635For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the636$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of637$W_{i+1}$.638 639\begin{proposition}\label{prop:syms}\Scl640Every feedforward architecture has the following symmetries.641\begin{enumerate}[label=(\roman*)]642\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.643\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.644\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and645$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by646$\lambda^{-1}$.647\end{enumerate}648For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.649\end{proposition}650 651\section{One hidden layer}652 653Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with654$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.655 656\begin{definition}\label{def:irr}657The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and658$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.659\end{definition}660 661\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]662Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of663hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and664only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;665so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.666\end{theorem}667 668For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of669Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a670finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on671it.672 673\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob674The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is675injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then676\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }677Z'\bigr\}\times\R .\]678If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.679\end{proposition}680