Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 601–700 of 2141. Source text is displayed literally.

601$\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2$ with respect to $(W,b,a,c)$.602\end{exercise}603 604\begin{exercise}\label{exer:ntk}605Show that $\rk K(\theta)=\rk J(\theta)$ and that, for $n$ data points and scalar output, $K(\theta)$ is606invertible if and only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, restricted607to the data, span $\R^n$.608\end{exercise}609 610\begin{exercise}\label{exer:hess}611Derive the formula for $\nabla^2(\lambda\circ\ev)$ from the composition of $2$-jets.612\end{exercise}613 614\begin{exercise}\label{exer:linear}615For the linear family $F(\theta,x)=\langle\theta,\phi(x)\rangle$ with a fixed feature map616$\phi\colon X\to\R^P$, compute $J$, $K$, the fibres of $\ev$, and the functional dimension.617\end{exercise}618%% ---- end ch/02-backprop.tex ----619%% ---- begin ch/03-symmetry.tex ----620\chapter{Symmetries and the Moduli of Networks}\label{ch:symmetry}621 622The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand623about an architecture, since every function on parameter space that is defined in terms of the network's624input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for625networks with one hidden layer, where the theory is complete, and draws the consequences that hold in626general.627 628\section{Symmetries}629 630\begin{definition}\label{def:sym}631A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with632$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.633\end{definition}634 635For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the636$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of637$W_{i+1}$.638 639\begin{proposition}\label{prop:syms}\Scl640Every feedforward architecture has the following symmetries.641\begin{enumerate}[label=(\roman*)]642\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.643\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.644\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and645$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by646$\lambda^{-1}$.647\end{enumerate}648For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.649\end{proposition}650 651\section{One hidden layer}652 653Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with654$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.655 656\begin{definition}\label{def:irr}657The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and658$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.659\end{definition}660 661\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]662Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of663hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and664only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;665so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.666\end{theorem}667 668For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of669Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a670finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on671it.672 673\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob674The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is675injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then676\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }677Z'\bigr\}\times\R .\]678If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.679\end{proposition}680 681\begin{proof}682A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive683identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,684times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and685the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space686of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the687fundamental group of the base.688\end{proof}689 690So there are loops of networks, along which the function computed changes continuously and returns to691itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged692roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space693of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.694 695\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}696\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]697At a reducible parameter the fibre of the realization map has positive dimension:698\begin{enumerate}[label=(\roman*)]699\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;700\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;