Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 621–720 of 2141. Source text is displayed literally.

621 622The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand623about an architecture, since every function on parameter space that is defined in terms of the network's624input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for625networks with one hidden layer, where the theory is complete, and draws the consequences that hold in626general.627 628\section{Symmetries}629 630\begin{definition}\label{def:sym}631A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with632$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.633\end{definition}634 635For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the636$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of637$W_{i+1}$.638 639\begin{proposition}\label{prop:syms}\Scl640Every feedforward architecture has the following symmetries.641\begin{enumerate}[label=(\roman*)]642\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.643\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.644\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and645$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by646$\lambda^{-1}$.647\end{enumerate}648For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.649\end{proposition}650 651\section{One hidden layer}652 653Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with654$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.655 656\begin{definition}\label{def:irr}657The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and658$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.659\end{definition}660 661\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]662Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of663hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and664only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;665so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.666\end{theorem}667 668For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of669Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a670finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on671it.672 673\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob674The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is675injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then676\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }677Z'\bigr\}\times\R .\]678If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.679\end{proposition}680 681\begin{proof}682A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive683identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,684times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and685the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space686of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the687fundamental group of the base.688\end{proof}689 690So there are loops of networks, along which the function computed changes continuously and returns to691itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged692roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space693of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.694 695\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}696\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]697At a reducible parameter the fibre of the realization map has positive dimension:698\begin{enumerate}[label=(\roman*)]699\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;700\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;701\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates702$(a_j,b_j,c)$.703\end{enumerate}704\end{proposition}705 706Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of707parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union708of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At709these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is710therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower711dimension, onto which the reducible strata of $\Theta$ are collapsed.712 713\section{Homogeneous activations}714 715\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]716Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector717field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing718parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$719wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$720at every parameter.