Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 661–760 of 2141. Source text is displayed literally.

661\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]662Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of663hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and664only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;665so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.666\end{theorem}667 668For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of669Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a670finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on671it.672 673\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob674The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is675injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then676\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }677Z'\bigr\}\times\R .\]678If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.679\end{proposition}680 681\begin{proof}682A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive683identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,684times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and685the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space686of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the687fundamental group of the base.688\end{proof}689 690So there are loops of networks, along which the function computed changes continuously and returns to691itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged692roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space693of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.694 695\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}696\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]697At a reducible parameter the fibre of the realization map has positive dimension:698\begin{enumerate}[label=(\roman*)]699\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;700\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;701\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates702$(a_j,b_j,c)$.703\end{enumerate}704\end{proposition}705 706Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of707parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union708of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At709these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is710therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower711dimension, onto which the reducible strata of $\Theta$ are collapsed.712 713\section{Homogeneous activations}714 715\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]716Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector717field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing718parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$719wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$720at every parameter.721\end{proposition}722 723\begin{proof}724$\xi_j$ is the generator of the one-parameter group of symmetries of \cref{prop:syms}(iii); differentiate725$F(g_\lambda\theta,x)=F(\theta,x)$ at $\lambda=1$. The $\xi_j$ for different units have disjoint supports in726the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and727the bound holds on the closure of that set by semicontinuity of rank.728\end{proof}729 730The script confirms both statements on small networks: for $\tanh$ the Jacobian has full rank $P$ at a731random parameter and loses rank at reducible ones; for $\relu$ its rank never exceeds $P-H$ and the fields732$\xi_j$ lie in its kernel.733 734For $\relu$ the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space:735the orbit of a unit with zero outgoing weight is not closed, since736$(\lambda w,\lambda b,0)\to(0,0,0)$ as $\lambda\to0$. The correct object is the quotient stack, in which737such points are remembered with their stabilizers. This is the same reason that vector bundles in738\DT{Ch.~5} formed a stack and not a space, and the loss is a function on it.739 740\section{Invariant functions}741 742\begin{proposition}\label{prop:invariant}\Scl743Let $\loss\colon\Theta\to\R$ be invariant under a group $G$ of symmetries. Then the critical set of744$\loss$ is a union of $G$-orbits. If $G$ contains a one-parameter group with generator $\xi$, then745$d\loss(\xi)=0$ identically and, at every critical point, $\xi$ lies in the kernel of the Hessian. If746moreover $\nabla\lambda$ vanishes at $\ev(\theta)$ for a loss $\loss=\lambda\circ\ev$, then the whole fibre747of $\ev$ through $\theta$ is critical.748\end{proposition}749 750\begin{proof}751Differentiating $\loss(g_s\theta)=\loss(\theta)$ in $s$ gives $d\loss_\theta(\xi)=0$ for all $\theta$;752differentiating that identity in a direction $v$ at a critical point gives753$\nabla^2\loss(\xi,v)+d\loss(D_v\xi)=\nabla^2\loss(\xi,v)=0$. The last statement is754$\nabla\loss=J^{\mathsf T}\nabla\lambda$.755\end{proof}756 757\begin{corollary}\label{cor:notmorse}\Sob758For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the759form $\lambda\circ\ev$ is a Morse function on any open set containing a critical point at which some hidden760unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least