Independent scholarship · Open accessThe monthly digest

Indexed text attachment

differential-topology-of-parametrized-neural-networks.tex

Source for Differential Topology of Parametrized Neural Networks

View paper · Download original file

Lines 381–480 of 1074. Source text is displayed literally.

381\begin{center}382\begin{tabular}{@{}lc@{}}383\toprule384Regular threshold $b$ & Boundary points \\385\midrule386$(-\infty,-3/4)$ & $0$\\387$(-3/4,-1/2)$ & $4$\\388$(-1/2,3/2)$ & $2$\\389$(3/2,\infty)$ & $0$\\390\bottomrule391\end{tabular}392\end{center}393At the lower threshold two critical points enter394simultaneously because the network score is even in $t$.395At the other thresholds there is a single nondegenerate396critical point. Each isolated nondegenerate event has397the fold normal form, so it adds or removes an opposite-sign398pair of local roots \cite[Ch.~20]{higher-hirsch}.399The unsigned boundary topology changes, while the signed400count at every regular threshold is zero.401 402More generally, for a compact $M$ and a scalar403evaluation $F:P\times M\to\mathbb R$ with $0$ a regular404value, the projection of the zero locus is a proper405submersion off its discriminant. It is a smooth406bundle over each regular chamber in its image407\cite[Ch.~20]{higher-hirsch}. Over the other chambers the408fibers are empty. Local triviality concerns a whole chamber. It leaves409open the possibility of monodromy around a loop in that410chamber, even if the point count is fixed. Section~\ref{sec:boundary-monodromy}411computes that effect in a different network.412\par413\section{A stabilizer obstruction to regular readouts}414\label{sec:eq-readout}415 416A model with moving parameters can be equivariant even when417most of its fixed-parameter maps have few symmetries. Let a418compact Lie group $G$ act smoothly on manifolds $P,X$, and419linearly on a finite-dimensional real space $V$. Suppose a420smooth readout $F:P\times X\to V$ satisfies421\[422 F(gp,gx)=gF(p,x).423\]424A single realization $F_p(x)=F(p,x)$ is equivariant for425$K=G_p$. At a zero $x$ of $F_p$, the subgroup governing its426derivative is $H=K_x=G_p\cap G_x$. The general relation between427stabilizers and normal derivatives is developed in428\cite[Chs.~25--26]{higher-hirsch}.429 430\begin{proposition}[Regularity at a parameter stabilizer]431\label{prop:eq-readout}432If $F_p$ is transverse to zero at $x$, then433$T_xX/T_x(Kx)$ admits a surjective $H$-linear map to $V$.434In particular its $\sigma$-isotypic summand has dimension435at least that of $V_\sigma$ for each real irreducible436$\sigma$ of $H$. Conversely, a surjection of these437representations is realizable as the normal derivative of438a smooth $K$-equivariant map defined near the orbit $Kx$.439\end{proposition}440 441\begin{proof}442For $h\in H$, differentiate $F_p(hx')=hF_p(x')$ at $x$.443The result is $D_x F_p\circ D_xh=h\circ D_xF_p$.444The derivative vanishes on $T_x(Kx)$ since $F_p$ is zero445along the orbit. Surjectivity is preserved when the derivative446factors through this quotient. Because $H$ is compact, an447averaged inner product gives an $H$-invariant complement of448the kernel of a surjection. Therefore a copy of the target449sits in the source representation, with the asserted450inequality on each isotypic part.451 452For the converse, identify $T_xX/T_x(Kx)$ with the normal453representation $N_x$ for an invariant metric. The compact454slice theorem gives an invariant neighborhood of $Kx$455modeled on $K\times_H S$, where $S$ is an invariant456neighborhood of $0$ in $N_x$457\cite[Ch.~25]{higher-hirsch}. For an $H$-linear surjection458$L:N_x\to V$, the assignment $[k,s]\mapsto kL(s)$ is459well defined by equivariance of $L$. Its normal derivative460at $[1,0]$ is $L$, and translation by $K$ transports this461surjectivity along the orbit.462\end{proof}463 464The converse concerns arbitrary equivariant smooth germs.465An architecture restricts which such germs can occur.466In the total space the parameter derivative contributes an467additional $H$-map $T_pP\to V$. Surjectivity at $(p,x)$468tests the sum of its image and the image of the feature469derivative. The two regularity tests correspond to different470domains of the same evaluation.471 472\subsection{Two sign readouts on a torus}473 474Take $G=C_2$ and $X=(\mathbb R/2\pi\mathbb Z)^2$, with475the reflection $r(\theta,\phi)=(-\theta,\phi)$. Write476$\mathbb R_+$ and $\mathbb R_-$ for the trivial and sign477representations. Use two real sign readouts, so that478$V=\mathbb R_-^2$. The fixed locus consists of the two479circles $\theta=0,\pi$. A $G$-equivariant readout vanishes480there because $V^G=0$. On each fixed circle the source