Independent scholarship · Open accessThe monthly digest

Indexed text attachment

differential-topology-of-parametrized-neural-networks.tex

Source for Differential Topology of Parametrized Neural Networks

View paper · Download original file

Lines 361–460 of 1074. Source text is displayed literally.

361 \fill[paperaccent] (.5,1.5) circle (1.3pt);362 \fill[paperaccent] (.5,-.5) circle (1.3pt);363 \fill[orange!62!black] (.5,-.75) circle (1.3pt);364 \fill[paperink] (.25,-.75) circle (1.25pt);365 \fill[paperink] (-.25,.75) circle (1.25pt);366\end{tikzpicture}367\caption{Boundary discriminant of the quadratic circle score.368The lines come from $t=0,\pi$ and the curved branches from369$\cos t=-1/(4a)$. The black points mark degenerate critical370points. The dashed slice gives the regular counts in the text.}371\label{fig:boundary-discriminant}372\end{figure}373 374For example, fix $a=1/2$. With $u=\cos t$, the logit375equals $u^2+u-1/2-b$. The minimum as a function of376$u\in[-1,1]$ occurs at $u=-1/2$, while the endpoint377values before subtracting $b$ are $-1/2$ and $3/2$.378Every interior root in $u$ yields two roots on the379circle and an endpoint root yields one. The number380of boundary points at regular thresholds is therefore381\begin{center}382\begin{tabular}{@{}lc@{}}383\toprule384Regular threshold $b$ & Boundary points \\385\midrule386$(-\infty,-3/4)$ & $0$\\387$(-3/4,-1/2)$ & $4$\\388$(-1/2,3/2)$ & $2$\\389$(3/2,\infty)$ & $0$\\390\bottomrule391\end{tabular}392\end{center}393At the lower threshold two critical points enter394simultaneously because the network score is even in $t$.395At the other thresholds there is a single nondegenerate396critical point. Each isolated nondegenerate event has397the fold normal form, so it adds or removes an opposite-sign398pair of local roots \cite[Ch.~20]{higher-hirsch}.399The unsigned boundary topology changes, while the signed400count at every regular threshold is zero.401 402More generally, for a compact $M$ and a scalar403evaluation $F:P\times M\to\mathbb R$ with $0$ a regular404value, the projection of the zero locus is a proper405submersion off its discriminant. It is a smooth406bundle over each regular chamber in its image407\cite[Ch.~20]{higher-hirsch}. Over the other chambers the408fibers are empty. Local triviality concerns a whole chamber. It leaves409open the possibility of monodromy around a loop in that410chamber, even if the point count is fixed. Section~\ref{sec:boundary-monodromy}411computes that effect in a different network.412\par413\section{A stabilizer obstruction to regular readouts}414\label{sec:eq-readout}415 416A model with moving parameters can be equivariant even when417most of its fixed-parameter maps have few symmetries. Let a418compact Lie group $G$ act smoothly on manifolds $P,X$, and419linearly on a finite-dimensional real space $V$. Suppose a420smooth readout $F:P\times X\to V$ satisfies421\[422 F(gp,gx)=gF(p,x).423\]424A single realization $F_p(x)=F(p,x)$ is equivariant for425$K=G_p$. At a zero $x$ of $F_p$, the subgroup governing its426derivative is $H=K_x=G_p\cap G_x$. The general relation between427stabilizers and normal derivatives is developed in428\cite[Chs.~25--26]{higher-hirsch}.429 430\begin{proposition}[Regularity at a parameter stabilizer]431\label{prop:eq-readout}432If $F_p$ is transverse to zero at $x$, then433$T_xX/T_x(Kx)$ admits a surjective $H$-linear map to $V$.434In particular its $\sigma$-isotypic summand has dimension435at least that of $V_\sigma$ for each real irreducible436$\sigma$ of $H$. Conversely, a surjection of these437representations is realizable as the normal derivative of438a smooth $K$-equivariant map defined near the orbit $Kx$.439\end{proposition}440 441\begin{proof}442For $h\in H$, differentiate $F_p(hx')=hF_p(x')$ at $x$.443The result is $D_x F_p\circ D_xh=h\circ D_xF_p$.444The derivative vanishes on $T_x(Kx)$ since $F_p$ is zero445along the orbit. Surjectivity is preserved when the derivative446factors through this quotient. Because $H$ is compact, an447averaged inner product gives an $H$-invariant complement of448the kernel of a surjection. Therefore a copy of the target449sits in the source representation, with the asserted450inequality on each isotypic part.451 452For the converse, identify $T_xX/T_x(Kx)$ with the normal453representation $N_x$ for an invariant metric. The compact454slice theorem gives an invariant neighborhood of $Kx$455modeled on $K\times_H S$, where $S$ is an invariant456neighborhood of $0$ in $N_x$457\cite[Ch.~25]{higher-hirsch}. For an $H$-linear surjection458$L:N_x\to V$, the assignment $[k,s]\mapsto kL(s)$ is459well defined by equivariance of $L$. Its normal derivative460at $[1,0]$ is $L$, and translation by $K$ transports this