Independent scholarship · Open accessThe monthly digest

Indexed text attachment

differential-topology-of-parametrized-neural-networks.tex

Source for Differential Topology of Parametrized Neural Networks

View paper · Download original file

Lines 321–420 of 1074. Source text is displayed literally.

321the allowed neural readouts.322 323\subsection{A two-parameter boundary discriminant}324\label{sec:threshold-discriminant}325 326Use the unit circle as a data manifold with coordinate327$x=(\cos t,\sin t)$. A network with one square unit and328affine readout can realize329\[330 F(a,b,t)=\cos t+a\cos(2t)-b331\]332because $\cos(2t)=2(\cos t)^2-1$.333Its zero locus in $(a,b,t)$ is smooth since the last bias334derivative is $-1$. Projection to $(a,b)$ is proper335because the circle is compact. A parameter belongs336to the discriminant when a root also satisfies337\[338 \partial_t F=-\sin t(1+4a\cos t)=0.339\]340Thus the discriminant consists of the two lines341$b=a+1$ and $b=a-1$ and the curve342$b=-a-1/(8a)$ for $|a|\ge1/4$.343At $|a|=1/4$ the latter joins a line with a degenerate344critical point. At $a=1/4$ and $t=\pi$ the first nonzero345variation of $\cos t+a\cos 2t$ in $t-\pi$ is quartic,346with coefficient $1/8$.347 348\begin{figure}[tbp]349\centering350\begin{tikzpicture}[x=2.3cm,y=.86cm,font=\small]351 \draw[->,gray!85] (-1.04,0) -- (1.08,0) node[right] {$a$};352 \draw[->,gray!85] (0,-2.02) -- (0,2.03) node[above] {$b$};353 \draw[paperaccent,thick,domain=-.94:.94] plot (\x,{\x+1});354 \draw[paperaccent,thick,domain=-.94:.94] plot (\x,{\x-1});355 \draw[orange!62!black,thick,domain=.25:.94,samples=60]356       plot (\x,{-\x-1/(8*\x)});357 \draw[orange!62!black,thick,domain=-.94:-.25,samples=60]358       plot (\x,{-\x-1/(8*\x)});359 \draw[dashed,gray!85] (.5,-1.96) -- (.5,1.90);360 \node[anchor=west] at (.54,.32) {$a=\tfrac12$};361 \fill[paperaccent] (.5,1.5) circle (1.3pt);362 \fill[paperaccent] (.5,-.5) circle (1.3pt);363 \fill[orange!62!black] (.5,-.75) circle (1.3pt);364 \fill[paperink] (.25,-.75) circle (1.25pt);365 \fill[paperink] (-.25,.75) circle (1.25pt);366\end{tikzpicture}367\caption{Boundary discriminant of the quadratic circle score.368The lines come from $t=0,\pi$ and the curved branches from369$\cos t=-1/(4a)$. The black points mark degenerate critical370points. The dashed slice gives the regular counts in the text.}371\label{fig:boundary-discriminant}372\end{figure}373 374For example, fix $a=1/2$. With $u=\cos t$, the logit375equals $u^2+u-1/2-b$. The minimum as a function of376$u\in[-1,1]$ occurs at $u=-1/2$, while the endpoint377values before subtracting $b$ are $-1/2$ and $3/2$.378Every interior root in $u$ yields two roots on the379circle and an endpoint root yields one. The number380of boundary points at regular thresholds is therefore381\begin{center}382\begin{tabular}{@{}lc@{}}383\toprule384Regular threshold $b$ & Boundary points \\385\midrule386$(-\infty,-3/4)$ & $0$\\387$(-3/4,-1/2)$ & $4$\\388$(-1/2,3/2)$ & $2$\\389$(3/2,\infty)$ & $0$\\390\bottomrule391\end{tabular}392\end{center}393At the lower threshold two critical points enter394simultaneously because the network score is even in $t$.395At the other thresholds there is a single nondegenerate396critical point. Each isolated nondegenerate event has397the fold normal form, so it adds or removes an opposite-sign398pair of local roots \cite[Ch.~20]{higher-hirsch}.399The unsigned boundary topology changes, while the signed400count at every regular threshold is zero.401 402More generally, for a compact $M$ and a scalar403evaluation $F:P\times M\to\mathbb R$ with $0$ a regular404value, the projection of the zero locus is a proper405submersion off its discriminant. It is a smooth406bundle over each regular chamber in its image407\cite[Ch.~20]{higher-hirsch}. Over the other chambers the408fibers are empty. Local triviality concerns a whole chamber. It leaves409open the possibility of monodromy around a loop in that410chamber, even if the point count is fixed. Section~\ref{sec:boundary-monodromy}411computes that effect in a different network.412\par413\section{A stabilizer obstruction to regular readouts}414\label{sec:eq-readout}415 416A model with moving parameters can be equivariant even when417most of its fixed-parameter maps have few symmetries. Let a418compact Lie group $G$ act smoothly on manifolds $P,X$, and419linearly on a finite-dimensional real space $V$. Suppose a420smooth readout $F:P\times X\to V$ satisfies