Indexed text attachment
differential-topology-of-parametrized-neural-networks.tex
Source for Differential Topology of Parametrized Neural Networks
View paper · Download original file
Lines 301–400 of 1074. Source text is displayed literally.
301 302There is a complementary global result when $\phi$303is an embedding. For different points $x,x'$ the304map $(A,x,x')\mapsto A(\phi(x)-\phi(x'))$ is a submersion305to $\mathbb R^q$. Indeed, $\phi(x)-\phi(x')$ is nonzero306and a linear map can assign it any value. The source307in the input variables is $M\times M\setminus\Delta$,308of dimension $2n$. When $q>2n$, parametric309transversality forces absence of off-diagonal coincidences310for a conull residual set of $A$. By the first-jet argument,311$r$-rank derivative strata of codimension312$(n-r)(q-r)$ are avoided when that codimension exceeds $n$.313This formula follows by choosing an invertible $r$-minor314and using block elimination: the remaining315$(q-r)\times(n-r)$ block after elimination must be zero.316In particular for $q\ge2n$ all nonimmersive strata317are avoided. For compact $M$ and $q\ge2n+1$ the generic318readout of an embedding is thus an embedding. An affine319bias has no effect on this two-point test. This is a version320of the projection argument for embeddings applied to321the allowed neural readouts.322 323\subsection{A two-parameter boundary discriminant}324\label{sec:threshold-discriminant}325 326Use the unit circle as a data manifold with coordinate327$x=(\cos t,\sin t)$. A network with one square unit and328affine readout can realize329\[330 F(a,b,t)=\cos t+a\cos(2t)-b331\]332because $\cos(2t)=2(\cos t)^2-1$.333Its zero locus in $(a,b,t)$ is smooth since the last bias334derivative is $-1$. Projection to $(a,b)$ is proper335because the circle is compact. A parameter belongs336to the discriminant when a root also satisfies337\[338 \partial_t F=-\sin t(1+4a\cos t)=0.339\]340Thus the discriminant consists of the two lines341$b=a+1$ and $b=a-1$ and the curve342$b=-a-1/(8a)$ for $|a|\ge1/4$.343At $|a|=1/4$ the latter joins a line with a degenerate344critical point. At $a=1/4$ and $t=\pi$ the first nonzero345variation of $\cos t+a\cos 2t$ in $t-\pi$ is quartic,346with coefficient $1/8$.347 348\begin{figure}[tbp]349\centering350\begin{tikzpicture}[x=2.3cm,y=.86cm,font=\small]351 \draw[->,gray!85] (-1.04,0) -- (1.08,0) node[right] {$a$};352 \draw[->,gray!85] (0,-2.02) -- (0,2.03) node[above] {$b$};353 \draw[paperaccent,thick,domain=-.94:.94] plot (\x,{\x+1});354 \draw[paperaccent,thick,domain=-.94:.94] plot (\x,{\x-1});355 \draw[orange!62!black,thick,domain=.25:.94,samples=60]356 plot (\x,{-\x-1/(8*\x)});357 \draw[orange!62!black,thick,domain=-.94:-.25,samples=60]358 plot (\x,{-\x-1/(8*\x)});359 \draw[dashed,gray!85] (.5,-1.96) -- (.5,1.90);360 \node[anchor=west] at (.54,.32) {$a=\tfrac12$};361 \fill[paperaccent] (.5,1.5) circle (1.3pt);362 \fill[paperaccent] (.5,-.5) circle (1.3pt);363 \fill[orange!62!black] (.5,-.75) circle (1.3pt);364 \fill[paperink] (.25,-.75) circle (1.25pt);365 \fill[paperink] (-.25,.75) circle (1.25pt);366\end{tikzpicture}367\caption{Boundary discriminant of the quadratic circle score.368The lines come from $t=0,\pi$ and the curved branches from369$\cos t=-1/(4a)$. The black points mark degenerate critical370points. The dashed slice gives the regular counts in the text.}371\label{fig:boundary-discriminant}372\end{figure}373 374For example, fix $a=1/2$. With $u=\cos t$, the logit375equals $u^2+u-1/2-b$. The minimum as a function of376$u\in[-1,1]$ occurs at $u=-1/2$, while the endpoint377values before subtracting $b$ are $-1/2$ and $3/2$.378Every interior root in $u$ yields two roots on the379circle and an endpoint root yields one. The number380of boundary points at regular thresholds is therefore381\begin{center}382\begin{tabular}{@{}lc@{}}383\toprule384Regular threshold $b$ & Boundary points \\385\midrule386$(-\infty,-3/4)$ & $0$\\387$(-3/4,-1/2)$ & $4$\\388$(-1/2,3/2)$ & $2$\\389$(3/2,\infty)$ & $0$\\390\bottomrule391\end{tabular}392\end{center}393At the lower threshold two critical points enter394simultaneously because the network score is even in $t$.395At the other thresholds there is a single nondegenerate396critical point. Each isolated nondegenerate event has397the fold normal form, so it adds or removes an opposite-sign398pair of local roots \cite[Ch.~20]{higher-hirsch}.399The unsigned boundary topology changes, while the signed400count at every regular threshold is zero.