Indexed text attachment
differential-topology-of-parametrized-neural-networks.tex
Source for Differential Topology of Parametrized Neural Networks
View paper · Download original file
Lines 294–393 of 1074. Source text is displayed literally.
294the bias space is restricted to vectors with coordinate295sum zero. The differences of any bias vector are296unchanged by subtracting a constant vector so the297variation still surjects onto the normal directions to298each tie. Thus the ordinary redundancy of adding the299same constant to every logit can be removed without300losing the relevant perturbations.301 302There is a complementary global result when $\phi$303is an embedding. For different points $x,x'$ the304map $(A,x,x')\mapsto A(\phi(x)-\phi(x'))$ is a submersion305to $\mathbb R^q$. Indeed, $\phi(x)-\phi(x')$ is nonzero306and a linear map can assign it any value. The source307in the input variables is $M\times M\setminus\Delta$,308of dimension $2n$. When $q>2n$, parametric309transversality forces absence of off-diagonal coincidences310for a conull residual set of $A$. By the first-jet argument,311$r$-rank derivative strata of codimension312$(n-r)(q-r)$ are avoided when that codimension exceeds $n$.313This formula follows by choosing an invertible $r$-minor314and using block elimination: the remaining315$(q-r)\times(n-r)$ block after elimination must be zero.316In particular for $q\ge2n$ all nonimmersive strata317are avoided. For compact $M$ and $q\ge2n+1$ the generic318readout of an embedding is thus an embedding. An affine319bias has no effect on this two-point test. This is a version320of the projection argument for embeddings applied to321the allowed neural readouts.322 323\subsection{A two-parameter boundary discriminant}324\label{sec:threshold-discriminant}325 326Use the unit circle as a data manifold with coordinate327$x=(\cos t,\sin t)$. A network with one square unit and328affine readout can realize329\[330 F(a,b,t)=\cos t+a\cos(2t)-b331\]332because $\cos(2t)=2(\cos t)^2-1$.333Its zero locus in $(a,b,t)$ is smooth since the last bias334derivative is $-1$. Projection to $(a,b)$ is proper335because the circle is compact. A parameter belongs336to the discriminant when a root also satisfies337\[338 \partial_t F=-\sin t(1+4a\cos t)=0.339\]340Thus the discriminant consists of the two lines341$b=a+1$ and $b=a-1$ and the curve342$b=-a-1/(8a)$ for $|a|\ge1/4$.343At $|a|=1/4$ the latter joins a line with a degenerate344critical point. At $a=1/4$ and $t=\pi$ the first nonzero345variation of $\cos t+a\cos 2t$ in $t-\pi$ is quartic,346with coefficient $1/8$.347 348\begin{figure}[tbp]349\centering350\begin{tikzpicture}[x=2.3cm,y=.86cm,font=\small]351 \draw[->,gray!85] (-1.04,0) -- (1.08,0) node[right] {$a$};352 \draw[->,gray!85] (0,-2.02) -- (0,2.03) node[above] {$b$};353 \draw[paperaccent,thick,domain=-.94:.94] plot (\x,{\x+1});354 \draw[paperaccent,thick,domain=-.94:.94] plot (\x,{\x-1});355 \draw[orange!62!black,thick,domain=.25:.94,samples=60]356 plot (\x,{-\x-1/(8*\x)});357 \draw[orange!62!black,thick,domain=-.94:-.25,samples=60]358 plot (\x,{-\x-1/(8*\x)});359 \draw[dashed,gray!85] (.5,-1.96) -- (.5,1.90);360 \node[anchor=west] at (.54,.32) {$a=\tfrac12$};361 \fill[paperaccent] (.5,1.5) circle (1.3pt);362 \fill[paperaccent] (.5,-.5) circle (1.3pt);363 \fill[orange!62!black] (.5,-.75) circle (1.3pt);364 \fill[paperink] (.25,-.75) circle (1.25pt);365 \fill[paperink] (-.25,.75) circle (1.25pt);366\end{tikzpicture}367\caption{Boundary discriminant of the quadratic circle score.368The lines come from $t=0,\pi$ and the curved branches from369$\cos t=-1/(4a)$. The black points mark degenerate critical370points. The dashed slice gives the regular counts in the text.}371\label{fig:boundary-discriminant}372\end{figure}373 374For example, fix $a=1/2$. With $u=\cos t$, the logit375equals $u^2+u-1/2-b$. The minimum as a function of376$u\in[-1,1]$ occurs at $u=-1/2$, while the endpoint377values before subtracting $b$ are $-1/2$ and $3/2$.378Every interior root in $u$ yields two roots on the379circle and an endpoint root yields one. The number380of boundary points at regular thresholds is therefore381\begin{center}382\begin{tabular}{@{}lc@{}}383\toprule384Regular threshold $b$ & Boundary points \\385\midrule386$(-\infty,-3/4)$ & $0$\\387$(-3/4,-1/2)$ & $4$\\388$(-1/2,3/2)$ & $2$\\389$(3/2,\infty)$ & $0$\\390\bottomrule391\end{tabular}392\end{center}393At the lower threshold two critical points enter