Independent scholarship · Open accessThe monthly digest

Indexed text attachment

differential-topology-of-parametrized-neural-networks.tex

Source for Differential Topology of Parametrized Neural Networks

View paper · Download original file

Lines 194–293 of 1074. Source text is displayed literally.

194 195A differentiable classifier with real logit $f_p$ has a196decision boundary $f_p^{-1}(0)$ and sublevel sets197$f_p^{-1}((-\infty,c])$. A scalar strictly monotone link, such198as the logistic link, preserves the boundary when the corresponding199threshold is transported. A classifier with several logits admits200a stratification by the indices realizing the maximum. An autoencoder,201by contrast, is a factorization of an approximation to the202identity on a data manifold. Its topology depends on the encoder's203image in the latent space and on the codomain of the decoder.204We use smooth activations for all jet and regular-value assertions.205The uniform reconstruction obstruction requires only continuous206realizations, so it also applies to continuous piecewise-linear207activations.208\par209\section{Readouts and the available jet directions}210\label{sec:readout-jets}211 212Let $M$ be an $n$-dimensional smooth manifold without boundary.213Fix a smooth feature extractor $\phi:M\to\mathbb R^d$ and214use the affine readout215\begin{equation}216 \label{eq:affine-readout}217 f_{A,b}(x)=A\phi(x)+b,\qquad218 (A,b)\in\mathcal A219 =\operatorname{Hom}(\mathbb R^d,\mathbb R^q)\times\mathbb R^q .220\end{equation}221The choice to hold the feature extractor fixed specifies a subfamily222of whatever full architecture realizes $\phi$. This separates the223derivatives supplied by the readout from those provided by224perturbing upstream layers.225 226\begin{theorem}[First-jet control by an affine readout]227\label{thm:jet-surjectivity}228If $\phi$ is an immersion, then229$J:\mathcal A\times M\to J^1(M,\mathbb R^q)$,230$(A,b,x)\mapsto j^1_x f_{A,b}$, is a submersion.231Without the immersion assumption, the corresponding map to232$J^0(M,\mathbb R^q)$ remains a submersion. In either case, for a233finite or countable family of smooth jet submanifolds at the234indicated order, the parameters for which the corresponding235jet section is transverse to all of them form a conull residual236set.237\end{theorem}238 239\begin{proof}240At fixed $x$, use jet coordinates $(y,L)$ with241$L:T_xM\to\mathbb R^q$. The differential in the readout242directions is243\[244 (\dot A,\dot b)245 \longmapsto246 \bigl(\dot A\phi(x)+\dot b,\ \dot A D\phi_x\bigr).247\]248Given any $L_0:T_xM\to\mathbb R^q$ and immersive $\phi$, extend249$L_0(D\phi_x)^{-1}$ linearly from $D\phi_x(T_xM)$ to250$\mathbb R^d$ to obtain $\dot A$. Then any value variation251$y_0$ is realized by $\dot b=y_0-\dot A\phi(x)$.252This proves surjectivity in the vertical jet coordinates.253Base coordinates are covered because $J$ lies over the254projection to $M$. For $J^0$ only the value variation is255needed, and free biases supply it at every $x$.256 257A submersion is transverse to each smooth jet submanifold $S$.258Its preimage is smooth, and regular values of the projection259of the preimage to $\mathcal A$ are exactly the parameters260whose jet extensions are transverse to $S$. This is the261parametric transversality argument262\cite[Ch.~6]{higher-hirsch}. Sard's theorem gives a null263exceptional set. The critical locus has a countable compact264exhaustion because it is closed in a second-countable265manifold. Its images under the projection are compact266null sets and therefore nowhere dense. Taking the countable267intersection of the good parameter sets proves the claim.268\end{proof}269 270For a binary classifier, Theorem~\ref{thm:jet-surjectivity}271implies that generically the decision boundary is empty or a272codimension-one smooth submanifold, even if $\phi$ has273singular points. The free final bias alone suffices. A274fixed-bias family can have additional constraints.275For example, if a feature identically vanishes on a276region, a bias-free readout may vanish on that entire region.277For vector outputs, immersion of $\phi$ supplies every278first-order normal perturbation at a given input. It makes279no assertion that independent perturbations at multiple280inputs can be realized by the same $\dot A$.281 282\subsection{Several classes and several points}283 284Consider $q$ logits and the open part of the tie stratum285where a subset $I$ of $r$ classes is exactly the set of286maximizers. Fix $i_0\in I$. Equalities287$y_i-y_{i_0}=0$ for $i\in I\setminus\{i_0\}$ are $r-1$288independent linear equations, and strict inequalities289separate the other coordinates. This is a smooth290codimension-$(r-1)$ submanifold of $\mathbb R^q$.291Consequently, the inverse image for a generic readout292is empty or has codimension $r-1$. No $r$-way maximum293tie occurs if $r>n+1$. The conclusion also holds when