Independent scholarship · Open accessThe monthly digest

Indexed text attachment

differential-topology-of-parametrized-neural-networks.tex

Source for Differential Topology of Parametrized Neural Networks

View paper · Download original file

Lines 181–280 of 1074. Source text is displayed literally.

181\]182from \cite[Ch.~18]{higher-hirsch}. Thus compatible evaluations and183parameter presentations on geometry charts glue. On charts of184smooth bundles, a change of input and output coordinates transforms185the vertical derivative by the derivative of the output change on186the left and the inverse input change on the right. It preserves rank,187so rank strata and regularity conditions are intrinsic to the188glued model. For an action, the same formula is conjugation by189the linear representations on tangent spaces. Its restriction to190stabilizers becomes a concrete selection rule in191Section~\ref{sec:eq-readout}.192 193\subsection{Which kind of realization is measured}194 195A differentiable classifier with real logit $f_p$ has a196decision boundary $f_p^{-1}(0)$ and sublevel sets197$f_p^{-1}((-\infty,c])$. A scalar strictly monotone link, such198as the logistic link, preserves the boundary when the corresponding199threshold is transported. A classifier with several logits admits200a stratification by the indices realizing the maximum. An autoencoder,201by contrast, is a factorization of an approximation to the202identity on a data manifold. Its topology depends on the encoder's203image in the latent space and on the codomain of the decoder.204We use smooth activations for all jet and regular-value assertions.205The uniform reconstruction obstruction requires only continuous206realizations, so it also applies to continuous piecewise-linear207activations.208\par209\section{Readouts and the available jet directions}210\label{sec:readout-jets}211 212Let $M$ be an $n$-dimensional smooth manifold without boundary.213Fix a smooth feature extractor $\phi:M\to\mathbb R^d$ and214use the affine readout215\begin{equation}216 \label{eq:affine-readout}217 f_{A,b}(x)=A\phi(x)+b,\qquad218 (A,b)\in\mathcal A219 =\operatorname{Hom}(\mathbb R^d,\mathbb R^q)\times\mathbb R^q .220\end{equation}221The choice to hold the feature extractor fixed specifies a subfamily222of whatever full architecture realizes $\phi$. This separates the223derivatives supplied by the readout from those provided by224perturbing upstream layers.225 226\begin{theorem}[First-jet control by an affine readout]227\label{thm:jet-surjectivity}228If $\phi$ is an immersion, then229$J:\mathcal A\times M\to J^1(M,\mathbb R^q)$,230$(A,b,x)\mapsto j^1_x f_{A,b}$, is a submersion.231Without the immersion assumption, the corresponding map to232$J^0(M,\mathbb R^q)$ remains a submersion. In either case, for a233finite or countable family of smooth jet submanifolds at the234indicated order, the parameters for which the corresponding235jet section is transverse to all of them form a conull residual236set.237\end{theorem}238 239\begin{proof}240At fixed $x$, use jet coordinates $(y,L)$ with241$L:T_xM\to\mathbb R^q$. The differential in the readout242directions is243\[244 (\dot A,\dot b)245 \longmapsto246 \bigl(\dot A\phi(x)+\dot b,\ \dot A D\phi_x\bigr).247\]248Given any $L_0:T_xM\to\mathbb R^q$ and immersive $\phi$, extend249$L_0(D\phi_x)^{-1}$ linearly from $D\phi_x(T_xM)$ to250$\mathbb R^d$ to obtain $\dot A$. Then any value variation251$y_0$ is realized by $\dot b=y_0-\dot A\phi(x)$.252This proves surjectivity in the vertical jet coordinates.253Base coordinates are covered because $J$ lies over the254projection to $M$. For $J^0$ only the value variation is255needed, and free biases supply it at every $x$.256 257A submersion is transverse to each smooth jet submanifold $S$.258Its preimage is smooth, and regular values of the projection259of the preimage to $\mathcal A$ are exactly the parameters260whose jet extensions are transverse to $S$. This is the261parametric transversality argument262\cite[Ch.~6]{higher-hirsch}. Sard's theorem gives a null263exceptional set. The critical locus has a countable compact264exhaustion because it is closed in a second-countable265manifold. Its images under the projection are compact266null sets and therefore nowhere dense. Taking the countable267intersection of the good parameter sets proves the claim.268\end{proof}269 270For a binary classifier, Theorem~\ref{thm:jet-surjectivity}271implies that generically the decision boundary is empty or a272codimension-one smooth submanifold, even if $\phi$ has273singular points. The free final bias alone suffices. A274fixed-bias family can have additional constraints.275For example, if a feature identically vanishes on a276region, a bias-free readout may vanish on that entire region.277For vector outputs, immersion of $\phi$ supplies every278first-order normal perturbation at a given input. It makes279no assertion that independent perturbations at multiple280inputs can be realized by the same $\dot A$.