Indexed text attachment
differential-topology-of-parametrized-neural-networks.tex
Source for Differential Topology of Parametrized Neural Networks
View paper · Download original file
Lines 134–233 of 1074. Source text is displayed literally.
134constraints remain even if local perturbations are plentiful.135A basic example is an encoder with a smooth, trainable readout.136The readout can have enough derivatives to move a decision boundary,137while a narrow intermediate space still obstructs reconstruction138on the whole data manifold.139 140The starting points are Hirsch's treatment of jets and transversality141\cite[Chs.~2--4]{hirsch} and their extension to relative families,142derived pullbacks, and equivariant representations in143\cite[Chs.~18--22, 25--28]{higher-hirsch}.144Earlier categorical models of learning represent layers by parametrized145maps, including their compositions and shared weights146\cite{fong-spivak-tuyeras,previous-networks}. Here the map from a147prescribed architecture to the function object is subjected to148differential-topological tests. Related empirical work investigates149changes in data topology across network layers \cite{naitzat}.150Our statements concern smooth realizations and their included151perturbations.152 153\subsection{Relative evaluation and representations}154 155Let $\mathcal E$ be the $\infty$-topos of sheaves of spaces on156smooth test manifolds, and let $B$ be a coefficient or geometry157base. For $X,Y\to B$ write $[X,Y]_B$ for the internal mapping158object in $\mathcal E_{/B}$. A model with weights $P\to B$ is an159evaluation160\[161 F:P\times_B X\longrightarrow Y162\]163over $B$, or its transpose $P\to[X,Y]_B$. For example,164if $B=*$ and $X,Y$ are represented by smooth manifolds, an165evaluation on $P$ is a smooth map $P\times X\to Y$ on ordinary166representable parameter charts. The classifying map records167the realizations available to that architecture. Two different168points of $P$ can specify the same evaluation, and a change169of coordinates acts on the presenting parameter space.170A parameter shared by multiple factors of the network is a171pullback along the corresponding diagonal into the product172of their parameter spaces.173 174Internal exponentials commute with base change in $B$.175For an effective epimorphism $U\to B$ with \v{C}ech nerve176$U_\bullet$, we have the descent equivalence177\[178 \E_{/[X,Y]_B}\simeq179 \lim_{[n]\in\boldsymbol\Delta}180 \E_{/[X|_{U_n},Y|_{U_n}]_{U_n}}181\]182from \cite[Ch.~18]{higher-hirsch}. Thus compatible evaluations and183parameter presentations on geometry charts glue. On charts of184smooth bundles, a change of input and output coordinates transforms185the vertical derivative by the derivative of the output change on186the left and the inverse input change on the right. It preserves rank,187so rank strata and regularity conditions are intrinsic to the188glued model. For an action, the same formula is conjugation by189the linear representations on tangent spaces. Its restriction to190stabilizers becomes a concrete selection rule in191Section~\ref{sec:eq-readout}.192 193\subsection{Which kind of realization is measured}194 195A differentiable classifier with real logit $f_p$ has a196decision boundary $f_p^{-1}(0)$ and sublevel sets197$f_p^{-1}((-\infty,c])$. A scalar strictly monotone link, such198as the logistic link, preserves the boundary when the corresponding199threshold is transported. A classifier with several logits admits200a stratification by the indices realizing the maximum. An autoencoder,201by contrast, is a factorization of an approximation to the202identity on a data manifold. Its topology depends on the encoder's203image in the latent space and on the codomain of the decoder.204We use smooth activations for all jet and regular-value assertions.205The uniform reconstruction obstruction requires only continuous206realizations, so it also applies to continuous piecewise-linear207activations.208\par209\section{Readouts and the available jet directions}210\label{sec:readout-jets}211 212Let $M$ be an $n$-dimensional smooth manifold without boundary.213Fix a smooth feature extractor $\phi:M\to\mathbb R^d$ and214use the affine readout215\begin{equation}216 \label{eq:affine-readout}217 f_{A,b}(x)=A\phi(x)+b,\qquad218 (A,b)\in\mathcal A219 =\operatorname{Hom}(\mathbb R^d,\mathbb R^q)\times\mathbb R^q .220\end{equation}221The choice to hold the feature extractor fixed specifies a subfamily222of whatever full architecture realizes $\phi$. This separates the223derivatives supplied by the readout from those provided by224perturbing upstream layers.225 226\begin{theorem}[First-jet control by an affine readout]227\label{thm:jet-surjectivity}228If $\phi$ is an immersion, then229$J:\mathcal A\times M\to J^1(M,\mathbb R^q)$,230$(A,b,x)\mapsto j^1_x f_{A,b}$, is a submersion.231Without the immersion assumption, the corresponding map to232$J^0(M,\mathbb R^q)$ remains a submersion. In either case, for a233finite or countable family of smooth jet submanifolds at the