Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 221–320 of 2141. Source text is displayed literally.
221 \node[lab] at (6.1,0) {$=$};222 \node[lab] (t2) at (7.2,0) {$\Theta$};223 \node[lab] (m) at (9.6,0) {$[X,Y]$};224 \draw[fw,black] (t2) -- (m);225\end{tikzpicture}\par226\vspace{0.5in}227\noindent{\large \bookauthor}\par228\vspace{4pt}229\noindent{\small 2026}230\end{titlepage}231 232\thispagestyle{empty}233\vspace*{\fill}234{\small235\noindent\textbf{About this text.}236This monograph applies the results of \emph{Differential Topology in Higher Categories} \cite{DTHC}, the237higher-categorical treatment of M.~W.~Hirsch's \emph{Differential Topology} \cite{Hir76} in the same series, to smooth238families of maps defined by neural network architectures. It is written to be read after that volume and239cites it by theorem number, in the form \DT{4.5}. It is an independent work and is not affiliated with or240endorsed by the publisher of Hirsch's book.241 242\medskip243\noindent\textbf{Status of results.}244Every numbered statement carries a tag: \textsf{classical} (standard mathematics, attributed),245\textsf{literature} (established in the machine learning or mathematical literature, attributed),246\textsf{reformulation} (a known result restated or specialized, with the derivation given here),247\textsf{observation} (an elementary consequence we have not found stated in this form), or248\textsf{programme} (a direction, not a result). The statements tagged \textsf{reformulation} and249\textsf{observation} are corollaries of classical differential topology applied to parametrized families;250none is claimed as a new theorem of differential topology, and none says anything quantitative about251training or generalization.252 253\medskip254\noindent\textbf{Authorship and assistance.}255The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of256the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall,257mathematical consistency checks, computation, and typesetting. The numerical examples and finite instances258of the structural claims are recomputed by the accompanying script \texttt{verify-parametrized-networks.py}259(\cref{app:computations}), which uses small networks implemented in pure Python. Proofs and bibliographic260details have not been independently refereed; corrections are welcome.261 262\medskip263\noindent Open access. Rights remain with the author.264}265\clearpage266 267\tableofcontents268 269\chapter{Preface}270 271A neural network architecture is a smooth map $F\colon\Theta\times X\to Y$: a family of maps from inputs to272outputs, parametrized by weights. Differential topology, in the form Hirsch gave it, is the study of what is273true of a generic member of a family of maps and of what happens along a generic path in one. The fit is274close enough that most of the qualitative questions asked about networks have answers that are corollaries275of theorems proved before 1976, once the questions are phrased as questions about families.276 277Is the decision boundary of a classifier a manifold? For almost every value of the parameters, yes, by the278parametric transversality theorem, provided the last layer has a bias. How does the boundary change during279training? By ambient isotopy, except at isolated times, at each of which it undergoes one surgery whose type280is the index of a critical point; this is Morse theory on the trace of the boundary. What is the set of281parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the282number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary283differential equation separate two linked circles? Not in dimension three, because its time-one map is284ambient isotopic to the identity; in dimension four always, because there any two embeddings of a285one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the286companion volume, and this book supplies the paragraphs.287 288The higher-categorical layer of the companion volume enters in three places. A parametrized network is a289plot of the mapping object $[X,Y]$ in smooth sets, and composition of layers is composition in a category of290parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an291architecture make the parameter space an atlas for a stack whose points are networks up to relabelling,292and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional293parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is294what one sees of them from nearby.295 296The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a297rate of convergence or a generalization error. Activation functions are assumed smooth except where stated,298and the piecewise-linear case, which needs a different transversality theory, is treated only by reference.299What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks300the examples.301 302\chapter{Introduction}303 304\section*{What is assumed}305 306The companion volume \cite{DTHC}, at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of307Hirsch's book. From machine learning, only the definition of a feedforward network and of gradient descent.308 309\section*{How the book is organized}310 311Part~I sets up families, their derivatives and their symmetries. Part~II studies level sets: of the network312in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss313in parameter space (the interpolation manifold). Part~III studies the maps themselves: when they are314embeddings, when they are isotopic, when they are invertible. Part~IV is an assessment.315\Cref{app:dictionary} gives the dictionary between the two subjects in one table.316 317\section*{Conventions}318 319Manifolds and maps are smooth. A \emph{family} is a smooth map $F\colon\Theta\times X\to Y$; we write320$f_\theta=F(\theta,-)$. $P=\dim\Theta$ is the number of parameters, $d=\dim X$ the input dimension, $n$ the