Parametrized Neural Networks and Their Representations in Differential Topology
Families of maps, transversality, surgery along training, and isotopy
A neural network architecture is a smooth map $F\colon\Theta\times X\to Y$: a family of maps from inputs to outputs, parametrized by weights. Differential topology in the form Hirsch gave it is the study of what is true of a generic member of a family and of what happens along a generic path in one. This monograph applies the companion volume Differential Topology in Higher Categories (submitted separately in this series) to such families, and cites it by theorem number. Most of the qualitative questions asked about networks turn out to have answers that are short corollaries of theorems proved before 1976, once the questions are phrased as questions about families and the hypothesis that makes each answer true is identified.
Size. 61 pages; 9 chapters in four parts; three appendices (dictionary, computations, solutions); 32 exercises with solutions to all of them; a companion script with 49 computational checks on small networks implemented in pure Python. Every numbered statement carries a status tag: classical, literature, reformulation, observation or programme.
The mechanism
A parametrized network is a plot $\Theta\to[X,Y]$ of a mapping object in smooth sets, and layers compose in the bicategory $\mathsf{Para}$:
$$\begin{CD} \Theta\times\Theta' @>>> [X,Y]\times[Y,Z] @>{\circ}>> [X,Z] \end{CD}$$
A condition on a member $f_\theta$, or on a parameter, is the condition that a point lies in the preimage of a submanifold under a map built from $F$. The preimage is a manifold when that map is transverse to the submanifold, and transversality can be checked on the architecture, usually by exhibiting one direction in parameter space (a bias) in which the map moves freely. The book calls this the richness principle and names, at each use, the parameters that provide it. Genericity among all smooth maps says nothing about a given architecture; genericity within the family is what is proved.
Results, with their sources
Derivatives (Ch. 2). Backpropagation is the functoriality of the reverse derivative into lenses (Fong–Spivak–Tuyéras; Cruttwell et al.). The tangent kernel is $JJ^{\mathsf T}$ for the Jacobian $J$ of the evaluation map; for analytic activations the maximal rank of $J$ is attained on an open dense set.
Symmetries and moduli (Ch. 3). For one hidden layer of $\tanh$ units, irreducible parameters are identifiable up to a finite group $G$ (Sussmann; Fukumizu). Observation: the moduli $\Theta^{\mathrm{irr}}/G$ is a bundle over a configuration space, and for input dimension $d\ge2$ its fundamental group maps onto $S_H$: there are loops of networks returning to the same function with two hidden units exchanged. For positively homogeneous activations the functional dimension is at most $P-H$, the quotient is a stack and not a space, and no loss $\lambda\circ\mathrm{ev}$ is a Morse function.
Decision boundaries (Ch. 4). If the last layer has a bias, then for every value of the other parameters and almost every bias, $f_\theta^{-1}(0)$ is a smooth hypersurface (Sard). For compact input domains the discriminant $\Delta\subset\Theta$ is closed of measure zero, and the pair $(X,f_\theta^{-1}(0))$ is constant up to ambient isotopy on each component of $\Theta\setminus\Delta$ (Ehresmann, isotopy extension). For one hidden layer with $\sigma'>0$ and weight matrix of rank $d$, the map $(F,d_xF)$ is a submersion, so $\Delta$ is the image of a manifold of dimension $P-1$: chambers separated by walls. Every compact hypersurface in a domain is isotopic to a decision boundary of a wide enough one-hidden-layer network (tubular neighbourhoods, Hornik's $C^1$ approximation). For ReLU networks the corresponding genericity theorem is Grigsby–Lindsey's and is quoted, not reproved.
Surgery along training paths (Ch. 5). For a generic path $\gamma$ in $\Theta$, the union of the decision boundaries is a cobordism $W_\gamma\subset[0,1]\times X$, the times at which $\gamma$ crosses $\Delta$ are the critical values of the height function on $W_\gamma$, and at each the boundary undergoes one surgery whose type is the index of a non-degenerate critical point of $f_t$ crossing the threshold. The Euler characteristic of the boundary changes by $2(-1)^\lambda$ in odd input dimension and by $0$ in even. A compact decision boundary in $\mathbb R^d$, $d\ge2$, always bounds, so its cobordism class is zero; the information is in the isotopy type per chamber and in the surgery sequence, which is well defined up to Cerf moves.
The loss landscape (Ch. 6). For almost every label vector the set of interpolating parameters is a submanifold of dimension $P-N$ on which the Hessian is $J^{\mathsf T}J$ of rank $N$ (regular value theorem and Sard; Cooper). Barriers between minima are death levels in the degree-zero barcode of the sublevel functor. Three independent mechanisms make the loss fail to be Morse, and bounded activations make it fail to be proper; a two-parameter example ($\tfrac12(ab-1)^2$, barrier $\tfrac12$, removed by the symmetry) is worked completely.
Embeddings and isotopy (Ch. 7). A first layer of width $k\ge2m+1$ embeds an $m$-dimensional data manifold for almost every weight matrix. An injective network on $\mathbb R^d$ sends the centre of a ball into the convex hull of the image of its boundary, so it cannot linearly separate nested classes. A neural ODE with bounded velocity is a diffeotopy: it cannot reorder points on a line, separate nested spheres, or unlink circles in $\mathbb R^3$ (Dupont–Doucet–Teh). Augmentation: if $e_0,e_1\colon K\to\mathbb R^N$ are embeddings of a compact $m$-manifold and $N\ge2m+2$, there is a compactly supported flow carrying $e_0$ to $e_1$; so the obstructions vanish in dimension $2m+2$, where $m$ is the dimension of the data.
Degree and invertibility (Ch. 8). Residual blocks with Lipschitz constant below one are diffeomorphisms of degree one isotopic to the identity. A homeomorphism carries the support of a measure to the support of its image, which obstructs normalizing flows between non-homeomorphic supports (Cornish et al.). For an equilibrium layer $z=\tanh(Az+u)$: for every $A$ and almost every $u$ all fixed points are non-degenerate, their signed count is $1$, and so their number is odd.
What is and is not claimed
Nothing here is a new theorem of differential topology, and most of the statements about networks have appeared in the machine learning literature in some special case, with references given in the text. What the book adds is a uniform derivation with honest hypotheses, a dictionary (Appendix A), and a small number of statements tagged observation that we have not found in this form: the configuration-space description of the moduli and its monodromy; the wall structure of the discriminant; the surgery description of training with its Euler characteristic count and moves; the dimension count $2m+2$ for augmentation in terms of the data; and the parity of the number of equilibria. Each is a short corollary.
The closing chapter lists the limits. Activations are assumed smooth; the piecewise-linear case needs stratified methods and is treated by reference. "Almost every" is with respect to Lebesgue measure on parameters and says nothing about distinguished parameters; fixed-point sets of symmetries are invariant under gradient flow. The surgery theorem describes generic paths, and a training trajectory is one particular path; what is proved is that arbitrarily close to it there is a path with the stated behaviour. Parameter space is not compact. With one exception (the Polyak–Łojasiewicz inequality, quoted) there are no rates, and nothing concerns generalization. Four open directions are stated as problems.
Files
parametrized-neural-networks-and-their-representations-in-differential-topology.pdf— the monograph (61 pages).parametrized-neural-networks-and-their-representations-in-differential-topology.tex— complete LaTeX source as a single file (pdflatex, three passes).verify-parametrized-networks.py— 49 checks, Python standard library only, about three seconds; exits 0 if and only if all pass.
The script implements small feedforward networks with forward-mode and reverse-mode differentiation, a Runge–Kutta integrator and a component counter. It checks, among other things: backpropagation against dual numbers and finite differences; all 48 symmetries of a $(2,3,1)$ $\tanh$ network; Jacobian ranks $13$, $10$, $9$ at generic and reducible parameters and $\le P-H$ for ReLU; chambers and component counts through a birth and a merge; the Hessian $J^{\mathsf T}J$ with $4$ positive and $9$ zero eigenvalues at an interpolating parameter; invariance of the linking number under a neural ODE; separation of nested circles after augmentation; and odd numbers of equilibria with signed count $1$. A statement about almost every parameter is tested only at the parameters sampled.
Status and authorship
Contribution. Application and exposition: corollaries of classical differential topology for parametrized families, with attribution.
Limitations. As listed above; in addition, proofs marked as sketches are sketches, some literature results are quoted in simplified form, and neither proofs nor bibliographic details have been independently refereed. Corrections are welcome.
AI assistance. The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall, mathematical consistency checks, computation and typesetting.