Parametrized Neural Networks and Their Representations in Differential Topology
Continuous, single-column reading. Equations, tables, proofs, references and appendices stay in document order.
Contents · 84 sections
About this text. This monograph applies the results of Differential Topology in Higher Categories [9], the higher-categorical treatment of M. W. Hirsch’s Differential Topology [20] in the same series, to smooth families of maps defined by neural network architectures. It is written to be read after that volume and cites it by theorem number, in the form [9, 4.5]. It is an independent work and is not affiliated with or endorsed by the publisher of Hirsch’s book.
Status of results. Every numbered statement carries a tag: classical (standard mathematics, attributed), literature (established in the machine learning or mathematical literature, attributed), reformulation (a known result restated or specialized, with the derivation given here), observation (an elementary consequence we have not found stated in this form), or programme (a direction, not a result). The statements tagged reformulation and observation are corollaries of classical differential topology applied to parametrized families; none is claimed as a new theorem of differential topology, and none says anything quantitative about training or generalization.
Authorship and assistance. The text was drafted and typeset with a large language model (Anthropic’s Claude) under the direction of the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall, mathematical consistency checks, computation, and typesetting. The numerical examples and finite instances of the structural claims are recomputed by the accompanying script verify-parametrized-networks.py (3), which uses small networks implemented in pure Python. Proofs and bibliographic details have not been independently refereed; corrections are welcome.
Open access. Rights remain with the author.
Preface
A neural network architecture is a smooth map F:Θ×X→Y: a family of maps from inputs to outputs, parametrized by weights. Differential topology, in the form Hirsch gave it, is the study of what is true of a generic member of a family of maps and of what happens along a generic path in one. The fit is close enough that most of the qualitative questions asked about networks have answers that are corollaries of theorems proved before 1976, once the questions are phrased as questions about families.
Is the decision boundary of a classifier a manifold? For almost every value of the parameters, yes, by the parametric transversality theorem, provided the last layer has a bias. How does the boundary change during training? By ambient isotopy, except at isolated times, at each of which it undergoes one surgery whose type is the index of a critical point; this is Morse theory on the trace of the boundary. What is the set of parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary differential equation separate two linked circles? Not in dimension three, because its time-one map is ambient isotopic to the identity; in dimension four always, because there any two embeddings of a one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the companion volume, and this book supplies the paragraphs.
The higher-categorical layer of the companion volume enters in three places. A parametrized network is a plot of the mapping object [X,Y] in smooth sets, and composition of layers is composition in a category of parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an architecture make the parameter space an atlas for a stack whose points are networks up to relabelling, and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is what one sees of them from nearby.
The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a rate of convergence or a generalization error. Activation functions are assumed smooth except where stated, and the piecewise-linear case, which needs a different transversality theory, is treated only by reference. What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks the examples.
Introduction
What is assumed
The companion volume [9], at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of Hirsch’s book. From machine learning, only the definition of a feedforward network and of gradient descent.
How the book is organized
Part I sets up families, their derivatives and their symmetries. Part II studies level sets: of the network in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss in parameter space (the interpolation manifold). Part III studies the maps themselves: when they are embeddings, when they are isotopic, when they are invertible. Part IV is an assessment. gives the dictionary between the two subjects in one table.
Conventions
Manifolds and maps are smooth. A family is a smooth map F:Θ×X→Y; we write fθ=F(θ,−). P=dimΘ is the number of parameters, d=dimX the input dimension, n the number of data points. “Almost every” refers to Lebesgue measure on a Euclidean parameter space, and “generic” means belonging to a residual set whose complement has measure zero. An activation function σ:R→R is smooth unless it is explicitly ReLU(s)=max(s,0). Citations of the form [9, 4.5] refer to numbered statements of the companion volume.
Families
Families of Maps
Parametrized maps
Definition 1.1. A parametrized map from X to Y is a pair (Θ,F) of a manifold Θ and a smooth map F:Θ×X→Y. Its members are the maps fθ=F(θ,−), and its realization map is θ↦fθ, from Θ to C∞(X,Y).
By [9, Thm. 1.6] a parametrized map is exactly a plot of the mapping object [X,Y] in smooth sets, with domain Θ; the realization map is the function it induces on points. Everything in this book is a statement about plots of mapping objects. The topologies on C∞(X,Y) play no direct role, since parameter spaces are finite-dimensional; what is used is the relation between properties of the total map F and properties of its members.
Definition 1.2. Para is the bicategory whose objects are manifolds, whose morphisms X→Y are parametrized maps, and whose 2-cells (Θ′,F′)⇒(Θ,F) are smooth maps r:Θ′→Θ with F′=F∘(r×idX). The composite of (Θ,F):X→Y and (Θ′,G):Y→Z is (Θ×Θ′,(θ,θ′,x)↦G(θ′,F(θ,x))).
Composition is associative up to the canonical reassociation of products, and the identity of X is parametrized by a point. The construction is due to Fong, Spivak and Tuyéras [15] and is developed in [5]. A 2-cell is a reparametrization: sharing weights between layers is the 2-cell induced by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.
Proposition 1.3. [reformulation; cf. [9, Thm. 1.6]; [15]] Composition in Para is the internal composition [X,Y]×[Y,Z]→[X,Z] of smooth sets, evaluated on plots: the composite of two parametrized maps is the plot Θ×Θ′→[X,Y]×[Y,Z]→[X,Z]. Hence Para is the bicategory of plots of the category of manifolds enriched in smooth sets.
Proof. The internal composition is the map of sheaves which on plots with domain U sends (F,G) to (u,x)↦G(u,F(u,x)). Precomposing with the product plot, whose components depend on separate factors of U=Θ×Θ′, gives the formula of 1.2. ◻
Architectures
Definition 1.4. A feedforward architecture with widths (n0,…,nL) and activation σ is the composite in Para of the layers ℓiℓL:Rni−1→Rni,:RnL−1→RnL,ℓi(Wi,bi;x)ℓL(WL,bL;x)=σ(Wix+bi)(i<L),=WLx+bL, with σ applied to each coordinate. Its parameter space is Θ=∏i(Rni×ni−1×Rni), of dimension P=∑ini(ni−1+1). The number of hidden units is H=n1+⋯+nL−1.
If σ is smooth the family is smooth, and real-analytic if σ is (as tanh, the logistic function and softplus are). For σ=ReLU the family is continuous and piecewise polynomial in (θ,x), and it is not a morphism of Para as defined; statements about it are made separately. Convolutional and attention layers are parametrized maps in the same sense, with 2-cells expressing weight sharing. Nothing below depends on the feedforward form except where an explicit hypothesis on the last or first layer is made.
The image of the realization map, the set F={fθ} of functions the architecture can compute, is a subset of C∞(X,Y). With the plots that factor locally through Θ it is a diffeological space, the quotient of Θ by the relation fθ=fθ′. It is in general not a manifold; 3 describes its singularities.
Data, loss, and training
Fix inputs x1,…,xn∈X and targets y1,…,yn∈Y=Rm. The evaluation map is ev:Θ→Yn=Rnm,ev(θ)=(fθ(x1),…,fθ(xn)), and the squared loss is L(θ)=21∣ev(θ)−y∣2. More generally a loss is L=λ∘ev for a smooth λ:Rnm→R. A training path is a smooth curve γ:[0,T]→Θ; gradient flow is the curve with γ˙=−∇L(γ), for the Euclidean metric on Θ. A path in Θ is a plot [0,T]→[X,Y]: a one-parameter family of maps. When every member is an embedding of a submanifold K⊂X it is an isotopy of K in the sense of [9, Def. 10.1].
Three kinds of question organize the rest of the book.
In each case the answer comes from the same mechanism. A condition on a member, or on a parameter, is the condition that a point lies in the preimage of a submanifold under a map built from F; the preimage is a manifold when the map is transverse to the submanifold; and transversality of the map built from F can be checked on the architecture, usually by exhibiting one direction in parameter space, a bias, in which the map moves freely.
Principle 1.5 (Richness). [reformulation; cf. [9, Thm. 4.5]] Let F:Θ×X→Y be a family and A⊂Y a submanifold. If F is transverse to A, in particular if F is a submersion, then fθ is transverse to A for almost every θ. A feedforward architecture is a submersion onto Y=RnL because of the last-layer bias: ∂F/∂bL=id.
This is [9, Thm. 4.5] with its hypothesis verified. It is worth stating separately because the hypothesis is about the family and not about function space. A statement that a property is generic among all smooth maps (for instance, that Morse functions are dense) says nothing about a given architecture, whose realization is a thin subset of all maps. Genericity transfers to an architecture only through a transversality property of F itself, and each use of 1.5 below names the parameters that provide it.
The categorical treatment of learners as parametrized maps with update and request functions begins with [15]; the bicategory Para and its combination with lenses and reverse derivatives is in [5]. The point of view that a network is a family in the sense of singularity theory, to which transversality applies, is implicit in much work on loss landscapes, and explicit for piecewise-linear networks in [18].
Exercises
Exercise 1.1. Compute P and H for the architectures with widths (2,3,1), (2,4,4,1) and (d,k,1).
Exercise 1.2. Verify that composition in Para is associative up to the reassociation (Θ×Θ′)×Θ′′≅Θ×(Θ′×Θ′′), and that this isomorphism is a 2-cell in both directions.
Exercise 1.3. Let (Θ,F):X→X be a layer. Express the weight-tied network fθ∘fθ as the source of a 2-cell into the composite (Θ×Θ,F∘F), and show that its derivative with respect to θ is the sum of the two partial derivatives of the untied network on the diagonal.
Exercise 1.4. Give an example of a family F:R×R→R no member of which is a Morse function, although Morse functions are dense in C∞(R,R). Which hypothesis of 1.5 fails for the relevant jet condition?
Derivatives
Two derivatives of a family matter: with respect to the input, which governs the geometry of each member, and with respect to the parameters, which governs training. Both are instances of the tangent functor of [9, Ch. 2], and the algorithm that computes the second efficiently is the functoriality of its transpose.
Backpropagation
Recall from [9, Def. 2.10] the category Lens and from [9, Prop. 2.11] the functor f↦(f,R[f]), R[f](x,w)=(Dfx)Tw, from Euclidean spaces to lenses. For a parametrized map F:Θ×X→Y the reverse derivative has two components, R[F](θ,x,w)=(RΘ[F](θ,x,w),RX[F](θ,x,w))=((DθF)Tw,(DxF)Tw), the first being the update direction for the parameters and the second the covector passed back to the previous layer.
Theorem 2.1 (Backpropagation is a functor). [literature; Fong–Spivak–Tuyéras [15]; Cruttwell et al. [5]] The assignment (Θ,F)↦(Θ,(F,R[F])) is a functor of bicategories from Para over Euclidean spaces to parametrized lenses. In particular, for a composite of layers F=ℓL∘⋯∘ℓ1 the component RΘi[F] is obtained by composing the lenses of the layers: a forward pass computing the activations xi=ℓi(θi,xi−1), and a backward pass computing wi−1=RX[ℓi](θi,xi−1,wi) and emitting RΘ[ℓi](θi,xi−1,wi).
Proof. Composition in Para is composition of maps after taking products with identities, and R is a product-preserving functor to lenses by [9, Prop. 2.11]. ◻
The gradient of a loss L=λ∘ev is R[ev](θ,∇λ). The theorem says the cost of the gradient is that of one forward and one backward pass, whatever the number of parameters, which is the practical content of reverse-mode differentiation. The script implements both modes for a small network and checks them against each other and against finite differences.
The Jacobian of the evaluation map
Definition 2.2. For data x1,…,xn the parameter Jacobian is J(θ)=Devθ, an nm×P matrix. The tangent kernel is the nm×nm matrix K(θ)=J(θ)J(θ)T.
Proposition 2.3. [classical; Jacot–Gabriel–Hongler [23]] K(θ) is symmetric positive semidefinite of rank rkJ(θ). Along gradient flow of L=λ∘ev the outputs u(t)=ev(γ(t)) satisfy u˙=−K(γ)∇λ(u).
Proof.u˙=Jγ˙=−J∇L=−JJT∇λ. ◻
So the tangent kernel is the Riemannian metric that the Euclidean metric of Θ induces, by pushing forward along ev, on the cotangent directions of output space; where J is onto, K is invertible and the outputs follow a gradient flow for the metric K−1.
Rank
The rank of J is the first invariant of a parameter. It is lower semicontinuous, and for analytic architectures its generic value is attained almost everywhere.
Proposition 2.4. [classical; ] Let F be real-analytic in θ on a connected Θ, and let r be the maximum of rkJ(θ). Then {θ:rkJ(θ)=r} is open and dense and its complement has measure zero. On it the fibres of ev are submanifolds of dimension P−r, and ev is locally a submersion onto an r-dimensional submanifold of Rnm.
Proof. The complement is the common zero set of the r×r minors of J, which are analytic and not all identically zero. The zero set of a non-zero analytic function on a connected domain is closed with empty interior and measure zero. The rest is the constant rank theorem. ◻
Definition 2.5. The functional dimension of the architecture at θ is the supremum over all finite sets of inputs of rkJ(θ): the rank of the differential of the realization map [19].
The functional dimension is at most P, and 3 shows when it is smaller. The strata {rkJ=s} for s<r are where the fibres of ev jump in dimension. They contain the parameters at which hidden units are redundant, and they are responsible for most of what is singular in the loss landscape (3).
Second derivatives
By the composition law for 2-jets [9, Prop. 2.9], the Hessian of a composite λ∘ev is ∇2L(θ)=JT∇2λJ+k∑∂uk∂λ∇2evk(θ), a Gauss–Newton term and a term weighted by the gradient of λ in output space.
Proposition 2.6. [classical; ] At a parameter where ∇λ(ev(θ))=0, in particular at a zero of the squared loss, ∇2L=JT∇2λJ. For the squared loss this is JTJ, positive semidefinite of rank rkJ, with kernel kerJ.
The non-zero eigenvalues of JTJ and of the tangent kernel JJT coincide. So at an interpolating parameter the curvature of the loss in the directions normal to the solution set is the spectrum of the tangent kernel.
Notes
Reverse-mode differentiation has been rediscovered several times; Griewank’s history is the standard reference. The compositional formulation is from [15] and [5], and the axiomatics of reverse derivatives from [4]. The tangent kernel was introduced in [23]. Functional dimension for ReLU networks is studied in [19].
Exercises
Exercise 2.1. Write out the lens of a layer ℓ(W,b;x)=σ(Wx+b), and compose two of them to obtain the gradient of 21(aTσ(Wx+b)+c−y)2 with respect to (W,b,a,c).
Exercise 2.2. Show that rkK(θ)=rkJ(θ) and that, for n data points and scalar output, K(θ) is invertible if and only if the functions ∂F/∂θk(θ,⋅), k=1,…,P, restricted to the data, span Rn.
Exercise 2.3. Derive the formula for ∇2(λ∘ev) from the composition of 2-jets.
Exercise 2.4. For the linear family F(θ,x)=⟨θ,ϕ(x)⟩ with a fixed feature map ϕ:X→RP, compute J, K, the fibres of ev, and the functional dimension.
Symmetries and the Moduli of Networks
The realization map θ↦fθ is never injective. Its fibres are the first thing to understand about an architecture, since every function on parameter space that is defined in terms of the network’s input–output behaviour, such as the loss, is constant on them. This chapter describes the fibres for networks with one hidden layer, where the theory is complete, and draws the consequences that hold in general.
Symmetries
Definition 3.1. A symmetry of a family F:Θ×X→Y is a diffeomorphism g of Θ with F(gθ,x)=F(θ,x) for all θ,x: an invertible 2-cell of (Θ,F) in Para.
For a hidden unit j in layer i of a feedforward architecture, call its incoming parameters the j-th row of Wi and the j-th entry of bi, and its outgoing parameters the j-th column of Wi+1.
Proposition 3.2. [classical; ] Every feedforward architecture has the following symmetries.
Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.
If σ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.
If σ is positively homogeneous of degree one, as ReLU is: for each hidden unit and λ>0, multiplication of the incoming parameters by λ and of the outgoing ones by λ−1.
For σ=tanh these generate a finite group G of order ∏i=1L−12nini!.
One hidden layer
Consider F(θ,x)=∑j=1Hajσ(⟨wj,x⟩+bj)+c on X=Rd, with θ=(wj,bj,aj)j≤H and c, so P=H(d+2)+1, and σ=tanh.
Definition 3.3. The parameter θ is irreducible if every aj=0, every wj=0, and (wj,bj)=±(wk,bk) for j=k. The set of irreducible parameters is Θirr.
Theorem 3.4 (Identifiability). [literature; Sussmann [30]; Fukumizu [16]] Let θ be irreducible. Then fθ′=fθ for a parameter θ′ with the same number of hidden units if and only if θ′=gθ for some g∈G. Moreover θ is irreducible if and only if the functions ∂F/∂θk(θ,⋅), k=1,…,P, are linearly independent; so the functional dimension is P exactly on Θirr.
For deep networks with tanh the corresponding statement holds for generic parameters, by a theorem of Fefferman [13]; for ReLU networks see [28]. The complement of Θirr is a finite union of proper linear subspaces, so Θirr is open and dense, and G acts freely on it.
Proposition 3.5 (The moduli of irreducible networks). [observation] The quotient Θirr/G is a manifold of dimension P on which the realization map is injective. Let Z={(w,b,a):w=0,a>0} and Z′={(w,b):w=0}/±. Then Θirr/G≅{sets of H points of Z with pairwise distinct images in Z′}×R. If d≥2 it is connected and its fundamental group maps onto the symmetric group SH.
Proof. A free action of a finite group has a manifold quotient. Using the sign change to make each aj positive identifies Θirr/(Z/2)H with the ordered H-tuples in Z having distinct images in Z′, times the line of the output bias; dividing by SH forgets the order. For d≥2, Z is connected, and the tuples with a coincidence in Z′ form a closed subset of codimension d+1≥2 in ZH, so the space of ordered tuples is connected. A connected covering with group SH gives a surjection from the fundamental group of the base. ◻
So there are loops of networks, along which the function computed changes continuously and returns to itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged roles. An explicit one moves (w1,b1,a1) to (w2,b2,a2) and back along two different arcs. The space of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.
Proposition 3.6 (Fibres at reducible parameters). [classical; cf. [30], [16], [29]] At a reducible parameter the fibre of the realization map has positive dimension:
if aj=0, the fibre contains the affine space of all values of (wj,bj), of dimension d+1;
if (wj,bj)=(wk,bk), it contains the line aj+ak=const;
if wj=0, it contains the surface ajσ(bj)+c=const in the coordinates (aj,bj,c).
Each case is a way for a network with H hidden units to compute a function that needs fewer. The set of parameters in ΘH computing a given function of an irreducible network with H′<H units is a union of such families, permuted by G; it is the subject of a detailed combinatorial study in [29]. At these parameters the rank of J drops, by 3.4. The image F of the realization map is therefore a P-dimensional manifold Θirr/G together with a singular set, of lower dimension, onto which the reducible strata of Θ are collapsed.
Homogeneous activations
Proposition 3.7. [literature; cf. [19]] Let σ be positively homogeneous of degree one. For each hidden unit j let ξj be the vector field on Θ equal to the incoming parameters of j in the incoming coordinates, minus the outgoing parameters in the outgoing coordinates, and zero elsewhere. Then DθF(θ,x)ξj(θ)=0 wherever F is differentiable. Consequently the functional dimension of the architecture is at most P−H at every parameter.
Proof.ξj is the generator of the one-parameter group of symmetries of 3.2(iii); differentiate F(gλθ,x)=F(θ,x) at λ=1. The ξj for different units have disjoint supports in the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and the bound holds on the closure of that set by semicontinuity of rank. ◻
The script confirms both statements on small networks: for tanh the Jacobian has full rank P at a random parameter and loses rank at reducible ones; for ReLU its rank never exceeds P−H and the fields ξj lie in its kernel.
For ReLU the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space: the orbit of a unit with zero outgoing weight is not closed, since (λw,λb,0)→(0,0,0) as λ→0. The correct object is the quotient stack, in which such points are remembered with their stabilizers. This is the same reason that vector bundles in [9, Ch. 5] formed a stack and not a space, and the loss is a function on it.
Invariant functions
Proposition 3.8. [classical; ] Let L:Θ→R be invariant under a group G of symmetries. Then the critical set of L is a union of G-orbits. If G contains a one-parameter group with generator ξ, then dL(ξ)=0 identically and, at every critical point, ξ lies in the kernel of the Hessian. If moreover ∇λ vanishes at ev(θ) for a loss L=λ∘ev, then the whole fibre of ev through θ is critical.
Proof. Differentiating L(gsθ)=L(θ) in s gives dLθ(ξ)=0 for all θ; differentiating that identity in a direction v at a critical point gives ∇2L(ξ,v)+dL(Dvξ)=∇2L(ξ,v)=0. The last statement is ∇L=JT∇λ. ◻
Corollary 3.9. [observation] For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the form λ∘ev is a Morse function on any open set containing a critical point at which some hidden unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least the number of such units.
So Morse theory in the form of [9, Ch. 8] never applies directly to the loss of a ReLU network. It can apply to the function induced on a slice transverse to the orbits, or in Morse–Bott form, with critical manifolds in place of critical points. For tanh networks there is no continuous symmetry and no such obstruction on Θirr; the degeneracies there come from the reducible strata and from overparametrization (3).
Notes
Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann and by Albertini and Sontag, and for deep tanh networks by Fefferman. The relation between reducibility and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of Watanabe’s singular learning theory [31]. The use of permutation symmetry to compare trained networks is discussed in 3.
Exercises
Exercise 3.1. Verify the three symmetries of 3.2 for a network with one hidden layer.
Exercise 3.2. For the one-hidden-layer tanh network, show that if aj=0 then rkJ(θ)≤P−(d+1) for every set of inputs, and if (wj,bj)=(wk,bk) then rkJ(θ)≤P−1.
Exercise 3.3. For F(θ,x)=aReLU(wx+b)+c with d=1, write down ξ and verify DθF⋅ξ=0 by hand. What is the functional dimension?
Exercise 3.4. Compute the order of G for the tanh architectures with widths (2,3,1) and (2,4,4,1).
Level Sets
Transversality and Decision Boundaries
A binary classifier is a function fθ:X→R whose sign is the predicted class, and its decision boundary is Bθ=fθ−1(0). This chapter answers question (Q1) of 1 for decision boundaries: for almost every parameter the boundary is a smooth hypersurface; its isotopy type is constant on the components of the complement of a closed null set of parameters; and every isotopy type of compact hypersurface occurs for wide enough networks.
The boundary is generically a hypersurface
Theorem 1.1 (Generic decision boundaries). [reformulation; cf. [9, Thm. 4.1, Thm. 1.12]] Let F:Θ×X→R be a family of the form F((θ′,c),x)=G(θ′,x)+c.
For every θ′, the set of c for which 0 fails to be a regular value of f(θ′,c) has measure zero.
For almost every θ, the decision boundary Bθ is a smooth hypersurface in X, closed as a subset and possibly empty, with trivial normal bundle oriented by ∇fθ; and X∖Bθ is the disjoint union of the open sets {fθ>0} and {fθ<0}.
If X is a compact manifold with boundary, then for almost every θ the boundary Bθ is moreover a neat submanifold: it meets ∂X transversally, in ∂Bθ.
Proof. (a) 0 is a regular value of G(θ′,⋅)+c exactly when −c is a regular value of G(θ′,⋅), and the critical values of a smooth function have measure zero by [9, Thm. 4.1]. (b) follows from (a) by Fubini’s theorem and [9, Thm. 1.12]. (c) Apply (a) also to the restriction to ∂X and use [9, Thm. 1.17]. ◻
Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever smooth activation it uses. Part (a) is stronger than the conclusion of 1.5, since it holds for each value of the other parameters separately; it says that a non-smooth decision boundary can always be repaired by an arbitrarily small change of the output bias alone. The level sets fθ−1(s) for other thresholds s are covered by the same statement.
Example 1.2. Let f(x)=x12−x22+c. For c=0 the boundary is a hyperbola, a 1-manifold with two components. For c=0 it is a pair of crossing lines, not a manifold. In the language of [9, Ch. 4], the boundary at c=0 is a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.
Several classes
A classifier with k classes is a map fθ:X→Rk, and the predicted class is the index of the largest coordinate. For a non-empty I⊂{1,…,k} let AI⊂Rk be the set of points whose largest coordinate is attained exactly on I. Then AI is an open subset of the linear subspace {yi=yj:i,j∈I}, a submanifold of codimension ∣I∣−1, and the AI partition Rk.
Proposition 1.3. [reformulation; cf. [9, Thm. 4.5]] Let F:Θ×X→Rk be a family whose last layer is affine with a bias. For almost every θ, fθ is transverse to every AI. Then the decision region of class i is the open set fθ−1(A{i}); the set where exactly the classes in I tie is a submanifold of codimension ∣I∣−1; and no more than d+1 classes tie anywhere.
Proof.F is a submersion, so [9, Thm. 4.5] applies to each of the finitely many AI, and a finite intersection of sets of full measure has full measure. A submanifold of codimension greater than d is empty. ◻
In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time at isolated points, like the cells of a foam, and four regions never meet at a point.
Chambers
Let X be compact, and for simplicity without boundary. Define Σ={(θ,x):F(θ,x)=0,dxF(θ,x)=0}⊂Θ×X,Δ=π(Σ)⊂Θ, the set of parameters for which 0 is not a regular value: the discriminant of the family.
Theorem 1.4 (Chambers). [reformulation; cf. [9, Lem. 4.4, Thm. 8.5, Thm. 10.3]] Let F be as in 1.1 with X compact. Then Δ is closed and of measure zero. If γ is a path in Θ∖Δ, there is a diffeotopy φt of X with φt(Bγ(0))=Bγ(t). In particular the diffeomorphism type of the pair (X,Bθ) is constant on each connected component of Θ∖Δ.
Proof.Σ is closed and π∣Σ is proper because X is compact, so Δ is closed; its complement has full measure by 1.1. Over Θ∖Δ the set W=F−1(0) is a submanifold of Θ×X, since dxF=0 there, and by [9, Lem. 4.4] the projection W→Θ∖Δ is a submersion; it is proper. By Ehresmann’s theorem [9, Thm. 8.5] it is a fibre bundle. Pulled back along γ it is trivial, which gives an isotopy of embeddings Bγ(0)→X with image Bγ(t) at time t, and the isotopy extends to a diffeotopy of X by [9, Thm. 10.3]. ◻
The components of Θ∖Δ are the chambers of the architecture. The theorem says that a classifier has a well-defined topology (the number of components of its decision boundary and of its decision regions, their genera, how they are nested) which does not change under small changes of the parameters, and can change only when the parameter crosses Δ. What happens at a crossing is the subject of the next chapter. The symmetry group of 3 permutes the chambers.
For manifolds with boundary the same holds with Δ enlarged by the parameters for which Bθ is tangent to ∂X.
Proposition 1.5 (Walls). [observation] Consider the one-hidden-layer architecture F(θ,x)=∑j≤Hajσ(⟨wj,x⟩+bj)+c on Rd with σ′>0. At every point where the matrix W with rows wj has rank d, the map Φ(θ,x)=(F,dxF)∈R×Rd is a submersion. Hence, over the open set of such parameters, Σ is a submanifold of dimension P−1, and Δ is the image of a manifold of dimension P−1.
Proof.dxF=WT(a⊙σ′(z)) with z=Wx+b. A variation δa changes dxF by WT(δa⊙σ′(z)), and δa↦δa⊙σ′(z) is a linear isomorphism since σ′>0; so if WT is onto, every variation of dxF is produced. The resulting change in F is compensated by δc. ◻
So the discriminant is no larger than a hypersurface, and the picture of parameter space is that of open chambers separated by walls of codimension one. The proposition is stated for the simplest architecture; the argument needs only that the parameters can move the value and the input-gradient of the network independently at each point, which holds for deeper networks at parameters where the layer Jacobians have full rank.
Every hypersurface is a decision boundary
Theorem 1.6 (Realization). [reformulation; cf. [9, Thm. 6.2]; Hornik [21]] Let K⊂Rd be a compact domain with smooth boundary and B⊂intK a closed hypersurface. Let σ be smooth, bounded and non-constant. Then for all sufficiently large H there is a non-empty open set of parameters θ of the one-hidden-layer architecture with H units for which Bθ∩K is carried to B by a diffeomorphism of K isotopic to the identity.
Proof. A closed hypersurface in Rd separates and so has a trivial normal bundle; using a tubular neighbourhood [9, Thm. 6.2] one constructs a smooth g:K→R with g−1(0)=B, 0 a regular value, and g=0 on ∂K. Since B is compact, there is ε>0 such that every h with ∥h−g∥C1(K)<ε has 0 as a regular value and does not vanish on ∂K. By Hornik’s theorem, networks with one hidden layer are dense in C1(K), so some fθ0 is within ε of g, and so is fθ for θ near θ0, the realization map being continuous into C1(K). The family (1−s)g+sfθ, s∈[0,1], stays within ε of g, and the argument of 1.4 applied to it gives the diffeotopy. ◻
So there is no topological restriction on decision boundaries beyond those satisfied by every compact hypersurface in Rd: it is orientable, it separates, and it bounds (2.6). The theorem is silent on how large H must be. Upper bounds for the Betti numbers of decision regions in terms of the architecture, which give lower bounds for H, are due to Bianchini and Scarselli [2].
Piecewise-linear networks
For σ=ReLU a network is piecewise linear, its decision boundary is a polyhedral complex, and the smooth theory does not apply as it stands. The counterpart of 1.1 is due to Grigsby and Lindsey [18]. They define what it means for a ReLU network to be generic and transversal, in terms of the bent hyperplane arrangement formed by the zero sets of its units, and prove that almost all ReLU networks have both properties. As an application they show that for the architecture with one hidden layer of d+1 units, a decision region has at most one bounded connected component: a restriction on topology coming from the architecture, of the kind 1.6 rules out for unbounded width.
Notes
That decision boundaries are generically manifolds is folklore, and is usually justified by Sard’s theorem in the form (a). The statement about chambers is the standard structure theory of discriminants in singularity theory, applied to this family. For topological complexity of decision regions see [2], and for an empirical study of how networks change the topology of data, [27].
Exercises
Exercise 1.1. Let d=1 and X=[−R,R]. Show that for almost every θ the decision boundary is a finite set at which fθ changes sign, and that the parity of its cardinality is determined by the signs of fθ(±R).
Exercise 1.2. Let X be a closed disc in the plane. Describe the generic decision boundary up to diffeomorphism.
Exercise 1.3. For the family F(c,x)=x12−x22+c on the plane, determine Σ and Δ, and describe the chambers.
Exercise 1.4. For a three-class classifier in the plane satisfying the conclusion of 1.3, show that near a point where all three classes tie the decision regions look like three sectors meeting at a point.
Topology Change along Paths: Surgery and Cobordism
By 1.4 the topology of a decision boundary can change during training only when the parameter crosses the discriminant. This chapter answers question (Q2): for a generic path the crossings are isolated, and at each of them the boundary changes by one surgery, determined by the index of a critical point. The union of the boundaries along the path is a cobordism, and the surgeries are the critical points of its height function. This is the Morse theory of [9, Ch. 8] applied to a manifold that the path itself creates.
The trace of a path
Let F:Θ×X→R be a family as in 1.1, with X compact of dimension d, and let γ:[0,1]→Θ be a path. Write ft=fγ(t) and Bt=ft−1(0), and define Φγ:[0,1]×X→R,Φγ(t,x)=F(γ(t),x);Wγ=Φγ−1(0)⊂[0,1]×X.
Theorem 2.1 (The trace is a cobordism). [reformulation; cf. [9, Thm. 1.17, Thm. 4.5]] If 0 is a regular value of Φγ and of f0 and f1, then Wγ is a compact neat hypersurface in [0,1]×X with ∂Wγ=B0⊔B1: a cobordism from B0 to B1 embedded in [0,1]×X. For a path with endpoints outside Δ, this holds after an arbitrarily small perturbation of the bias component of γ fixed at the endpoints.
Proof. The first statement is [9, Thm. 1.17]. For the second, replace the bias component c(t) by c(t)+sβ(t) with β>0 on (0,1) and vanishing with all derivatives at the ends. The map (s,t,x)↦Φγ(t,x)+sβ(t) is a submersion on {0<t<1}, so by [9, Thm. 4.5] zero is a regular value for almost every s; near t=0,1 it is one already because f0,f1 have 0 as a regular value. ◻
The times at which γ meets Δ are, by [9, Lem. 4.4], exactly the critical values of the height function τ:Wγ→[0,1], τ(t,x)=t. At such a point (t0,x0) we have dxft0(x0)=0: x0 is a critical point of ft0 with critical value 0.
Surgery
Theorem 2.2 (One surgery at a time). [reformulation; cf. [9, Thm. 8.6, Cor. 8.7]] Let γ be a path for which Wγ is a manifold and τ is a Morse function with distinct critical values t1<⋯<tr in (0,1). Then:
On each interval of [0,1]∖{t1,…,tr} the boundary Bt moves by an ambient isotopy of X.
At each ti there is exactly one point xi∈Bti at which Bti fails to be a manifold. It is a non-degenerate critical point of fti with ∂tΦγ(ti,xi)=0, and near (ti,xi) there are coordinates on X in which Φγ(t,x)=±(t−ti)−x12−⋯−xk2+xk+12+⋯+xd2, with k the index of xi as a critical point of fti.
Bti+ε is obtained from Bti−ε by a surgery: a copy of Sλ−1×Dd−λ is removed and Dλ×Sd−λ−1 is glued in, where λ is the index of τ at (ti,xi); λ=k or d−k according to the sign in (b). Correspondingly the region {ft≤0} gains or loses a handle of index k.
Proof. (a) is 1.4. For (b): at a critical point of τ on Wγ the tangent space of Wγ is {dt=0}, so dxΦγ=0 and, Wγ being a manifold, ∂tΦγ=0. Solving Φγ=0 for t as a function of x near the point exhibits Wγ as a graph over X on which τ is the function t(x), whose Hessian at xi is −∇2fti(xi)/∂tΦγ. So τ is non-degenerate there exactly when xi is a non-degenerate critical point of fti, and the normal form follows from the Morse lemma with parameters. (c) is [9, Thm. 8.6] for τ on Wγ: the level sets of τ are the Bt, and passing a critical point of index λ changes the level set by the stated surgery. ◻
In words: the topology of a decision boundary changes exactly when a critical point of the network function crosses the threshold, and the change is read off from the index of the critical point and the direction of crossing. lists the cases for inputs of dimension two and three.
Elementary changes of topology of a decision region, by the index of the critical point crossing zero from above. Crossing in the other direction reverses each.
d
index k of xi
effect on the region {ft≤0} when the critical value becomes negative
2
0 (minimum)
a new disc appears; the boundary gains a circle
2
1 (saddle)
two parts of the region join, or one part meets itself; two boundary arcs reconnect, so the number of boundary circles changes by ±1
2
2 (maximum)
a hole in the region is filled; a boundary circle disappears
3
0
a new ball appears; the boundary gains a sphere
3
1
a tube joins two parts of the region, or adds a handle to one
3
2
a tunnel through the region is closed by a disc, or a cavity is split in two
3
3
a cavity is filled; a boundary sphere disappears
Proposition 2.3. [observation] In the situation of 2.2(c), χ(Bti+ε)−χ(Bti−ε) equals 2(−1)λ if d is odd and 0 if d is even, and the Euler characteristic of the region {ft≤0} changes by ±(−1)k.
Proof. The change is χ(Dλ×Sd−λ−1)−χ(Sλ−1×Dd−λ)=χ(Sd−λ−1)−χ(Sλ−1)=(−1)d−λ−1−(−1)λ−1. The second statement is additivity of χ under attaching a k-cell. ◻
Example 2.4. Let F(t,x)=(x12−1)2+x22−t on the plane. The function (x12−1)2+x22 has minima at (±1,0) with value 0 and a saddle at the origin with value 1. For t<0 the region {F≤0} is empty. At t=0 two discs are born (two critical points cross at the same time, which is not generic; a small asymmetry separates them). At t=1 the saddle crosses and the two discs merge. The numbers of components are 0, 2, 1, and the Euler characteristics of the region 0, 2, 1. The script counts components on a grid.
Genericity of paths
Are the hypotheses of 2.2 generic? For paths in the space of all functions, yes: this is the content of [9, Thm. 8.12]. For paths in a given architecture one needs enough parameters to move critical points and critical values independently.
Proposition 2.5. [reformulation; cf. [9, Thm. 4.6, Thm. 8.12]] For the architecture of 1.5, with H≥d, let γ be a path with endpoints in Θ∖Δ along which W has rank d. Then arbitrarily close to γ, with the same endpoints, there is a path satisfying the hypotheses of 2.2.
Sketch. By 1.5 the map (θ,x)↦(F,dxF) is a submersion along the path, so the family of perturbations of γ by compactly supported variations of (a,c) induces a family of maps (t,x)↦(Φ,dxΦ) transverse to the zero section; by [9, Thm. 4.5] a generic member has Σ pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at those points, and distinctness of the times, are further conditions of positive codimension on the 2-jet, which a variation of W and b moves freely; [9, Thm. 4.6] applies. ◻
A training path produced by gradient descent is one particular path, and nothing guarantees that it is generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the typical exception (2.4): a symmetric network crossing a wall has several critical points crossing together.
What cobordism does and does not see
Proposition 2.6. [classical; ] Let d≥2 and let B=f−1(0)⊂Rd be compact with 0 a regular value. Then B is the boundary of a compact region of Rd; it is orientable, its normal bundle is trivial, and TB⊕R is trivial. In particular B is null-cobordant, all its Stiefel–Whitney numbers vanish, and no closed non-orientable surface is a decision boundary in R3.
Proof. Choose a closed ball D containing B in its interior. Its boundary sphere is connected and f does not vanish on it, so f has a constant sign there, say positive. Then {f≤0}∩D is a compact manifold with boundary B. The normal bundle is trivialized by ∇f, and TB⊕ν=TRd∣B is trivial. ◻
So the cobordism class of a decision boundary carries no information: it is always zero, as [9, Thm. 4.16] predicts for a fibre of a map to a contractible space. What carries information is finer: the diffeomorphism type of the pair (X,Bt) in each chamber, and the sequence of surgeries along a path.
That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between generic paths meets a short list of codimension-two events.
Proposition 2.7 (Moves). [reformulation; cf. Cerf; [9, Thm. 8.12]] Let γ0,γ1 be generic paths with the same endpoints in an architecture rich enough for jet transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or deleting a pair of consecutive surgeries of indices λ,λ+1 that cancel (a birth–death point of ft on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that touches zero and returns).
The proposition is the two-parameter analogue of 2.2, and is the form Cerf’s theorem takes for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and that anything invariant under them, for instance the parity of the number of surgeries of each index modulo the cancellation of adjacent pairs, depends only on the endpoints.
Notes
The description of how level sets change in one-parameter families is classical Morse and Cerf theory; see Milnor’s lectures on the h-cobordism theorem. Its use for decision boundaries during training appears in empirical form in studies that track Betti numbers of decision regions and of data representations, such as [27]. Persistent homology of the family {ft≤s} in the two parameters (t,s) is the natural computational tool, and the events of 2.7 are the generic singularities of its bifiltration.
Exercises
Exercise 2.1. Verify 2.4: find the critical points and indices of (x12−1)2+x22 and the topology of the sublevel sets.
Exercise 2.2. For d=1, show that along a generic path the points of the decision boundary are born and die in pairs, and that the parity of their number is constant if X is a circle.
Exercise 2.3. In d=3, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries, and what are their indices?
Exercise 2.4. Prove the formula for the Hessian of τ used in the proof of 2.2(b).
The Loss Landscape
Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the landscape is organized by the sublevel sets of the loss, to which the Morse theory of [9, Ch. 8] applies with two caveats established in 3 and below: the loss is rarely a Morse function, and parameter space is not compact.
The interpolation manifold
Let ev:Θ→RN be the evaluation map on n data points, N=nm, and L=21∣ev−y∣2. Write My=ev−1(y)=L−1(0).
For almost every y∈RN, My is a submanifold of Θ of dimension P−N, possibly empty. If P<N it is empty for almost every y.
If y is a regular value and θ∈My, then TθMy=kerJ(θ) and ∇2L(θ)=JTJ has rank N: it vanishes on TθMy and is positive definite on its orthogonal complement.
My is non-empty for y in the image of ev, and that image has non-empty interior if and only if J has rank N somewhere.
Proof. (a) By [9, Thm. 4.1] almost every y is a regular value of ev, and then [9, Thm. 1.12] applies; if P<N every point of the image is a critical value. (b) TθMy=kerDevθ by [9, Thm. 1.12], and the Hessian is given by 2.6. (c) If J has rank N at θ then ev is open near θ; conversely if the rank is everywhere less than N the image has measure zero. ◻
Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in addition that for sufficiently wide networks My is non-empty for generic data. In the overparametrized regime P>N, then, the global minima of the loss are not isolated: they form a manifold whose dimension is the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it. In the terminology of Morse theory the loss is a Morse–Bott function near My. By 2.6 its normal curvature is the spectrum of the tangent kernel.
Proposition 3.2 (Convergence near the interpolation manifold). [literature; Liu–Zhu–Belkin [25]] Let U⊂Θ be an open set on which the smallest eigenvalue of the tangent kernel K=JJT is at least μ>0. Then ∣∇L∣2≥2μL on U. A gradient flow line starting at θ0 satisfies L(γ(t))≤L(θ0)e−2μt and has length at most 2L(θ0)/μ for as long as it stays in U; if the ball of that radius about θ0 lies in U, the flow line converges to a point of My.
Proof.∣∇L∣2=(ev−y)TJJT(ev−y)≥μ∣ev−y∣2=2μL. Then dtdL=−∣∇L∣2≤−2μL, and dtdL=−∣∇L∣2/(2L)≤−μ/2∣∇L∣, which integrates to the bound on the length ∫∣γ˙∣=∫∣∇L∣. ◻
The inequality is the quantitative form of transversality: μ>0 says that ev is a submersion on U with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the qualitative theory hands over to analysis.
Sublevel sets and barriers
For c∈R let Θc={L≤c}. Two parameters are connected at level c if they lie in the same component of Θc, and the barrier between them is the infimum of such c. This is the notion studied empirically under the name of mode connectivity [17, 10].
Proposition 3.3 (Barriers). [reformulation; cf. [9, Cor. 8.7, Prop. 8.10]] Let L:Θ→R be a Morse function with compact sublevel sets. Then the components of Θc change only at critical values: a component is born at each local minimum, and two components merge at a critical point of index 1 whose two descending directions lead to different components. The barrier between two local minima is the level at which their components merge; it is attained at a critical point of index 1, and equals the minimum over paths joining them of the maximum of L along the path.
So barriers are the death times in the degree-zero barcode of the sublevel functor [9, Prop. 8.10], and the barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a generic analytic loss with weight decay, L+2ε∣θ∣2, on an architecture without continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail (3.5).
Example 3.4 (Two parameters). Let F(a,b;x)=abx with one data point (x,y)=(1,1), so L=21(ab−1)2. The interpolation manifold is the hyperbola ab=1, of dimension P−N=1, with two components exchanged by the symmetry (a,b)↦(−a,−b). The only other critical point is the origin, a saddle with L=21 and Hessian eigenvalues ±1. For c<21 the sublevel set has two components; at c=21 they merge. So the barrier between (1,1) and (−1,−1) is 21, attained on the straight segment between them, and no path does better. After applying the symmetry to one of the two points they coincide. The script checks the component counts on a grid.
The example is the smallest instance of a general phenomenon. By 3.8 minima come in orbits of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a barrier. The barrier between two trained networks should therefore be measured after choosing representatives of their orbits, which is the content of the conjecture of Entezari et al. that barriers largely disappear once permutations are accounted for [11]. In the language of 3, the relevant sublevel sets are those of the function induced on the quotient Θ/G.
Two results from the literature describe the overparametrized case. For a class of deep networks with piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen proved that every sublevel set of the loss is connected [26]. And Şimşek et al. describe the critical manifolds that the reducible strata of 3.6 contribute, and show how adding hidden units connects the ∣G∣ copies of a minimum [29].
Why the loss is rarely a Morse function
Three mechanisms make the loss degenerate, and they are independent.
Overparametrization. If P>N the zero-loss minima form the manifold My of positive dimension (3.1).
Continuous symmetry. For homogeneous activations every critical point has a Hessian of nullity at least H (3.9).
Reducible strata. A critical point of a smaller network gives a positive-dimensional family of critical points of a larger one, along the fibres of 3.6; these are typically saddles [12].
In each case the degeneracy is of Morse–Bott type on an open dense part of the critical set, and the handle theorem [9, Thm. 8.6] has a Morse–Bott version in which a critical manifold contributes a bundle of cells over itself. But the density of Morse functions [9, Cor. 4.8] is of no direct use: a generic perturbation of the loss as a function on Θ is not the loss of any data set, and perturbing the data preserves all three mechanisms. This is the point of 1.5 again. Genericity must be established within the family one actually has.
Critical points at infinity
Proposition 3.5. [observation] For architectures with a bounded activation the loss is not proper, and its infimum need not be attained. For F(w;x)=tanh(wx) with the single data point (1,1), L(w)=21(tanhw−1)2 has no critical point, its infimum 0 is not attained, and every sublevel set Θc with 0<c<2 is a half-line.
Proof.L′(w)=(tanhw−1)sech2w<0 for all w. ◻
So the topology of sublevel sets can change with no critical point to account for it, the change being carried off to infinity, and the regular interval theorem [9, Thm. 8.4], which needs compactness, does not apply. Any use of Morse theory for losses must either restore compactness (weight decay, a bounded parameter domain) or account for these ends explicitly, as one does for the Palais–Smale condition in infinite dimensions.
Notes
The manifold of global minima was described by Cooper [6]; the Polyak–Łojasiewicz condition in this setting is due to Liu, Zhu and Belkin. Mode connectivity was observed by Garipov et al. and Draxler et al., and analysed for wide networks by Freeman and Bruna and by Nguyen; permutation alignment is studied in [11]. For the singular structure of the landscape at reducible parameters see [12, 29] and Watanabe’s theory [31].
Exercises
Exercise 3.1. For the linear family of 2.4 with P>n and features in general position, describe My and the Hessian of the loss.
Exercise 3.2. Verify 3.4: find all critical points of 21(ab−1)2, their Hessians, and the components of the sublevel sets.
Exercise 3.3. In the setting of 3.2, show that the limit point of the flow line lies within 2L(θ0)/μ of θ0.
Exercise 3.4. For F(a1,…,ak;x)=a1⋯akx with data (1,1), show that My has 2k−1 components and dimension k−1, and that the origin is a degenerate critical point for k≥3.
Maps
Embeddings, Isotopy, and Depth
The previous three chapters studied level sets. This one studies the maps themselves, and in particular what a network does to a data set regarded as a subset of input space. The standing assumption, usually called the manifold hypothesis [14], is that the data lie on or near a compact submanifold K⊂RD of dimension m much smaller than D, with the classes occupying disjoint closed subsets K+,K−. A hidden layer of width k is a map hθ:RD→Rk, and the question is what hθ(K) can be.
The first layer generically embeds
Proposition 1.1. [reformulation; cf. Whitney; [9, Thm. 4.5, Cor. 4.8, Prop. 3.12]] Let K⊂RD be a compact submanifold of dimension m, let k≥2m+1, and let σ be a diffeomorphism of R onto an open interval. Then for almost every W∈Rk×D and every b, the map x↦σ(Wx+b) restricted to K is an embedding. The set of such (W,b) is open.
Proof. Applying σ coordinatewise after a translation is a diffeomorphism onto its image, so the question is whether W∣K is an embedding. The map (W,(x,y))↦W(x−y) on Rk×D×(K×K∖Δ) is a submersion to Rk, so by [9, Thm. 4.5] for almost every W its restriction to K×K∖Δ, a manifold of dimension 2m<k, is transverse to 0 and hence misses it: W∣K is injective. Likewise (W,(x,v))↦Wv on the unit tangent bundle of K, of dimension 2m−1, is a submersion, so for almost every W no unit tangent vector is killed: W∣K is an immersion. An injective immersion of a compact manifold is an embedding. Openness is [9, Prop. 3.12]. ◻
So a hidden layer wider than twice the intrinsic dimension of the data loses no topological information for almost every choice of weights, however small it is compared with the ambient dimension D. This is Whitney’s embedding theorem read as a statement about random projections. Narrower layers may fail to be injective on K, and that is where the topology of a representation can change.
What injective networks cannot do
A classifier ends with a linear map, so the last hidden representation must make the classes linearly separable. If the network up to that point is injective, its restriction to K is a homeomorphism onto the image, and separability becomes a constraint on how K+ and K− sit in the ambient space.
Theorem 1.2 (Nested classes). [classical; invariance of domain; cf. [24]] Let h:Rd→Rd be continuous and injective, let β⊂Rd be a closed topological ball with boundary sphere S, and let p be an interior point of β. Then h(p) lies in the convex hull of h(S). In particular no hyperplane separates h(p) from h(S).
Proof. Suppose a linear functional λ satisfies λ>0 on h(S) and λ(h(p))≤0. Let q be a point of the compact set h(β) at which λ is smallest. Then λ(q)≤0, so q∈/h(S) and q∈h(intβ), which is open in Rd by invariance of domain. But a linear functional has no local minimum on an open set. ◻
Corollary 1.3. [classical; [24, 22]] A feedforward network on Rd in which every hidden layer has width d, every weight matrix is invertible, and the activation is injective, cannot separate a point from a sphere surrounding it. To do so a network must either have a layer of width at least d+1 or use a layer that is not injective.
Both escapes are used in practice. A wider layer provides the extra dimension into which the inner class can be lifted. A ReLU layer is not injective: it folds space along hyperplanes, and a network of ReLU layers can map a nested data set to a separable one by collapsing. Naitzat, Zhitnikov and Lim [27] measured the Betti numbers of the two classes layer by layer and found that well-trained networks reduce them steadily, faster with ReLU than with smooth invertible activations. In the terms of this chapter: smooth invertible layers of constant width act by diffeomorphisms and cannot change topology, and the network must then rely on the few layers where injectivity fails.
Neural ordinary differential equations
A neural ODE [7] is the time-T map φθ of the equation x˙=vθ(x,t), where vθ is a network with input (x,t) and output in Rd.
Proposition 1.4. [reformulation; cf. [9, Thm. 10.2]] If vθ is smooth with bounded velocity, as it is when its last hidden layer has a bounded activation, then t↦φθt is a diffeotopy of Rd. Hence φθ is a diffeomorphism isotopic to the identity, it preserves orientation, and for every compact K⊂Rd the image φθ(K) is ambient isotopic to K.
Corollary 1.5 (Obstructions). [classical; cf. Dupont–Doucet–Teh [8]] A neural ODE on Rd cannot:
for d=1, reverse the order of two points;
map a sphere and a point inside it to linearly separable sets (1.2);
for d=3, map two linked circles to unlinked ones, or a knotted circle to a round one.
Proof. (a) A diffeomorphism of R isotopic to the identity is increasing. (c) The linking number is the degree of the Gauss map [9, Ex. 7.12] and is invariant under isotopy by [9, Thm. 7.2]; the knot type is an isotopy invariant by definition. ◻
These are statements about exact representation. A neural ODE can still approximate a map that violates them, at the cost of a vector field with large derivatives, and [8] observed that training then becomes slow and unstable. Their remedy was to append extra coordinates to the state.
Augmentation
Theorem 1.6 (Flows in the stable range). [reformulation; cf. [9, Thm. 10.9, Thm. 10.3, Thm. 10.2]] Let K be a compact manifold of dimension m and let e0,e1:K→RN be embeddings, with N≥2m+2. Then there is a smooth, compactly supported, time-dependent vector field on RN whose time-one map φ satisfies φ∘e0=e1.
Proof. By [9, Thm. 10.9] the two embeddings are isotopic. By the isotopy extension theorem [9, Thm. 10.3] the isotopy is induced by a diffeotopy of RN with compact support, and by [9, Thm. 10.2] the diffeotopy is the flow of its velocity field. ◻
Corollary 1.7 (Separation after augmentation). [observation] Let K=K+⊔K−⊂Rd be a compact submanifold of dimension m, the union of two disjoint compact submanifolds, and let N≥max(d+1,2m+2). Then there is a compactly supported flow on RN whose time-one map carries K+×{0} and K−×{0} into the open half-spaces {xN>0} and {xN<0}. The vector field may be taken to be a neural network with one hidden layer and input (x,t).
Proof. Take e0 the inclusion of K in Rd×{0}⊂RN and e1=e0±(0,…,0,1) on K±, and apply 1.6 to get a field v with flow φ. For the last statement, approximate v in C1 on a compact set containing the support, by Hornik’s theorem [21]; by Grönwall’s inequality the time-one maps are then close in C1 on compact sets, and the conclusion, being an open condition on the images of the compact sets K±, persists. ◻
So the obstructions of 1.5 disappear once the state space has dimension at least 2m+2, where m is the dimension of the data and not of the ambient space. For finite data sets m=0 and two dimensions suffice, in agreement with (a): points on a line cannot be reordered by a flow, points in the plane can. For curves the bound is 4, and it is sharp, since circles link in R3. For connected data of dimension m≥2 the bound can be lowered to 2m+1 by the second part of [9, Thm. 10.9]. Zhang et al. [32] prove a related statement about maps: every homeomorphism of Rd can be embedded in a flow on R2d, and hence approximated by a neural ODE or an invertible residual network acting there. The dimension count of 1.6 depends on the data alone.
Residual networks fit the same description. A block x↦x+g(x) with g of Lipschitz constant less than one is a diffeomorphism isotopic to the identity (2.2), and a composite of such blocks is a discretized isotopy. Depth, for such architectures, is time.
The space of representations
Open Problem 1.8. [programme] For a compact data manifold K of dimension m and width k≥m+3, the space Emb(K,Rk) of representations that preserve the topology of the data is described by manifold calculus [9, Thm. 10.10]: it is the limit of a tower whose first stage is the space of immersions and whose higher stages are built from configuration spaces of K. Determine which components and which homotopy classes of this space are reached by the realization map of a given architecture, as a function of width and depth.
For k≥2m+2 the space is connected and the question is about higher homotopy; in the range m+3≤k≤2m+1 already the set of components is interesting, and 1.5(c) is the first example outside that range.
Notes
The picture of a network as progressively untangling data manifolds goes back at least to expository writing of Olah in 2014, where 1.2 for d=2 and the role of an extra dimension are described. Minimal width for universal approximation is treated in [22, 24]. Neural ODEs are from [7], their limitations and augmentation from [8], and approximation properties from [32].
Exercises
Exercise 1.1. Show that the time-one map of a bounded smooth vector field on R is increasing.
Exercise 1.2. Write down a compactly supported vector field on R2 whose time-one map exchanges the points (±1,0).
Exercise 1.3. Show that the linking number of two disjoint closed curves in R3 is unchanged when both are moved by a diffeomorphism isotopic to the identity.
Exercise 1.4. Let K be the curve t↦(cost,sint,cos2t,sin2t,cos3t) in R5. Show that the projection to the first two coordinates is an embedding of K, and that the projection to the third and fourth is not.
Degree and Invertibility
Some architectures are designed so that every member is invertible: normalizing flows, invertible residual networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from [9, Ch. 7].
Global invertibility
Theorem 2.1 (Hadamard). [classical; ] A local diffeomorphism f:Rd→Rd is a diffeomorphism if and only if it is proper.
Proof. A proper local diffeomorphism is a covering map, and Rd is simply connected. In terms of degree: every point is a regular value, the sign of detDf is constant, and so the number of preimages of a point is the absolute value of the degree, which is constant; it is 1 because a covering of a simply connected space is trivial. ◻
A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded one.
Proposition 2.2 (Invertible residual blocks). [classical; Banach; Behrmann et al. [1]] Let g:Rd→Rd have Lipschitz constant L<1 and let f=id+g.
f is a bijection, its inverse is Lipschitz with constant (1−L)−1, and xj+1=y−g(xj) converges to f−1(y) with ∣xj−f−1(y)∣≤Lj∣x0−f−1(y)∣.
If g is smooth then f is a diffeomorphism with detDf>0.
fs=id+sg, 0≤s≤1, is a diffeotopy from the identity to f. In particular f is proper of degree 1.
Proof. (a) x↦y−g(x) is a contraction; and ∣f(x)−f(x′)∣≥(1−L)∣x−x′∣. (b), (c): the eigenvalues of Dg have modulus at most L, so 1+sDg is invertible for s∈[0,1], and its determinant is continuous, non-zero, and equal to 1 at s=0. ◻
Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common use all realize diffeomorphisms in the identity component of Diff(Rd), and by [9, Thm. 10.2] every such diffeomorphism with compact support is conversely the time-one map of a flow.
Topological mismatch
A normalizing flow models a probability measure ν on Rd as the image h∗μ of a simple measure μ, usually a Gaussian, under a diffeomorphism h.
Proposition 2.3. [classical; cf. Cornish et al. [3]] For a homeomorphism h of Rd and a Borel probability measure μ, supp(h∗μ)=h(suppμ). Hence if the supports of μ and ν are not homeomorphic, no homeomorphism satisfies h∗μ=ν.
A Gaussian has support Rd. A target supported on two disjoint balls, on a sphere, or on any proper closed subset cannot be represented exactly, and Cornish et al. show that a sequence of flows approximating such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature are the ones differential topology suggests: a target supported on a submanifold Mm⊂Rd is modelled by a flow on Rm followed by an embedding Rm→Rd, and when M is not diffeomorphic to an open subset of Rm, by several such charts with a discrete index, which is an atlas in the sense of [9, Ch. 1]. The minimal number of charts is an invariant of M; for a sphere it is two.
Equilibrium layers
A deep equilibrium layer outputs a solution z of z=gθ(z,x) for a network gθ. The existence and number of solutions is a fixed-point problem, and [9, Prop. 7.11] describes its three aspects: a generic signed count, a derived fibre product, and a trace.
Theorem 2.4 (Equilibria of a bounded layer). [reformulation; cf. [9, Thm. 4.5, Thm. 7.2, Thm. 7.9]] Let g(z)=tanh(Az+u) on Rk, with A∈Rk×k and u∈Rk.
For every A and almost every u, every fixed point of g is non-degenerate: 1 is not an eigenvalue of Dg there. The fixed points are then finite in number.
In that case ∑g(z)=zsigndet(1−Dg(z))=1. In particular the number of fixed points is odd, and at least one exists.
Each non-degenerate fixed point is locally a smooth function of (A,u), with dz=(1−Dzg)−1d(A,u)g.
Proof. (a) The map Ψ(u,z)=z−tanh(Az+u) has DuΨ=−diag(sech2(Az+u)), which is invertible; so Ψ is a submersion and by [9, Thm. 4.5] zero is a regular value of Ψ(u,⋅) for almost every u. Regularity at a zero z means that 1−Dg(z) is invertible. All fixed points lie in the open cube (−1,1)k; being isolated in a compact set they are finite. (b) For 0≤s≤1 the map gs(z)=tanh(s(Az+u)) sends the closed cube Q=[−1,1]k into its interior, so id−gs has no zero on ∂Q, and the degree at 0 of id−gs:(Q,∂Q)→(Rk,Rk∖0) is independent of s by [9, Thm. 7.2]. At s=0 it is the degree of the identity. At s=1, zero being a regular value, it is the stated sum. (c) is the implicit function theorem. ◻
In one dimension, z=tanh(az+u) has one solution for a≤1, and for a>1 and small ∣u∣ three, with signs +,−,+: two stable equilibria and an unstable one between them. The script finds the fixed points for several (a,u) and for a two-dimensional example and checks that the signed count is 1.
Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set Δ of 1; crossing it creates or destroys a pair of equilibria of opposite sign, as in the birth–death family of [9, Thm. 8.12], and it is there that the output of an equilibrium layer can jump.
Notes
The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and Mazur, Palais). Invertible residual networks are from [1]. The observation that the topology of the support obstructs normalizing flows, with a quantitative form, is in [3]. Deep equilibrium models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone operator parametrizations) are a separate line of work, and 2.4 describes what happens without them.
lists the main statements of this book with the theorems of the companion volume from which they follow. Every entry in the right-hand column was known by the mid-1970s.
Nothing in this book is a new theorem of differential topology, and most of its statements about networks have appeared in the machine learning literature in some special case, with the references given in the text: generic smoothness of decision boundaries, the manifold of global minima, the obstruction to neural ODEs and its removal by augmentation, the mismatch of supports for flows. What the book adds is the uniform derivation, with the hypothesis that makes each statement true stated as a transversality property of the architecture (1.5), and a small number of statements tagged as observations that we have not found in this form: the description of the moduli of irreducible one-hidden-layer networks as a configuration space, with its monodromy (3.5); the wall structure of the discriminant (1.5); the surgery description of training with its Euler characteristic count (Theorem 2.2, Proposition 2.3) and its moves (2.7); the dimension count 2m+2 for augmentation in terms of the data (Theorem 1.6, Corollary 1.7); and the parity of the number of equilibria (2.4). Each is a short corollary and is offered as such.
The higher-categorical layer was used as language more than as a tool. The bicategory Para and the lens functor organize Families of Maps, Derivatives; the quotient by symmetries is a stack and not a space for homogeneous activations (3); and a degenerate level set is a derived manifold with a cobordism class (1.2). The theorems that do the work are those of 1.1.
Limits
Smoothness. The standing assumption excludes ReLU. For piecewise-linear networks the analogue of 1.1 is the theorem of Grigsby and Lindsey [18]; analogues of Theorem 1.4, Theorem 2.2 would require stratified Morse theory, and are not proved here.
Genericity within the family. “Almost every parameter” is with respect to Lebesgue measure on Θ, so the statements apply to parameters drawn from a distribution with a density, as at a random initialization. They do not apply to distinguished parameters. In particular, if g is an orthogonal symmetry of the architecture, the gradient of an invariant loss is equivariant, and so the fixed-point set of g is invariant under gradient flow: a network initialized with two identical hidden units keeps them identical for ever, and stays in a reducible stratum of measure zero where none of the generic statements need hold.
Paths. describes generic paths. A training trajectory is a particular path, the solution of a differential equation, and we have proved nothing about whether it is generic. The honest statement is 2.5: arbitrarily close to it there is a path with the stated behaviour.
Compactness. Input domains are assumed compact for the isotopy statements, and parameter space is never compact, which restricts the use of Morse theory for losses (3.5).
No rates. With the single exception of 3.2, the statements are qualitative. They say that a set has measure zero and not how small its neighbourhoods are; that an isotopy exists and not how large its velocity must be; that a boundary of a given topology is realizable and not at what width. Each of these has a quantitative counterpart in the literature that the present methods do not reach.
No learning theory. Nothing here concerns generalization.
Directions
Open Problem 1.1 (Piecewise-linear version). [programme] Prove the analogues of Theorem 1.4, Theorem 2.2 for ReLU networks that are generic and transversal in the sense of [18], with handle attachments replaced by the elementary moves of stratified Morse theory on the canonical polyhedral complex.
Open Problem 1.2 (Quantitative discriminant). [programme] For a given architecture and initialization distribution, estimate the measure of the ε-neighbourhood of the discriminant Δ of 1, and hence the expected number of surgeries along a path of given length. suggests that Δ is a hypersurface and that the answer is governed by its volume.
Open Problem 1.3 (The stack of networks). [programme] For homogeneous activations, describe the quotient stack [Θ/G] of 3 and the critical locus of a loss on it as a derived object, so that 3.1 and the degeneracies listed in 3 become statements about a single space with a virtual dimension.
Together with 1.8, these are the places where the methods of the companion volume, and in particular its higher-categorical parts (derived manifolds, stacks, the calculus of embeddings), would be doing more than providing a vocabulary.
Dictionary
Network notion
Differential topology
Where
architecture F:Θ×X→Y
plot of the mapping object [X,Y]; morphism of Para
The script verify-parametrized-networks.py uses only the Python standard library. It implements small feedforward networks (widths (2,3,1), (2,4,4,1) and a few others) with forward evaluation, forward-mode differentiation by dual numbers and reverse-mode differentiation by backpropagation, a Runge–Kutta integrator, a Jacobi eigenvalue routine and a flood-fill component counter. It performs 49 checks in about three seconds, each printing PASS or FAIL, and exits with status zero if and only if all pass. The checks recompute the examples in the text and test numerical instances of the structural statements on random parameters with a fixed seed. They are not proofs, and a statement about almost every parameter is tested only at the parameters sampled.
backpropagation against forward mode and finite differences for two architectures, and for the input covector (2.1); the tangent kernel is positive semidefinite of rank rkJ; the output dynamics u˙=−K(u−y) (2.3); the loss gradient; the Hessian as Gauss–Newton plus curvature term (2.3)
all 48 symmetries of the tanh network (2,3,1) and the size of an orbit (3.2); the scaling symmetry for ReLU; rank P=13 at a random parameter and ranks 10 and 9 at reducible ones (3.4, 3.2); for ReLU, the generators ξj in kerJ and rkJ≤P−H (3.7); a path of irreducible networks exchanging two hidden units (3.5)
∂F/∂c=1; degenerate zeros of a one-dimensional network occur only at the critical values (1.1(a)); the two chambers of x12−x22+c (1.3); three sectors at a triple point (1.4); the submersion of 1.5
component counts, critical points and indices in 2.4; the Euler characteristic formula of 2.3 for d≤6; the Hessian of the height function on the trace (2.4); births and deaths of boundary points on a circle (2.2)
rkJ=N and the dimension P−N of the solution set; the Hessian JTJ with 4 positive and 9 zero eigenvalues (3.1); the inequality of 3.2; sublevel components, the barrier and the Hessians of 3.4; 3.5; 3.4; invariance of the loss under the symmetries
random first layers of width 3 embed a circle in R5 (1.1), and a coordinate projection that does not (1.4); the convex hull property of 1.2 for random injective networks; order preservation on the line; positivity of the Jacobian determinant and Liouville’s formula for a planar neural ODE (1.4); the swap of 1.2; invariance of the linking number under a neural ODE in R3 (1.5); separation of nested circles after augmentation (1.7)
convergence of the fixed-point inversion and the determinant bound for a residual block (2.2, 2.2); a non-proper local diffeomorphism (2.1); odd numbers of equilibria with signed count 1 in one and two dimensions (2.4)
Solutions to the Exercises
Solutions to all exercises follow, by chapter. Where a solution relies on a computation, the computation is also carried out by verify-parametrized-networks.py.
1.2. Let (Θ,F):X→Y, (Θ′,G):Y→Z, (Θ′′,K):Z→V. Both bracketings have underlying function (θ,θ′,θ′′,x)↦K(θ′′,G(θ′,F(θ,x))), on the parameter spaces (Θ×Θ′)×Θ′′ and Θ×(Θ′×Θ′′) respectively. The reassociation diffeomorphism r satisfies Fleft=Fright∘(r×idX), so r and r−1 are 2-cells, mutually inverse.
1.3. The diagonal δ:Θ→Θ×Θ gives a 2-cell (Θ,(F∘F)∘(δ×id))⇒(Θ×Θ,F∘F), and its source is (θ,x)↦F(θ,F(θ,x)), the weight-tied network. By the chain rule, the derivative of θ↦G(δ(θ)) is ∂1G+∂2G evaluated on the diagonal, for G(θ1,θ2)=F(θ2,F(θ1,x)).
1.4.F(θ,x)=θ+x3. Every member has the degenerate critical point x=0. A function is Morse when its 1-jet prolongation is transverse to the set of jets with vanishing derivative; the family of prolongations (θ,x)↦(x,θ+x3,3x2) is not transverse to {third coordinate=0}, since the third coordinate has vanishing derivative in both θ and x at x=0. The parameter moves the value of the function but not its derivative, so the family is rich for conditions on values (1.5) and not for conditions on first derivatives. Adding a parameter θ1x repairs this: θ0+θ1x+x3 is Morse for θ1=0.
Chapter 2
2.1. Let z=Wx+b and h=σ(z). The lens of the layer has forward part (W,b,x)↦h and backward part (W,b,x,w)↦(δxT,δ,WTδ) with δ=σ′(z)⊙w: the three components are the covectors for W, b and x. The output layer (a,c;h)↦aTh+c has backward part (a,c,h,w)↦(wh,w,wa). The loss 21(u−y)2 has gradient e=u−y in u. Composing: the covector entering the output layer is e, so ∇a=eh, ∇c=e, and the covector passed back is ea; then δ=σ′(z)⊙(ea), ∇W=δxT, ∇b=δ.
2.2.kerJJT=kerJT since ⟨v,JJTv⟩=∣JTv∣2; so rkK=rkJT=rkJ. For scalar output K is n×n, invertible if and only if rkJ=n, that is, if and only if the columns of J span Rn; the k-th column is the vector (∂F/∂θk(θ,xi))i.
2.3. By [9, Prop. 2.9] the 2-jet of λ∘ev at θ has linear part CA and quadratic part C∘B+E∘(A×A), where (A,B) is the 2-jet of ev (so A=J and B=(∇2evk)k) and (C,E) that of λ (so C=∇λT, E=∇2λ). The quadratic part is ∑k∂kλ∇2evk+JT∇2λJ.
2.4.ev(θ)=Φθ with Φ the n×P matrix of rows ϕ(xi)T; so J=Φ for all θ and K=ΦΦT, the Gram matrix of the features. The fibres of ev are the affine subspaces θ0+kerΦ, of dimension P−rkΦ. The realization map is linear with kernel {θ:⟨θ,ϕ(x)⟩=0∀x}, so the functional dimension is the dimension of the span of ϕ(X).
Chapter 3
3.1.F(θ,x)=∑jajσ(⟨wj,x⟩+bj)+c. (i) Permuting the index j reorders the sum. (ii) If σ is odd, (−aj)σ(−⟨wj,x⟩−bj)=ajσ(⟨wj,x⟩+bj). (iii) If σ(λs)=λσ(s) for λ>0, then (aj/λ)σ(λ⟨wj,x⟩+λbj)=ajσ(⟨wj,x⟩+bj).
3.2. The columns of J belonging to (wj,bj) are (ajσ′(⟨wj,xi⟩+bj)xi)i and (ajσ′(⟨wj,xi⟩+bj))i; if aj=0 these d+1 columns vanish, so rkJ≤P−(d+1). If (wj,bj)=(wk,bk), the columns for aj and ak are both (σ(⟨wj,xi⟩+bj))i, so two columns coincide and rkJ≤P−1. (In the second case the columns for (wj,bj) and (wk,bk) are also proportional, with ratio aj/ak, so the rank drops by d+2 in all.)
3.3.θ=(w,b,a,c) and ξ=(w,b,−a,0). Where wx+b>0, F=a(wx+b)+c and DθF=(ax,a,wx+b,1), so DθF⋅ξ=awx+ab−a(wx+b)=0. Where wx+b<0, F=c and DθF=(0,0,0,1), again orthogonal to ξ. Here P=4, H=1, so the functional dimension is at most 3; it equals 3 when a=0 and w=0, since the function x↦aReLU(wx+b)+c is determined by its breakpoint −b/w, the slope aw on the active side, and c (and the side, which is discrete).
3.4.(2,3,1): one hidden layer of 3 units, ∣G∣=23⋅3!=48. (2,4,4,1): two hidden layers of 4 units, ∣G∣=(24⋅4!)2=3842=147456.
Chapter 4
4.1. By 1.1(c), for almost every θ the set Bθ is a compact neat 0-dimensional submanifold of [−R,R]: a finite set of interior points at which fθ′=0, with fθ(±R)=0. At each such point fθ changes sign. So the number of zeros is even if fθ(−R) and fθ(R) have the same sign and odd otherwise.
4.2. A compact neat 1-dimensional submanifold of the disc: a finite disjoint union of circles in the interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints by non-crossing arcs.
4.3.dxF=(2x1,−2x2) vanishes only at x=0, where F=c. So Σ={(0,0)} and Δ={0}. There are two chambers, c>0 and c<0. In each the boundary is a hyperbola; for c<0 its branches open along the x1-axis (x12=x22−c≥−c) and for c>0 along the x2-axis. The region {f<0} is connected for c<0 (it contains the origin) and has two components for c>0. (The plane is not compact; the statement about chambers holds here because the family is a product near infinity up to scaling. Restricted to a disc the same description holds.)
4.4. At a triple point p, fθ is transverse to A{1,2,3}, the line y1=y2=y3 in R3, of codimension 2. So the map x↦(f1−f2,f2−f3) from the plane to R2 is a local diffeomorphism at p, and in the coordinates (u,v)=(f1−f2,f2−f3) the decision regions are {u>0,u+v>0}, {u<0,v>0} and {v<0,u+v<0}: three sectors bounded by the rays {u=0,v>0}, {v=0,u>0} and {u+v=0,u<0}.
Chapter 5
5.1.g=(x12−1)2+x22 has ∇g=(4x1(x12−1),2x2), vanishing at (0,0) and (±1,0). The Hessian is diag(12x12−4,2): at (±1,0) it is diag(8,2), index 0, value 0; at the origin diag(−4,2), index 1, value 1. The sublevel set {g≤t} is empty for t<0, two discs around (±1,0) for 0<t<1 (on the line x2=0 it is {∣x12−1∣≤t}, two intervals when t<1), and one disc for t>1. Its Euler characteristic is 0,2,1, consistent with attaching two 0-cells and then one 1-cell.
5.2. Here Bt is a finite set and Wγ is a compact 1-manifold with boundary B0⊔B1. A critical point of τ on a 1-manifold is a local minimum or maximum of t along a curve: two points of Bt appear together or disappear together (surgery with λ=0 or 1 on a 0-manifold: add or remove an S0). Between critical times the points move continuously. If X is a circle, ft changes sign at each point of Bt and returns to its original sign after one circuit, so #Bt is even for every regular t.
5.3. Two. By 2.3 with d=3, the Euler characteristic of the region changes by ±(−1)k at each surgery; the solid torus has χ=0 and needs at least one 0-handle to be non-empty, so at least one more handle of odd index is needed. A 0-handle (a ball appears, k=0) followed by a 1-handle attached to it (the ball becomes a solid torus, k=1) suffices. The boundary goes from empty to a sphere to a torus.
5.4. Write Φ=Φγ and let t(x) be defined by Φ(t(x),x)=0 near xi, with t(xi)=ti. Differentiating once, Φt∂jt+∂jΦ=0, so ∂jt(xi)=0 because dxΦ=0 there. Differentiating again and evaluating at xi, where the terms containing first derivatives of t vanish, gives Φt∂j∂kt+∂j∂kΦ=0. Hence ∇2t(xi)=−∇x2Φ(ti,xi)/Φt(ti,xi).
Chapter 6
6.1.ev(θ)=Φθ with Φ of rank n. Every y is a regular value; My=θ∗+kerΦ is an affine subspace of dimension P−n, where θ∗ is any solution, for instance the minimum-norm one ΦT(ΦΦT)−1y. The Hessian is the constant matrix ΦTΦ, of rank n, with kernel kerΦ.
6.2.∇L=(ab−1)(b,a), which vanishes when ab=1 or (a,b)=0. The Hessian is (b22ab−12ab−1a2). On ab=1 it is (b211a2), of determinant a2b2−1=0 and trace a2+b2>0: rank 1, with kernel spanned by (a,−b), the tangent to the hyperbola. At the origin it is (0−1−10), with eigenvalues ±1: a saddle of index 1 with value 21. The sublevel set is {1−2c≤ab≤1+2c}. For c<21 the lower bound is positive, and the set lies in the two quadrants ab>0, one component in each. For c≥21 it contains the axes and is connected. Any path from (1,1) to (−1,−1) must cross the set ab=0, where L=21.
6.3. The flow line γ has length at most ℓ=2L(θ0)/μ while it stays in U. If the closed ball of radius ℓ about θ0 lies in U, the flow line cannot leave U: at the first exit time it would be at distance at most its length, which is at most ℓ, from θ0, hence still inside. So the flow line is defined for all time, has finite length, and therefore converges; the limit θ∞ satisfies ∣θ∞−θ0∣≤ℓ and L(θ∞)=limL(γ(t))=0.
6.4.L=21(∏ai−1)2 and My={∏ai=1}, the preimage of the regular value 1 of the product map (its gradient at a point with all ai=0 is non-zero); so dimMy=k−1. On My every ai=0 and the number of negative ai is even; each admissible sign pattern gives one component (a graph over the positive orthant in the first k−1 coordinates), and there are 2k−1 sign patterns with an even number of minus signs. The gradient is (∏ai−1)(∏j=iaj)i, which vanishes at the origin for k≥2; the Hessian at the origin has entries −∂i∂j∏al, which are zero for k≥3. So the origin is a critical point with zero Hessian.
Chapter 7
7.1. Let φt be the flow. If x<y then φt(x)=φt(y) for all t by uniqueness of solutions, so φt(y)−φt(x) is continuous in t, never zero, and positive at t=0; hence positive at t=1.
7.2. Let ρ:[0,∞)→[0,1] be smooth with ρ=1 on [0,2] and ρ=0 on [3,∞), and let v(x)=πρ(∣x∣)(−x2,x1). The flow preserves ∣x∣, since v is tangent to circles, and on {∣x∣≤2} it is rotation with angular speed π. At time one the points (±1,0) have been rotated by the angle π and are exchanged.
7.3. Let φt be the diffeotopy and Kt=φt(K), Lt=φt(L); they are disjoint for every t because φt is injective. The Gauss maps Gt:K×L→S2, Gt(x,y)=(φt(y)−φt(x))/∣φt(y)−φt(x)∣, form a homotopy, and the degree is a homotopy invariant by [9, Thm. 7.2].
7.4. The first projection is t↦(cost,sint), the standard embedding of the circle. The second is t↦(cos2t,sin2t), which takes the same value at t and t+π; it is an immersion but not injective, a double cover of the circle.
Chapter 8
8.1. For fixed y the map T(x)=y−g(x) satisfies ∣T(x)−T(x′)∣≤L∣x−x′∣, so it has a unique fixed point x∗, which is the unique solution of x+g(x)=y, and ∣xj−x∗∣=∣Tj(x0)−Tj(x∗)∣≤Lj∣x0−x∗∣. So f is a bijection. And ∣f(x)−f(x′)∣≥∣x−x′∣−∣g(x)−g(x′)∣≥(1−L)∣x−x′∣, which says that f−1 is Lipschitz with constant (1−L)−1.
8.2. The singular values of 1+Dg are at least 1−L, since ∣(1+Dg)v∣≥∣v∣−∣Dgv∣≥(1−L)∣v∣. The absolute value of the determinant is the product of the singular values, so it is at least (1−L)d; and the determinant is positive by 2.2(b).
8.3.z=0 is always a fixed point, with 1−g′(0)=1−a. For a≤1 it is the only one: tanh(az) is strictly concave for z>0 with slope a≤1 at 0, so tanh(az)<z for z>0, and symmetrically for z<0. For a>1, tanh(az)>z for small z>0 and tanh(az)<1<z for z>1, so there is a fixed point z∗∈(0,1), unique by concavity, and −z∗ by symmetry. At z∗ the graph of tanh(az) crosses the diagonal from above, so g′(z∗)<1 and the sign is +; at 0 the sign of 1−a is −. The signed count is 1−1+1=1. For a<1 it is the single +1. At a=1 the fixed point 0 is degenerate (1−g′(0)=0): this is the exceptional parameter at which the pair ±z∗ is born, a pitchfork, which is not a generic birth–death because of the symmetry z↦−z present when u=0.
8.4. A point y lies in supp(h∗μ) if and only if every open neighbourhood V of y has μ(h−1(V))>0. As h is a homeomorphism, the sets h−1(V) are exactly the open neighbourhoods of h−1(y); so the condition is h−1(y)∈suppμ. If h∗μ=ν then h restricts to a homeomorphism from suppμ to suppν.
References
References
J. Behrmann, W. Grathwohl, R. T. Q. Chen, D. Duvenaud, J.-H. Jacobsen, Invertible residual networks, in ICML 2019, PMLR 97, 573–582.
M. Bianchini, F. Scarselli, On the complexity of neural network classifiers: a comparison between shallow and deep architectures, IEEE Trans. Neural Netw. Learn. Syst. 25 (2014), 1553–1565.
R. Cornish, A. Caterini, G. Deligiannidis, A. Doucet, Relaxing bijectivity constraints with continuously indexed normalising flows, in ICML 2020, PMLR 119.
R. Cockett, G. Cruttwell, J. Gallagher, J.-S. P. Lemay, B. MacAdam, G. Plotkin, D. Pronk, Reverse derivative categories, in Computer Science Logic 2020, LIPIcs 152, 18:1–18:16.
G. S. H. Cruttwell, B. Gavranović, N. Ghani, P. Wilson, F. Zanasi, Categorical foundations of gradient-based learning, in ESOP 2022, Lecture Notes in Computer Science 13240, Springer, 1–28.
Y. Cooper, Global minima of overparameterized neural networks, SIAM J. Math. Data Sci. 3 (2021), 676–691.
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud, Neural ordinary differential equations, in NeurIPS 2018.
E. Dupont, A. Doucet, Y. W. Teh, Augmented neural ODEs, in NeurIPS 2019.
F. Draxler, K. Veschgini, M. Salmhofer, F. Hamprecht, Essentially no barriers in neural network energy landscape, in ICML 2018, PMLR 80.
R. Entezari, H. Sedghi, O. Saukh, B. Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, in ICLR 2022.
K. Fukumizu, S. Amari, Local minima and plateaus in hierarchical structures of multilayer perceptrons, Neural Networks 13 (2000), 317–327.
C. Fefferman, Reconstructing a neural net from its output, Rev. Mat. Iberoamericana 10 (1994), 507–555.
C. Fefferman, S. Mitter, H. Narayanan, Testing the manifold hypothesis, J. Amer. Math. Soc. 29 (2016), 983–1049.
B. Fong, D. I. Spivak, R. Tuyéras, Backprop as functor: a compositional perspective on supervised learning, in LICS 2019, IEEE, 2019.
K. Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network, Neural Networks 9 (1996), 871–879.
T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, A. G. Wilson, Loss surfaces, mode connectivity, and fast ensembling of DNNs, in NeurIPS 2018.
J. E. Grigsby, K. Lindsey, On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks, SIAM J. Appl. Algebra Geom. 6 (2022), 216–242.
J. E. Grigsby, K. Lindsey, R. Meyerhoff, C. Wu, Functional dimension of feedforward ReLU neural networks, preprint, arXiv:2209.04036.
M. W. Hirsch, Differential Topology, Graduate Texts in Mathematics 33, Springer, 1976.
K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks 4 (1991), 251–257.
B. Hanin, M. Sellke, Approximating continuous functions by ReLU nets of minimal width, preprint, arXiv:1710.11278.
A. Jacot, F. Gabriel, C. Hongler, Neural tangent kernel: convergence and generalization in neural networks, in NeurIPS 2018.
J. Johnson, Deep, skinny neural networks are not universal approximators, in ICLR 2019.
C. Liu, L. Zhu, M. Belkin, Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, Appl. Comput. Harmon. Anal. 59 (2022), 85–116.
Q. Nguyen, On connected sublevel sets in deep learning, in ICML 2019, PMLR 97.
G. Naitzat, A. Zhitnikov, L.-H. Lim, Topology of deep neural networks, J. Mach. Learn. Res. 21 (2020), no. 184, 1–40.
M. Phuong, C. H. Lampert, Functional vs. parametric equivalence of ReLU networks, in ICLR 2020.
B. Şimşek, F. Ged, A. Jacot, F. Spadaro, C. Hongler, W. Gerstner, J. Brea, Geometry of the loss landscape in overparameterized neural networks: symmetries and invariances, in ICML 2021, PMLR 139.
H. J. Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, Neural Networks 5 (1992), 589–593.
S. Watanabe, Algebraic Geometry and Statistical Learning Theory, Cambridge University Press, 2009.
H. Zhang, X. Gao, J. Unterman, T. Arodz, Approximation capabilities of neural ODEs and invertible residual networks, in ICML 2020, PMLR 119.
Complete flattened LaTeX source
\documentclass[11pt,openany]{book}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage[sc,osf]{mathpazo}
\linespread{1.05}
\usepackage[scaled=0.9]{helvet}
\usepackage[paperwidth=7in,paperheight=10in,inner=0.95in,outer=0.85in,top=0.95in,bottom=1.0in,headsep=14pt]{geometry}
\usepackage{amsmath,amssymb,amsthm,mathtools,stmaryrd}
\usepackage{microtype}
\usepackage{xcolor}
\definecolor{accent}{HTML}{7A1F2B}
\definecolor{ink2}{HTML}{4A5560}
\definecolor{fwd}{HTML}{1F4E79}
\definecolor{bwd}{HTML}{A8432C}
\usepackage{tikz,tikz-cd}
\usetikzlibrary{arrows.meta,positioning,calc,decorations.markings,shapes.geometric,backgrounds,fit}
\usepackage{booktabs,array,tabularx,longtable}
\usepackage{enumitem}
\setlist{itemsep=2pt,topsep=4pt}
\usepackage{titlesec}
\usepackage{fancyhdr}
\usepackage[font=small,labelfont={bf,sf},margin=1.5em]{caption}
\usepackage[colorlinks=true,linkcolor=accent,citecolor=accent,urlcolor=accent,
pdftitle={Parametrized Neural Networks and Their Representations in Differential Topology},
pdfsubject={Differential topology of parametrized families of maps, applied to neural networks},
pdfkeywords={neural networks, transversality, decision boundaries, loss landscapes, isotopy, neural ODEs, symmetries, differential topology}]{hyperref}
\usepackage[nameinlink,capitalise,noabbrev]{cleveref}
\titleformat{\chapter}[display]{\normalfont\Large\scshape\color{accent}}{\chaptertitlename\ \thechapter}{4pt}{\normalfont\huge\bfseries\color{black}}
\titlespacing*{\chapter}{0pt}{10pt}{26pt}
\titleformat{\section}{\normalfont\large\bfseries}{\thesection}{0.7em}{}
\titleformat{\subsection}{\normalfont\normalsize\bfseries\itshape}{\thesubsection}{0.7em}{}
\titleformat{\part}[display]{\normalfont\centering\Large\scshape\color{accent}}{\partname\ \thepart}{12pt}{\normalfont\Huge\bfseries\color{black}}
\pagestyle{fancy}\fancyhf{}
\fancyhead[LE]{\small\thepage\quad\textsc{\nouppercase{\leftmark}}}
\fancyhead[RO]{\small\textsc{\nouppercase{\rightmark}}\quad\thepage}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\chaptermark}[1]{\markboth{#1}{#1}}
\renewcommand{\sectionmark}[1]{\markright{\thesection\ \ #1}}
\fancypagestyle{plain}{\fancyhf{}\fancyfoot[C]{\small\thepage}}
\setlength{\headheight}{14pt}
\setcounter{tocdepth}{1}
\setcounter{secnumdepth}{2}
\raggedbottom
\emergencystretch=2em
\tolerance=1200
\hbadness=2000
\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[chapter]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{principle}[theorem]{Principle}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\newtheorem{construction}[theorem]{Construction}
\newtheorem{problem}[theorem]{Open Problem}
\newtheorem{exercise}{Exercise}[chapter]
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\crefname{principle}{Principle}{Principles}
\crefname{construction}{Construction}{Constructions}
\crefname{conjecture}{Conjecture}{Conjectures}
\crefname{exercise}{Exercise}{Exercises}
\crefname{problem}{Open Problem}{Open Problems}
\newcommand{\stag}[2]{{\footnotesize\sffamily\upshape\color{#1}[#2]}\enspace\ignorespaces}
\newcommand{\Scl}[1][]{\stag{ink2}{classical\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Sli}[1][]{\stag{fwd}{literature\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Srf}[1][]{\stag{accent}{reformulation\if\relax\detokenize{#1}\relax\else; cf.\ #1\fi}}
\newcommand{\Sob}{\stag{accent}{observation}}
\newcommand{\Spg}{\stag{bwd}{programme}}
\newcommand{\cat}[1]{\mathsf{#1}}
\newcommand{\Set}{\cat{Set}}
\newcommand{\FinSet}{\cat{FinSet}}
\newcommand{\Meas}{\cat{Meas}}
\newcommand{\Stoch}{\cat{Stoch}}
\newcommand{\FinStoch}{\cat{FinStoch}}
\newcommand{\BorelStoch}{\cat{BorelStoch}}
\newcommand{\FinProb}{\cat{FinProb}}
\newcommand{\FinStat}{\cat{FinStat}}
\newcommand{\Krn}{\cat{Krn}}
\newcommand{\Hilb}{\cat{Hilb}}
\newcommand{\Ban}{\cat{Ban}}
\newcommand{\Src}{\cat{Src}}
\newcommand{\Chan}{\cat{Chan}}
\newcommand{\Dyn}{\cat{Dyn}}
\newcommand{\Cat}{\cat{Cat}}
\newcommand{\CC}{\mathcal{C}}
\newcommand{\FF}{\mathcal{F}}
\newcommand{\GG}{\mathcal{G}}
\newcommand{\II}{\mathcal{I}}
\newcommand{\Sinf}{\mathcal{S}}
\newcommand{\Qlaw}{\mathcal{Q}}
\newcommand{\Fun}{\mathcal{F}}
\newcommand{\Dist}{\Delta}
\newcommand{\Giry}{\mathcal{G}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Rbar}{\overline{\mathbb{R}}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\Fp}{\mathbb{F}_p}
\newcommand{\E}{\mathbb{E}}
\newcommand{\Prob}{\mathbb{P}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\colim}{colim}
\DeclareMathOperator{\Hom}{Hom}
\DeclareMathOperator{\Ext}{Ext}
\DeclareMathOperator{\Fix}{Fix}
\DeclareMathOperator{\supp}{supp}
\DeclareMathOperator{\TV}{TV}
\DeclareMathOperator{\rk}{rk}
\newcommand{\id}{\mathrm{id}}
\newcommand{\op}{\mathrm{op}}
\newcommand{\adj}{\dashv}
\newcommand{\iso}{\cong}
\newcommand{\st}{\mid}
\newcommand{\cpy}{\mathrm{copy}}
\newcommand{\del}{\mathrm{del}}
\newcommand{\KL}[2]{D(#1\,\|\,#2)}
\newcommand{\hb}{h}
\newcommand{\AZ}{A^{\Z}}
\newcommand{\BZ}{B^{\Z}}
\newcommand{\term}[1]{\textbf{\itshape #1}}
\newcommand{\Mfd}{\cat{Mfd}}
\newcommand{\Cart}{\cat{Cart}}
\newcommand{\SmSet}{\cat{SmSet}}
\newcommand{\Dflg}{\cat{Dflg}}
\newcommand{\CinfRing}{C^\infty\cat{Ring}}
\newcommand{\DMfd}{\cat{DMfd}}
\newcommand{\Spc}{\mathcal{S}}
\newcommand{\Sp}{\cat{Sp}}
\newcommand{\Cob}{\cat{Cob}}
\newcommand{\Bord}{\cat{Bord}}
\newcommand{\Vect}{\cat{Vect}}
\newcommand{\Top}{\cat{Top}}
\newcommand{\Para}{\cat{Para}}
\DeclareMathOperator{\Diff}{Diff}
\DeclareMathOperator{\Emb}{Emb}
\DeclareMathOperator{\Imm}{Imm}
\DeclareMathOperator{\Map}{Map}
\DeclareMathOperator{\Sing}{Sing}
\DeclareMathOperator{\Fr}{Fr}
\DeclareMathOperator{\Th}{Th}
\DeclareMathOperator{\Gr}{Gr}
\DeclareMathOperator{\GL}{GL}
\DeclareMathOperator{\ind}{ind}
\DeclareMathOperator{\codim}{codim}
\DeclareMathOperator{\vdim}{vdim}
\DeclareMathOperator{\tr}{tr}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Tub}{Tub}
\DeclareMathOperator{\Conf}{Conf}
\DeclareMathOperator{\diag}{diag}
\DeclareMathOperator{\relu}{ReLU}
\newcommand{\Or}{\mathrm{O}}
\newcommand{\SO}{\mathrm{SO}}
\newcommand{\RP}{\mathbb{RP}}
\newcommand{\CP}{\mathbb{CP}}
\newcommand{\C}{\mathbb{C}}
\newcommand{\trans}{\pitchfork}
\newcommand{\pt}{\mathrm{pt}}
\newcommand{\loss}{\mathcal{L}}
\tikzset{
ogbox/.style={draw, thick, rounded corners=2pt, minimum width=1.25cm, minimum height=1.05cm, fill=black!4},
fw/.style={-{Stealth[length=5pt]}, thick, fwd},
bw/.style={-{Stealth[length=5pt]}, thick, bwd, densely dashed},
dec/.style={circle, fill, inner sep=1.7pt},
nat/.style={circle, draw, thick, fill=white, inner sep=1.7pt},
leaf/.style={inner sep=2pt, font=\small},
lab/.style={font=\small, inner sep=1.5pt},
infoset/.style={densely dashed, thick, accent},
st/.style={circle, draw, thick, fill=white, inner sep=1.6pt},
ty/.style={rectangle, draw, thick, fill=black!8, inner sep=2.6pt},
}
\newcommand{\sol}[1]{\par\medskip\noindent\textbf{#1.}\enspace\ignorespaces}
\newcommand{\bookauthor}{Anonymous}
\newcommand{\DT}[1]{\cite[#1]{DTHC}}
\newcommand{\ev}{\mathrm{ev}}
\begin{document}
\frontmatter
\begin{titlepage}
\thispagestyle{empty}
\vspace*{1.1in}
\noindent{\Large\scshape\color{accent} Higher Categorical Semantics}\par
\vspace{0.9in}
\noindent{\fontsize{28}{36}\selectfont\bfseries\raggedright Parametrized Neural\\ Networks and Their\\ Representations in\\ Differential Topology\par}
\vspace{0.35in}
\noindent{\Large\itshape Families of maps, transversality,\\[2pt] surgery along training, and isotopy}\par
\vfill
\noindent\begin{tikzpicture}[x=1cm,y=1cm]
\node[ogbox] (th) at (0,0.7) {$\Theta$};
\node[ogbox] (x) at (0,-0.7) {$X$};
\node[ogbox,minimum width=1.6cm] (F) at (2.6,0) {$F$};
\node[lab] (y) at (5.0,0) {$Y$};
\draw[fw] (th) -- (F);
\draw[fw] (x) -- (F);
\draw[fw] (F) -- (y);
\node[lab] at (6.1,0) {$=$};
\node[lab] (t2) at (7.2,0) {$\Theta$};
\node[lab] (m) at (9.6,0) {$[X,Y]$};
\draw[fw,black] (t2) -- (m);
\end{tikzpicture}\par
\vspace{0.5in}
\noindent{\large \bookauthor}\par
\vspace{4pt}
\noindent{\small 2026}
\end{titlepage}
\thispagestyle{empty}
\vspace*{\fill}
{\small
\noindent\textbf{About this text.}
This monograph applies the results of \emph{Differential Topology in Higher Categories} \cite{DTHC}, the
higher-categorical treatment of M.~W.~Hirsch's \emph{Differential Topology} \cite{Hir76} in the same series, to smooth
families of maps defined by neural network architectures. It is written to be read after that volume and
cites it by theorem number, in the form \DT{4.5}. It is an independent work and is not affiliated with or
endorsed by the publisher of Hirsch's book.
\medskip
\noindent\textbf{Status of results.}
Every numbered statement carries a tag: \textsf{classical} (standard mathematics, attributed),
\textsf{literature} (established in the machine learning or mathematical literature, attributed),
\textsf{reformulation} (a known result restated or specialized, with the derivation given here),
\textsf{observation} (an elementary consequence we have not found stated in this form), or
\textsf{programme} (a direction, not a result). The statements tagged \textsf{reformulation} and
\textsf{observation} are corollaries of classical differential topology applied to parametrized families;
none is claimed as a new theorem of differential topology, and none says anything quantitative about
training or generalization.
\medskip
\noindent\textbf{Authorship and assistance.}
The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of
the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall,
mathematical consistency checks, computation, and typesetting. The numerical examples and finite instances
of the structural claims are recomputed by the accompanying script \texttt{verify-parametrized-networks.py}
(\cref{app:computations}), which uses small networks implemented in pure Python. Proofs and bibliographic
details have not been independently refereed; corrections are welcome.
\medskip
\noindent Open access. Rights remain with the author.
}
\clearpage
\tableofcontents
\chapter{Preface}
A neural network architecture is a smooth map $F\colon\Theta\times X\to Y$: a family of maps from inputs to
outputs, parametrized by weights. Differential topology, in the form Hirsch gave it, is the study of what is
true of a generic member of a family of maps and of what happens along a generic path in one. The fit is
close enough that most of the qualitative questions asked about networks have answers that are corollaries
of theorems proved before 1976, once the questions are phrased as questions about families.
Is the decision boundary of a classifier a manifold? For almost every value of the parameters, yes, by the
parametric transversality theorem, provided the last layer has a bias. How does the boundary change during
training? By ambient isotopy, except at isolated times, at each of which it undergoes one surgery whose type
is the index of a critical point; this is Morse theory on the trace of the boundary. What is the set of
parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the
number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary
differential equation separate two linked circles? Not in dimension three, because its time-one map is
ambient isotopic to the identity; in dimension four always, because there any two embeddings of a
one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the
companion volume, and this book supplies the paragraphs.
The higher-categorical layer of the companion volume enters in three places. A parametrized network is a
plot of the mapping object $[X,Y]$ in smooth sets, and composition of layers is composition in a category of
parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an
architecture make the parameter space an atlas for a stack whose points are networks up to relabelling,
and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional
parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is
what one sees of them from nearby.
The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a
rate of convergence or a generalization error. Activation functions are assumed smooth except where stated,
and the piecewise-linear case, which needs a different transversality theory, is treated only by reference.
What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks
the examples.
\chapter{Introduction}
\section*{What is assumed}
The companion volume \cite{DTHC}, at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of
Hirsch's book. From machine learning, only the definition of a feedforward network and of gradient descent.
\section*{How the book is organized}
Part~I sets up families, their derivatives and their symmetries. Part~II studies level sets: of the network
in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss
in parameter space (the interpolation manifold). Part~III studies the maps themselves: when they are
embeddings, when they are isotopic, when they are invertible. Part~IV is an assessment.
\Cref{app:dictionary} gives the dictionary between the two subjects in one table.
\section*{Conventions}
Manifolds and maps are smooth. A \emph{family} is a smooth map $F\colon\Theta\times X\to Y$; we write
$f_\theta=F(\theta,-)$. $P=\dim\Theta$ is the number of parameters, $d=\dim X$ the input dimension, $n$ the
number of data points. ``Almost every'' refers to Lebesgue measure on a Euclidean parameter space, and
``generic'' means belonging to a residual set whose complement has measure zero. An activation function
$\sigma\colon\R\to\R$ is smooth unless it is explicitly $\relu(s)=\max(s,0)$. Citations of the form
\DT{4.5} refer to numbered statements of the companion volume.
\mainmatter
\part{Families}
\chapter{Families of Maps}\label{ch:families}
\section{Parametrized maps}
\begin{definition}\label{def:family}
A \term{parametrized map} from $X$ to $Y$ is a pair $(\Theta,F)$ of a manifold $\Theta$ and a smooth map
$F\colon\Theta\times X\to Y$. Its \term{members} are the maps $f_\theta=F(\theta,-)$, and its
\term{realization map} is $\theta\mapsto f_\theta$, from $\Theta$ to $C^\infty(X,Y)$.
\end{definition}
By \DT{Thm.~1.6} a parametrized map is exactly a plot of the mapping object $[X,Y]$ in smooth sets, with
domain $\Theta$; the realization map is the function it induces on points. Everything in this book is a
statement about plots of mapping objects. The topologies on $C^\infty(X,Y)$ play no direct role, since
parameter spaces are finite-dimensional; what is used is the relation between properties of the total map
$F$ and properties of its members.
\begin{definition}\label{def:para}
$\Para$ is the bicategory whose objects are manifolds, whose morphisms $X\to Y$ are parametrized maps, and
whose $2$-cells $(\Theta',F')\Rightarrow(\Theta,F)$ are smooth maps $r\colon\Theta'\to\Theta$ with
$F'=F\circ(r\times\id_X)$. The composite of $(\Theta,F)\colon X\to Y$ and $(\Theta',G)\colon Y\to Z$ is
\[\bigl(\Theta\times\Theta',\ (\theta,\theta',x)\mapsto G(\theta',F(\theta,x))\bigr).\]
\end{definition}
Composition is associative up to the canonical reassociation of products, and the identity of $X$ is
parametrized by a point. The construction is due to Fong, Spivak and Tuy\'eras \cite{FST19} and is developed
in \cite{CGGWZ22}. A $2$-cell is a reparametrization: sharing weights between layers is the $2$-cell induced
by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.
\begin{proposition}\label{prop:plot}\Srf[\DT{Thm.~1.6}; \cite{FST19}]
Composition in $\Para$ is the internal composition
$[X,Y]\times[Y,Z]\to[X,Z]$ of smooth sets, evaluated on plots: the composite of two parametrized maps is the
plot $\Theta\times\Theta'\to[X,Y]\times[Y,Z]\to[X,Z]$. Hence $\Para$ is the bicategory of plots of the
category of manifolds enriched in smooth sets.
\end{proposition}
\begin{proof}
The internal composition is the map of sheaves which on plots with domain $U$ sends
$(F,G)$ to $(u,x)\mapsto G(u,F(u,x))$. Precomposing with the product plot, whose components depend on
separate factors of $U=\Theta\times\Theta'$, gives the formula of \cref{def:para}.
\end{proof}
\section{Architectures}
\begin{definition}\label{def:arch}
A \term{feedforward architecture} with widths $(n_0,\dots,n_L)$ and activation $\sigma$ is the composite in
$\Para$ of the layers
\begin{align*}
\ell_i&\colon\R^{n_{i-1}}\to\R^{n_i}, & \ell_i(W_i,b_i;x)&=\sigma(W_ix+b_i)\qquad(i<L),\\
\ell_L&\colon\R^{n_{L-1}}\to\R^{n_L}, & \ell_L(W_L,b_L;x)&=W_Lx+b_L,
\end{align*}
with $\sigma$ applied to each coordinate. Its parameter space is
$\Theta=\prod_i(\R^{n_i\times n_{i-1}}\times\R^{n_i})$, of dimension $P=\sum_in_i(n_{i-1}+1)$. The
number of hidden units is $H=n_1+\dots+n_{L-1}$.
\end{definition}
If $\sigma$ is smooth the family is smooth, and real-analytic if $\sigma$ is (as $\tanh$, the logistic
function and softplus are). For $\sigma=\relu$ the family is continuous and piecewise polynomial in
$(\theta,x)$, and it is not a morphism of $\Para$ as defined; statements about it are made separately.
Convolutional and attention layers are parametrized maps in the same sense, with $2$-cells expressing weight
sharing. Nothing below depends on the feedforward form except where an explicit hypothesis on the last or
first layer is made.
The image of the realization map, the set $\mathcal F=\{f_\theta\}$ of functions the architecture can
compute, is a subset of $C^\infty(X,Y)$. With the plots that factor locally through $\Theta$ it is a
diffeological space, the quotient of $\Theta$ by the relation $f_\theta=f_{\theta'}$. It is in general not a
manifold; \cref{ch:symmetry} describes its singularities.
\section{Data, loss, and training}
Fix inputs $x_1,\dots,x_n\in X$ and targets $y_1,\dots,y_n\in Y=\R^m$. The \term{evaluation map} is
\[\ev\colon\Theta\to Y^n=\R^{nm},\qquad\ev(\theta)=\bigl(f_\theta(x_1),\dots,f_\theta(x_n)\bigr),\]
and the squared loss is $\loss(\theta)=\tfrac12|\ev(\theta)-y|^2$. More generally a loss is
$\loss=\lambda\circ\ev$ for a smooth $\lambda\colon\R^{nm}\to\R$. A \term{training path} is a smooth curve
$\gamma\colon[0,T]\to\Theta$; gradient flow is the curve with $\dot\gamma=-\nabla\loss(\gamma)$, for the
Euclidean metric on $\Theta$. A path in $\Theta$ is a plot $[0,T]\to[X,Y]$: a one-parameter family of
maps. When every member is an embedding of a submanifold $K\subset X$ it is an isotopy of $K$ in the sense of
\DT{Def.~10.1}.
Three kinds of question organize the rest of the book.
\begin{enumerate}[label=(Q\arabic*)]
\item What does a generic member $f_\theta$ look like? (\cref{ch:decision,ch:isotopy,ch:degree})
\item What happens to the members along a generic path? (\cref{ch:paths})
\item What does the set of parameters satisfying a condition, such as $\loss(\theta)=0$, look like?
(\cref{ch:symmetry,ch:loss})
\end{enumerate}
In each case the answer comes from the same mechanism. A condition on a member, or on a parameter, is the
condition that a point lies in the preimage of a submanifold under a map built from $F$; the preimage is a
manifold when the map is transverse to the submanifold; and transversality of the map built from $F$ can be
checked on the architecture, usually by exhibiting one direction in parameter space, a bias, in which the
map moves freely.
\begin{principle}[Richness]\label{prin:rich}\Srf[\DT{Thm.~4.5}]
Let $F\colon\Theta\times X\to Y$ be a family and $A\subset Y$ a submanifold. If $F$ is transverse to $A$, in
particular if $F$ is a submersion, then $f_\theta$ is transverse to $A$ for almost every $\theta$. A
feedforward architecture is a submersion onto $Y=\R^{n_L}$ because of the last-layer bias:
$\partial F/\partial b_L=\id$.
\end{principle}
This is \DT{Thm.~4.5} with its hypothesis verified. It is worth stating separately because the hypothesis is
about the family and not about function space. A statement that a property is generic among all smooth maps
(for instance, that Morse functions are dense) says nothing about a given architecture, whose realization is
a thin subset of all maps. Genericity transfers to an architecture only through a transversality property of
$F$ itself, and each use of \cref{prin:rich} below names the parameters that provide it.
\section{The dictionary in brief}
\begin{center}\small
\begin{tabular}{@{}p{0.34\textwidth}p{0.42\textwidth}l@{}}
\toprule
network notion & object of differential topology & Ch.\\
\midrule
architecture & plot of a mapping object; morphism of $\Para$ & \ref{ch:families}\\
backpropagation & functoriality of the reverse derivative & \ref{ch:backprop}\\
weight-space symmetry & group action with invariant realization & \ref{ch:symmetry}\\
decision boundary & preimage of a regular value & \ref{ch:decision}\\
change of topology in training & handle attachment on the trace & \ref{ch:paths}\\
interpolating parameters & fibre of the evaluation map & \ref{ch:loss}\\
barrier between minima & death level in the sublevel barcode & \ref{ch:loss}\\
hidden representation of data & immersion or embedding of a manifold & \ref{ch:isotopy}\\
neural ODE, residual network & diffeotopy; isotopy of embeddings & \ref{ch:isotopy}\\
invertible network & diffeomorphism of degree one & \ref{ch:degree}\\
equilibrium of an implicit layer & transverse fixed point & \ref{ch:degree}\\
\bottomrule
\end{tabular}
\end{center}
\section*{Notes}
The categorical treatment of learners as parametrized maps with update and request functions begins with
\cite{FST19}; the bicategory $\Para$ and its combination with lenses and reverse derivatives is in
\cite{CGGWZ22}. The point of view that a network is a family in the sense of singularity theory, to which
transversality applies, is implicit in much work on loss landscapes, and explicit for piecewise-linear
networks in \cite{GL22}.
\section*{Exercises}
\begin{exercise}\label{exer:count}
Compute $P$ and $H$ for the architectures with widths $(2,3,1)$, $(2,4,4,1)$ and $(d,k,1)$.
\end{exercise}
\begin{exercise}\label{exer:paracomp}
Verify that composition in $\Para$ is associative up to the reassociation
$(\Theta\times\Theta')\times\Theta''\cong\Theta\times(\Theta'\times\Theta'')$, and that this isomorphism
is a $2$-cell in both directions.
\end{exercise}
\begin{exercise}\label{exer:sharing}
Let $(\Theta,F)\colon X\to X$ be a layer. Express the weight-tied network $f_\theta\circ f_\theta$ as the
source of a $2$-cell into the composite $(\Theta\times\Theta,F\circ F)$, and show that its derivative with
respect to $\theta$ is the sum of the two partial derivatives of the untied network on the diagonal.
\end{exercise}
\begin{exercise}\label{exer:notrich}
Give an example of a family $F\colon\R\times\R\to\R$ no member of which is a Morse function, although Morse
functions are dense in $C^\infty(\R,\R)$. Which hypothesis of \cref{prin:rich} fails for the relevant jet
condition?
\end{exercise}
\chapter{Derivatives}\label{ch:backprop}
Two derivatives of a family matter: with respect to the input, which governs the geometry of each member, and
with respect to the parameters, which governs training. Both are instances of the tangent functor of
\DT{Ch.~2}, and the algorithm that computes the second efficiently is the functoriality of its transpose.
\section{Backpropagation}
Recall from \DT{Def.~2.10} the category $\cat{Lens}$ and from \DT{Prop.~2.11} the functor
$f\mapsto(f,R[f])$, $R[f](x,w)=(Df_x)^{\mathsf T}w$, from Euclidean spaces to lenses. For a parametrized map
$F\colon\Theta\times X\to Y$ the reverse derivative has two components,
\[R[F](\theta,x,w)=\bigl(R_\Theta[F](\theta,x,w),\ R_X[F](\theta,x,w)\bigr)
=\bigl((D_\theta F)^{\mathsf T}w,\ (D_xF)^{\mathsf T}w\bigr),\]
the first being the update direction for the parameters and the second the covector passed back to the
previous layer.
\begin{theorem}[Backpropagation is a functor]\label{thm:backprop}
\Sli[Fong--Spivak--Tuy\'eras \cite{FST19}; Cruttwell et al.\ \cite{CGGWZ22}]
The assignment $(\Theta,F)\mapsto(\Theta,(F,R[F]))$ is a functor of bicategories from $\Para$ over Euclidean
spaces to parametrized lenses. In particular, for a composite of layers $F=\ell_L\circ\dots\circ\ell_1$ the
component $R_{\Theta_i}[F]$ is obtained by composing the lenses of the layers: a forward pass computing the
activations $x_i=\ell_i(\theta_i,x_{i-1})$, and a backward pass computing $w_{i-1}=R_X[\ell_i](\theta_i,
x_{i-1},w_i)$ and emitting $R_\Theta[\ell_i](\theta_i,x_{i-1},w_i)$.
\end{theorem}
\begin{proof}
Composition in $\Para$ is composition of maps after taking products with identities, and $R$ is a
product-preserving functor to lenses by \DT{Prop.~2.11}.
\end{proof}
The gradient of a loss $\loss=\lambda\circ\ev$ is $R[\ev](\theta,\nabla\lambda)$. The theorem says the
cost of the gradient is that of one forward and one backward pass, whatever the number of parameters, which
is the practical content of reverse-mode differentiation. The script implements both modes for a small
network and checks them against each other and against finite differences.
\section{The Jacobian of the evaluation map}
\begin{definition}\label{def:jac}
For data $x_1,\dots,x_n$ the \term{parameter Jacobian} is $J(\theta)=D\ev_\theta$, an $nm\times P$ matrix.
The \term{tangent kernel} is the $nm\times nm$ matrix $K(\theta)=J(\theta)J(\theta)^{\mathsf T}$.
\end{definition}
\begin{proposition}\label{prop:ntk}\Scl[Jacot--Gabriel--Hongler \cite{JGH18}]
$K(\theta)$ is symmetric positive semidefinite of rank $\rk J(\theta)$. Along gradient flow of
$\loss=\lambda\circ\ev$ the outputs $u(t)=\ev(\gamma(t))$ satisfy $\dot u=-K(\gamma)\nabla\lambda(u)$.
\end{proposition}
\begin{proof}
$\dot u=J\dot\gamma=-J\nabla\loss=-JJ^{\mathsf T}\nabla\lambda$.
\end{proof}
So the tangent kernel is the Riemannian metric that the Euclidean metric of $\Theta$ induces, by pushing
forward along $\ev$, on the cotangent directions of output space; where $J$ is onto, $K$ is invertible and
the outputs follow a gradient flow for the metric $K^{-1}$.
\section{Rank}
The rank of $J$ is the first invariant of a parameter. It is lower semicontinuous, and for analytic
architectures its generic value is attained almost everywhere.
\begin{proposition}\label{prop:rank}\Scl
Let $F$ be real-analytic in $\theta$ on a connected $\Theta$, and let $r$ be the maximum of $\rk J(\theta)$.
Then $\{\theta:\rk J(\theta)=r\}$ is open and dense and its complement has measure zero. On it the
fibres of $\ev$ are submanifolds of dimension $P-r$, and $\ev$ is locally a submersion onto an
$r$-dimensional submanifold of $\R^{nm}$.
\end{proposition}
\begin{proof}
The complement is the common zero set of the $r\times r$ minors of $J$, which are analytic and not all
identically zero. The zero set of a non-zero analytic function on a connected domain is closed with empty
interior and measure zero. The rest is the constant rank theorem.
\end{proof}
\begin{definition}\label{def:fdim}
The \term{functional dimension} of the architecture at $\theta$ is the supremum over all finite sets of
inputs of $\rk J(\theta)$: the rank of the differential of the realization map \cite{GLMW22}.
\end{definition}
The functional dimension is at most $P$, and \cref{ch:symmetry} shows when it is smaller. The strata
$\{\rk J=s\}$ for $s<r$ are where the fibres of $\ev$ jump in dimension. They contain the parameters at
which hidden units are redundant, and they are responsible for most of what is singular in the loss
landscape (\cref{ch:loss}).
\section{Second derivatives}
By the composition law for $2$-jets \DT{Prop.~2.9}, the Hessian of a composite $\lambda\circ\ev$ is
\[\nabla^2\loss(\theta)=J^{\mathsf T}\,\nabla^2\lambda\,J+\sum_k\frac{\partial\lambda}{\partial u_k}\,
\nabla^2\ev_k(\theta),\]
a Gauss--Newton term and a term weighted by the gradient of $\lambda$ in output space.
\begin{proposition}\label{prop:hess}\Scl
At a parameter where $\nabla\lambda(\ev(\theta))=0$, in particular at a zero of the squared loss,
$\nabla^2\loss=J^{\mathsf T}\nabla^2\lambda\,J$. For the squared loss this is $J^{\mathsf T}J$, positive
semidefinite of rank $\rk J$, with kernel $\ker J$.
\end{proposition}
The non-zero eigenvalues of $J^{\mathsf T}J$ and of the tangent kernel $JJ^{\mathsf T}$ coincide. So at an
interpolating parameter the curvature of the loss in the directions normal to the solution set is the
spectrum of the tangent kernel.
\section*{Notes}
Reverse-mode differentiation has been rediscovered several times; Griewank's history is the standard
reference. The compositional formulation is from \cite{FST19} and \cite{CGGWZ22}, and the axiomatics of
reverse derivatives from \cite{CCG20}. The tangent kernel was introduced in \cite{JGH18}. Functional
dimension for $\relu$ networks is studied in \cite{GLMW22}.
\section*{Exercises}
\begin{exercise}\label{exer:bp}
Write out the lens of a layer $\ell(W,b;x)=\sigma(Wx+b)$, and compose two of them to obtain the gradient of
$\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2$ with respect to $(W,b,a,c)$.
\end{exercise}
\begin{exercise}\label{exer:ntk}
Show that $\rk K(\theta)=\rk J(\theta)$ and that, for $n$ data points and scalar output, $K(\theta)$ is
invertible if and only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, restricted
to the data, span $\R^n$.
\end{exercise}
\begin{exercise}\label{exer:hess}
Derive the formula for $\nabla^2(\lambda\circ\ev)$ from the composition of $2$-jets.
\end{exercise}
\begin{exercise}\label{exer:linear}
For the linear family $F(\theta,x)=\langle\theta,\phi(x)\rangle$ with a fixed feature map
$\phi\colon X\to\R^P$, compute $J$, $K$, the fibres of $\ev$, and the functional dimension.
\end{exercise}
\chapter{Symmetries and the Moduli of Networks}\label{ch:symmetry}
The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand
about an architecture, since every function on parameter space that is defined in terms of the network's
input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for
networks with one hidden layer, where the theory is complete, and draws the consequences that hold in
general.
\section{Symmetries}
\begin{definition}\label{def:sym}
A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with
$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.
\end{definition}
For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the
$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of
$W_{i+1}$.
\begin{proposition}\label{prop:syms}\Scl
Every feedforward architecture has the following symmetries.
\begin{enumerate}[label=(\roman*)]
\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.
\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.
\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and
$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by
$\lambda^{-1}$.
\end{enumerate}
For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.
\end{proposition}
\section{One hidden layer}
Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with
$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.
\begin{definition}\label{def:irr}
The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and
$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.
\end{definition}
\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]
Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of
hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and
only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;
so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.
\end{theorem}
For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of
Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a
finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on
it.
\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob
The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is
injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then
\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }
Z'\bigr\}\times\R .\]
If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.
\end{proposition}
\begin{proof}
A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive
identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,
times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and
the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space
of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the
fundamental group of the base.
\end{proof}
So there are loops of networks, along which the function computed changes continuously and returns to
itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged
roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space
of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.
\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}
\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]
At a reducible parameter the fibre of the realization map has positive dimension:
\begin{enumerate}[label=(\roman*)]
\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;
\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;
\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates
$(a_j,b_j,c)$.
\end{enumerate}
\end{proposition}
Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of
parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union
of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At
these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is
therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower
dimension, onto which the reducible strata of $\Theta$ are collapsed.
\section{Homogeneous activations}
\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]
Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector
field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing
parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$
wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$
at every parameter.
\end{proposition}
\begin{proof}
$\xi_j$ is the generator of the one-parameter group of symmetries of \cref{prop:syms}(iii); differentiate
$F(g_\lambda\theta,x)=F(\theta,x)$ at $\lambda=1$. The $\xi_j$ for different units have disjoint supports in
the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and
the bound holds on the closure of that set by semicontinuity of rank.
\end{proof}
The script confirms both statements on small networks: for $\tanh$ the Jacobian has full rank $P$ at a
random parameter and loses rank at reducible ones; for $\relu$ its rank never exceeds $P-H$ and the fields
$\xi_j$ lie in its kernel.
For $\relu$ the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space:
the orbit of a unit with zero outgoing weight is not closed, since
$(\lambda w,\lambda b,0)\to(0,0,0)$ as $\lambda\to0$. The correct object is the quotient stack, in which
such points are remembered with their stabilizers. This is the same reason that vector bundles in
\DT{Ch.~5} formed a stack and not a space, and the loss is a function on it.
\section{Invariant functions}
\begin{proposition}\label{prop:invariant}\Scl
Let $\loss\colon\Theta\to\R$ be invariant under a group $G$ of symmetries. Then the critical set of
$\loss$ is a union of $G$-orbits. If $G$ contains a one-parameter group with generator $\xi$, then
$d\loss(\xi)=0$ identically and, at every critical point, $\xi$ lies in the kernel of the Hessian. If
moreover $\nabla\lambda$ vanishes at $\ev(\theta)$ for a loss $\loss=\lambda\circ\ev$, then the whole fibre
of $\ev$ through $\theta$ is critical.
\end{proposition}
\begin{proof}
Differentiating $\loss(g_s\theta)=\loss(\theta)$ in $s$ gives $d\loss_\theta(\xi)=0$ for all $\theta$;
differentiating that identity in a direction $v$ at a critical point gives
$\nabla^2\loss(\xi,v)+d\loss(D_v\xi)=\nabla^2\loss(\xi,v)=0$. The last statement is
$\nabla\loss=J^{\mathsf T}\nabla\lambda$.
\end{proof}
\begin{corollary}\label{cor:notmorse}\Sob
For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the
form $\lambda\circ\ev$ is a Morse function on any open set containing a critical point at which some hidden
unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least
the number of such units.
\end{corollary}
So Morse theory in the form of \DT{Ch.~8} never applies directly to the loss of a $\relu$ network. It can
apply to the function induced on a slice transverse to the orbits, or in Morse--Bott form, with critical
manifolds in place of critical points. For $\tanh$ networks there is no continuous symmetry and no such
obstruction on $\Theta^{\mathrm{irr}}$; the degeneracies there come from the reducible strata and from
overparametrization (\cref{ch:loss}).
\section*{Notes}
Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann
and by Albertini and Sontag, and for deep $\tanh$ networks by Fefferman. The relation between reducibility
and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of
Watanabe's singular learning theory \cite{Wat09}. The use of permutation symmetry to compare trained
networks is discussed in \cref{ch:loss}.
\section*{Exercises}
\begin{exercise}\label{exer:verifysym}
Verify the three symmetries of \cref{prop:syms} for a network with one hidden layer.
\end{exercise}
\begin{exercise}\label{exer:rankdrop}
For the one-hidden-layer $\tanh$ network, show that if $a_j=0$ then $\rk J(\theta)\le P-(d+1)$ for every set of
inputs, and if $(w_j,b_j)=(w_k,b_k)$ then $\rk J(\theta)\le P-1$.
\end{exercise}
\begin{exercise}\label{exer:reluscale}
For $F(\theta,x)=a\,\relu(wx+b)+c$ with $d=1$, write down $\xi$ and verify $D_\theta F\cdot\xi=0$ by hand.
What is the functional dimension?
\end{exercise}
\begin{exercise}\label{exer:order}
Compute the order of $G$ for the $\tanh$ architectures with widths $(2,3,1)$ and $(2,4,4,1)$.
\end{exercise}
\part{Level Sets}
\chapter{Transversality and Decision Boundaries}\label{ch:decision}
A binary classifier is a function $f_\theta\colon X\to\R$ whose sign is the predicted class, and its
\term{decision boundary} is $B_\theta=f_\theta^{-1}(0)$. This chapter answers question (Q1) of
\cref{ch:families} for decision boundaries: for almost every parameter the boundary is a smooth hypersurface;
its isotopy type is constant on the components of the complement of a closed null set of parameters; and
every isotopy type of compact hypersurface occurs for wide enough networks.
\section{The boundary is generically a hypersurface}
\begin{theorem}[Generic decision boundaries]\label{thm:boundary}\Srf[\DT{Thm.~4.1, Thm.~1.12}]
Let $F\colon\Theta\times X\to\R$ be a family of the form $F((\theta',c),x)=G(\theta',x)+c$.
\begin{enumerate}[label=(\alph*)]
\item For every $\theta'$, the set of $c$ for which $0$ fails to be a regular value of $f_{(\theta',c)}$ has
measure zero.
\item For almost every $\theta$, the decision boundary $B_\theta$ is a smooth hypersurface in $X$, closed as
a subset and possibly empty, with trivial normal bundle oriented by $\nabla f_\theta$; and
$X\setminus B_\theta$ is the disjoint union of the open sets $\{f_\theta>0\}$ and $\{f_\theta<0\}$.
\item If $X$ is a compact manifold with boundary, then for almost every $\theta$ the boundary $B_\theta$ is
moreover a neat submanifold: it meets $\partial X$ transversally, in $\partial B_\theta$.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) $0$ is a regular value of $G(\theta',\cdot)+c$ exactly when $-c$ is a regular value of
$G(\theta',\cdot)$, and the critical values of a smooth function have measure zero by \DT{Thm.~4.1}. (b)
follows from (a) by Fubini's theorem and \DT{Thm.~1.12}. (c) Apply (a) also to the restriction to
$\partial X$ and use \DT{Thm.~1.17}.
\end{proof}
Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever
smooth activation it uses. Part (a) is stronger than the conclusion of \cref{prin:rich}, since it holds for
each value of the other parameters separately; it says that a non-smooth decision boundary can always be
repaired by an arbitrarily small change of the output bias alone. The level sets $f_\theta^{-1}(s)$ for
other thresholds $s$ are covered by the same statement.
\begin{example}\label{ex:cross}
Let $f(x)=x_1^2-x_2^2+c$. For $c\ne0$ the boundary is a hyperbola, a $1$-manifold with two components. For
$c=0$ it is a pair of crossing lines, not a manifold. In the language of \DT{Ch.~4}, the boundary at $c=0$ is
a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.
\end{example}
\section{Several classes}
A classifier with $k$ classes is a map $f_\theta\colon X\to\R^k$, and the predicted class is the index of the
largest coordinate. For a non-empty $I\subset\{1,\dots,k\}$ let $A_I\subset\R^k$ be the set of points whose
largest coordinate is attained exactly on $I$. Then $A_I$ is an open subset of the linear subspace
$\{y_i=y_j:i,j\in I\}$, a submanifold of codimension $|I|-1$, and the $A_I$ partition $\R^k$.
\begin{proposition}\label{prop:multiclass}\Srf[\DT{Thm.~4.5}]
Let $F\colon\Theta\times X\to\R^k$ be a family whose last layer is affine with a bias. For almost every
$\theta$, $f_\theta$ is transverse to every $A_I$. Then the decision region of class $i$ is the open set
$f_\theta^{-1}(A_{\{i\}})$; the set where exactly the classes in $I$ tie is a submanifold of codimension
$|I|-1$; and no more than $d+1$ classes tie anywhere.
\end{proposition}
\begin{proof}
$F$ is a submersion, so \DT{Thm.~4.5} applies to each of the finitely many $A_I$, and a finite intersection
of sets of full measure has full measure. A submanifold of codimension greater than $d$ is empty.
\end{proof}
In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time
at isolated points, like the cells of a foam, and four regions never meet at a point.
\section{Chambers}
Let $X$ be compact, and for simplicity without boundary. Define
\[\Sigma=\{(\theta,x):F(\theta,x)=0,\ d_xF(\theta,x)=0\}\subset\Theta\times X,
\qquad\Delta=\pi(\Sigma)\subset\Theta ,\]
the set of parameters for which $0$ is not a regular value: the \term{discriminant} of the family.
\begin{theorem}[Chambers]\label{thm:chambers}\Srf[\DT{Lem.~4.4, Thm.~8.5, Thm.~10.3}]
Let $F$ be as in \cref{thm:boundary} with $X$ compact. Then $\Delta$ is closed and of measure zero. If
$\gamma$ is a path in $\Theta\setminus\Delta$, there is a diffeotopy $\varphi_t$ of $X$ with
$\varphi_t(B_{\gamma(0)})=B_{\gamma(t)}$. In particular the diffeomorphism type of the pair $(X,B_\theta)$ is
constant on each connected component of $\Theta\setminus\Delta$.
\end{theorem}
\begin{proof}
$\Sigma$ is closed and $\pi|_\Sigma$ is proper because $X$ is compact, so $\Delta$ is closed; its complement
has full measure by \cref{thm:boundary}. Over $\Theta\setminus\Delta$ the set $W=F^{-1}(0)$ is a
submanifold of $\Theta\times X$, since $d_xF\ne0$ there, and by \DT{Lem.~4.4} the projection
$W\to\Theta\setminus\Delta$ is a submersion; it is proper. By Ehresmann's theorem \DT{Thm.~8.5} it is a fibre
bundle. Pulled back along $\gamma$ it is trivial, which gives an isotopy of embeddings
$B_{\gamma(0)}\to X$ with image $B_{\gamma(t)}$ at time $t$, and the isotopy extends to a diffeotopy of $X$ by
\DT{Thm.~10.3}.
\end{proof}
The components of $\Theta\setminus\Delta$ are the \term{chambers} of the architecture. The theorem says that
a classifier has a well-defined topology (the number of components of its decision boundary and of its
decision regions, their genera, how they are nested) which does not change under small changes of the
parameters, and can change only when the parameter crosses $\Delta$. What happens at a crossing is the
subject of the next chapter. The symmetry group of \cref{ch:symmetry} permutes the chambers.
For manifolds with boundary the same holds with $\Delta$ enlarged by the parameters for which $B_\theta$ is
tangent to $\partial X$.
\begin{proposition}[Walls]\label{prop:walls}\Sob
Consider the one-hidden-layer architecture
$F(\theta,x)=\sum_{j\le H}a_j\sigma(\langle w_j,x\rangle+b_j)+c$ on $\R^d$ with $\sigma'>0$. At every point
where the matrix $W$ with rows $w_j$ has rank $d$, the map
$\Phi(\theta,x)=(F,d_xF)\in\R\times\R^d$ is a submersion. Hence, over the open set of such parameters,
$\Sigma$ is a submanifold of dimension $P-1$, and $\Delta$ is the image of a manifold of dimension $P-1$.
\end{proposition}
\begin{proof}
$d_xF=W^{\mathsf T}(a\odot\sigma'(z))$ with $z=Wx+b$. A variation $\delta a$ changes $d_xF$ by
$W^{\mathsf T}(\delta a\odot\sigma'(z))$, and $\delta a\mapsto\delta a\odot\sigma'(z)$ is a linear
isomorphism since $\sigma'>0$; so if $W^{\mathsf T}$ is onto, every variation of $d_xF$ is produced. The
resulting change in $F$ is compensated by $\delta c$.
\end{proof}
So the discriminant is no larger than a hypersurface, and the picture of parameter space is that of open
chambers separated by walls of codimension one. The proposition is stated for the simplest architecture;
the argument needs only that the parameters can move the value and the input-gradient of the network
independently at each point, which holds for deeper networks at parameters where the layer Jacobians have
full rank.
\section{Every hypersurface is a decision boundary}
\begin{theorem}[Realization]\label{thm:realize}\Srf[\DT{Thm.~6.2}; Hornik \cite{Hor91}]
Let $K\subset\R^d$ be a compact domain with smooth boundary and $B\subset\operatorname{int}K$ a closed
hypersurface. Let $\sigma$ be smooth, bounded and non-constant. Then for all sufficiently large $H$ there is
a non-empty open set of parameters $\theta$ of the one-hidden-layer architecture with $H$ units for which
$B_\theta\cap K$ is carried to $B$ by a diffeomorphism of $K$ isotopic to the identity.
\end{theorem}
\begin{proof}
A closed hypersurface in $\R^d$ separates and so has a trivial normal bundle; using a tubular neighbourhood
\DT{Thm.~6.2} one constructs a smooth $g\colon K\to\R$ with $g^{-1}(0)=B$, $0$ a regular value, and $g\ne0$ on
$\partial K$. Since $B$ is compact, there is $\varepsilon>0$ such that every $h$ with
$\|h-g\|_{C^1(K)}<\varepsilon$ has $0$ as a regular value and does not vanish on $\partial K$. By Hornik's
theorem, networks with one hidden layer are dense in $C^1(K)$, so some $f_{\theta_0}$ is within
$\varepsilon$ of $g$, and so is $f_\theta$ for $\theta$ near $\theta_0$, the realization map being continuous
into $C^1(K)$. The family $(1-s)g+sf_\theta$, $s\in[0,1]$, stays within $\varepsilon$ of $g$, and the
argument of \cref{thm:chambers} applied to it gives the diffeotopy.
\end{proof}
So there is no topological restriction on decision boundaries beyond those satisfied by every compact
hypersurface in $\R^d$: it is orientable, it separates, and it bounds (\cref{prop:constraints}). The theorem
is silent on how large $H$ must be. Upper bounds for the Betti numbers of decision regions in terms of the
architecture, which give lower bounds for $H$, are due to Bianchini and Scarselli \cite{BS14}.
\section{Piecewise-linear networks}
For $\sigma=\relu$ a network is piecewise linear, its decision boundary is a polyhedral complex, and the
smooth theory does not apply as it stands. The counterpart of \cref{thm:boundary} is due to Grigsby and
Lindsey \cite{GL22}. They define what it means for a $\relu$ network to be generic and transversal, in
terms of the bent hyperplane arrangement formed by the zero sets of its units, and prove that almost all
$\relu$ networks have both properties. As an application they show that for the architecture with one
hidden layer of $d+1$ units, a decision region has at most one bounded connected component: a restriction on
topology coming from the architecture, of the kind \cref{thm:realize} rules out for unbounded width.
\section*{Notes}
That decision boundaries are generically manifolds is folklore, and is usually justified by Sard's theorem
in the form (a). The statement about chambers is the standard structure theory of discriminants in
singularity theory, applied to this family. For topological complexity of decision regions see
\cite{BS14}, and for an empirical study of how networks change the topology of data, \cite{NZL20}.
\section*{Exercises}
\begin{exercise}\label{exer:d1}
Let $d=1$ and $X=[-R,R]$. Show that for almost every $\theta$ the decision boundary is a finite set at which
$f_\theta$ changes sign, and that the parity of its cardinality is determined by the signs of
$f_\theta(\pm R)$.
\end{exercise}
\begin{exercise}\label{exer:plane}
Let $X$ be a closed disc in the plane. Describe the generic decision boundary up to diffeomorphism.
\end{exercise}
\begin{exercise}\label{exer:crossex}
For the family $F(c,x)=x_1^2-x_2^2+c$ on the plane, determine $\Sigma$ and $\Delta$, and describe the
chambers.
\end{exercise}
\begin{exercise}\label{exer:triple}
For a three-class classifier in the plane satisfying the conclusion of \cref{prop:multiclass}, show that
near a point where all three classes tie the decision regions look like three sectors meeting at a point.
\end{exercise}
\chapter{Topology Change along Paths: Surgery and Cobordism}\label{ch:paths}
By \cref{thm:chambers} the topology of a decision boundary can change during training only when the
parameter crosses the discriminant. This chapter answers question (Q2): for a generic path the crossings
are isolated, and at each of them the boundary changes by one surgery, determined by the index of a
critical point. The union of the boundaries along the path is a cobordism, and the surgeries are the
critical points of its height function. This is the Morse theory of \DT{Ch.~8} applied to a manifold that the
path itself creates.
\section{The trace of a path}
Let $F\colon\Theta\times X\to\R$ be a family as in \cref{thm:boundary}, with $X$ compact of dimension $d$, and
let $\gamma\colon[0,1]\to\Theta$ be a path. Write $f_t=f_{\gamma(t)}$ and $B_t=f_t^{-1}(0)$, and define
\[\Phi_\gamma\colon[0,1]\times X\to\R,\quad\Phi_\gamma(t,x)=F(\gamma(t),x);\qquad
W_\gamma=\Phi_\gamma^{-1}(0)\subset[0,1]\times X .\]
\begin{theorem}[The trace is a cobordism]\label{thm:trace}\Srf[\DT{Thm.~1.17, Thm.~4.5}]
If $0$ is a regular value of $\Phi_\gamma$ and of $f_0$ and $f_1$, then $W_\gamma$ is a compact neat
hypersurface in $[0,1]\times X$ with $\partial W_\gamma=B_0\sqcup B_1$: a cobordism from $B_0$ to $B_1$
embedded in $[0,1]\times X$. For a path with endpoints outside $\Delta$, this holds after an arbitrarily
small perturbation of the bias component of $\gamma$ fixed at the endpoints.
\end{theorem}
\begin{proof}
The first statement is \DT{Thm.~1.17}. For the second, replace the bias component $c(t)$ by
$c(t)+s\,\beta(t)$ with $\beta>0$ on $(0,1)$ and vanishing with all derivatives at the ends. The map
$(s,t,x)\mapsto\Phi_\gamma(t,x)+s\beta(t)$ is a submersion on $\{0<t<1\}$, so by \DT{Thm.~4.5} zero is a
regular value for almost every $s$; near $t=0,1$ it is one already because $f_0,f_1$ have $0$ as a
regular value.
\end{proof}
The times at which $\gamma$ meets $\Delta$ are, by \DT{Lem.~4.4}, exactly the critical values of the height
function $\tau\colon W_\gamma\to[0,1]$, $\tau(t,x)=t$. At such a point $(t_0,x_0)$ we have $d_xf_{t_0}(x_0)=0$:
$x_0$ is a critical point of $f_{t_0}$ with critical value $0$.
\section{Surgery}
\begin{theorem}[One surgery at a time]\label{thm:surgery}\Srf[\DT{Thm.~8.6, Cor.~8.7}]
Let $\gamma$ be a path for which $W_\gamma$ is a manifold and $\tau$ is a Morse function with distinct
critical values $t_1<\dots<t_r$ in $(0,1)$. Then:
\begin{enumerate}[label=(\alph*)]
\item On each interval of $[0,1]\setminus\{t_1,\dots,t_r\}$ the boundary $B_t$ moves by an ambient isotopy of
$X$.
\item At each $t_i$ there is exactly one point $x_i\in B_{t_i}$ at which $B_{t_i}$ fails to be a manifold. It
is a non-degenerate critical point of $f_{t_i}$ with $\partial_t\Phi_\gamma(t_i,x_i)\ne0$, and near
$(t_i,x_i)$ there are coordinates on $X$ in which
\[\Phi_\gamma(t,x)=\pm(t-t_i)-x_1^2-\dots-x_k^2+x_{k+1}^2+\dots+x_d^2 ,\]
with $k$ the index of $x_i$ as a critical point of $f_{t_i}$.
\item $B_{t_i+\varepsilon}$ is obtained from $B_{t_i-\varepsilon}$ by a surgery: a copy of
$S^{\lambda-1}\times D^{d-\lambda}$ is removed and $D^\lambda\times S^{d-\lambda-1}$ is glued in, where
$\lambda$ is the index of $\tau$ at $(t_i,x_i)$; $\lambda=k$ or $d-k$ according to the sign in (b).
Correspondingly the region $\{f_t\le0\}$ gains or loses a handle of index $k$.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) is \cref{thm:chambers}. For (b): at a critical point of $\tau$ on $W_\gamma$ the tangent space of
$W_\gamma$ is $\{dt=0\}$, so $d_x\Phi_\gamma=0$ and, $W_\gamma$ being a manifold, $\partial_t\Phi_\gamma\ne0$.
Solving $\Phi_\gamma=0$ for $t$ as a function of $x$ near the point exhibits $W_\gamma$ as a graph over $X$
on which $\tau$ is the function $t(x)$, whose Hessian at $x_i$ is
$-\nabla^2f_{t_i}(x_i)/\partial_t\Phi_\gamma$. So $\tau$ is non-degenerate there exactly when $x_i$ is a
non-degenerate critical point of $f_{t_i}$, and the normal form follows from the Morse lemma with parameters.
(c) is \DT{Thm.~8.6} for $\tau$ on $W_\gamma$: the level sets of $\tau$ are the $B_t$, and passing a critical
point of index $\lambda$ changes the level set by the stated surgery.
\end{proof}
In words: the topology of a decision boundary changes exactly when a critical point of the network function
crosses the threshold, and the change is read off from the index of the critical point and the direction of
crossing. \Cref{tab:surgery} lists the cases for inputs of dimension two and three.
\begin{table}[ht]\centering\small
\begin{tabular}{@{}llp{0.58\textwidth}@{}}
\toprule
$d$ & index $k$ of $x_i$ & effect on the region $\{f_t\le0\}$ when the critical value becomes negative\\
\midrule
$2$ & $0$ (minimum) & a new disc appears; the boundary gains a circle\\
$2$ & $1$ (saddle) & two parts of the region join, or one part meets itself; two boundary arcs
reconnect, so the number of boundary circles changes by $\pm1$\\
$2$ & $2$ (maximum) & a hole in the region is filled; a boundary circle disappears\\
$3$ & $0$ & a new ball appears; the boundary gains a sphere\\
$3$ & $1$ & a tube joins two parts of the region, or adds a handle to one\\
$3$ & $2$ & a tunnel through the region is closed by a disc, or a cavity is split in two\\
$3$ & $3$ & a cavity is filled; a boundary sphere disappears\\
\bottomrule
\end{tabular}
\caption{Elementary changes of topology of a decision region, by the index of the critical point crossing
zero from above. Crossing in the other direction reverses each.}\label{tab:surgery}
\end{table}
\begin{proposition}\label{prop:chi}\Sob
In the situation of \cref{thm:surgery}(c), $\chi(B_{t_i+\varepsilon})-\chi(B_{t_i-\varepsilon})$ equals
$2(-1)^\lambda$ if $d$ is odd and $0$ if $d$ is even, and the Euler characteristic of the region
$\{f_t\le0\}$ changes by $\pm(-1)^k$.
\end{proposition}
\begin{proof}
The change is $\chi(D^\lambda\times S^{d-\lambda-1})-\chi(S^{\lambda-1}\times D^{d-\lambda})
=\chi(S^{d-\lambda-1})-\chi(S^{\lambda-1})=(-1)^{d-\lambda-1}-(-1)^{\lambda-1}$. The second statement is
additivity of $\chi$ under attaching a $k$-cell.
\end{proof}
\begin{example}\label{ex:twoblobs}
Let $F(t,x)=(x_1^2-1)^2+x_2^2-t$ on the plane. The function $(x_1^2-1)^2+x_2^2$ has minima at $(\pm1,0)$ with
value $0$ and a saddle at the origin with value $1$. For $t<0$ the region $\{F\le0\}$ is empty. At $t=0$ two
discs are born (two critical points cross at the same time, which is not generic; a small asymmetry
separates them). At $t=1$ the saddle crosses and the two discs merge. The numbers of components are $0$,
$2$, $1$, and the Euler characteristics of the region $0$, $2$, $1$. The script counts components on a
grid.
\end{example}
\section{Genericity of paths}
Are the hypotheses of \cref{thm:surgery} generic? For paths in the space of all functions, yes: this is the
content of \DT{Thm.~8.12}. For paths in a given architecture one needs enough parameters to move critical
points and critical values independently.
\begin{proposition}\label{prop:genericpath}\Srf[\DT{Thm.~4.6, Thm.~8.12}]
For the architecture of \cref{prop:walls}, with $H\ge d$, let $\gamma$ be a path with endpoints in
$\Theta\setminus\Delta$ along which $W$ has rank $d$. Then arbitrarily close to $\gamma$, with the same
endpoints, there is a path satisfying the hypotheses of \cref{thm:surgery}.
\end{proposition}
\begin{proof}[Sketch]
By \cref{prop:walls} the map $(\theta,x)\mapsto(F,d_xF)$ is a submersion along the path, so the family of
perturbations of $\gamma$ by compactly supported variations of $(a,c)$ induces a family of maps
$(t,x)\mapsto(\Phi,d_x\Phi)$ transverse to the zero section; by \DT{Thm.~4.5} a generic member has $\Sigma$
pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at
those points, and distinctness of the times, are further conditions of positive codimension on the
$2$-jet, which a variation of $W$ and $b$ moves freely; \DT{Thm.~4.6} applies.
\end{proof}
A training path produced by gradient descent is one particular path, and nothing guarantees that it is
generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and
that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the
typical exception (\cref{ex:twoblobs}): a symmetric network crossing a wall has several critical points
crossing together.
\section{What cobordism does and does not see}
\begin{proposition}\label{prop:constraints}\Scl
Let $d\ge2$ and let $B=f^{-1}(0)\subset\R^d$ be compact with $0$ a regular value. Then $B$ is the boundary of a
compact region of $\R^d$; it is orientable, its normal bundle is trivial, and $TB\oplus\R$ is trivial. In
particular $B$ is null-cobordant, all its Stiefel--Whitney numbers vanish, and no closed non-orientable
surface is a decision boundary in $\R^3$.
\end{proposition}
\begin{proof}
Choose a closed ball $D$ containing $B$ in its interior. Its boundary sphere is connected and $f$ does not
vanish on it, so $f$ has a constant sign there, say positive. Then $\{f\le0\}\cap D$ is a compact manifold
with boundary $B$. The normal bundle is trivialized by $\nabla f$, and $TB\oplus\nu=T\R^d|_B$ is trivial.
\end{proof}
So the cobordism class of a decision boundary carries no information: it is always zero, as
\DT{Thm.~4.16} predicts for a fibre of a map to a contractible space. What carries information is
finer: the diffeomorphism type of the pair $(X,B_t)$ in each chamber, and the sequence of surgeries along a
path.
That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is
contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between
generic paths meets a short list of codimension-two events.
\begin{proposition}[Moves]\label{prop:moves}\Srf[Cerf; \DT{Thm.~8.12}]
Let $\gamma_0,\gamma_1$ be generic paths with the same endpoints in an architecture rich enough for jet
transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three
kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or
deleting a pair of consecutive surgeries of indices $\lambda,\lambda+1$ that cancel (a birth--death point of
$f_t$ on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that
touches zero and returns).
\end{proposition}
The proposition is the two-parameter analogue of \cref{thm:surgery}, and is the form Cerf's theorem takes
for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and
that anything invariant under them, for instance the parity of the number of surgeries of each index
modulo the cancellation of adjacent pairs, depends only on the endpoints.
\section*{Notes}
The description of how level sets change in one-parameter families is classical Morse and Cerf theory;
see Milnor's lectures on the h-cobordism theorem. Its use for decision boundaries during training appears
in empirical form in studies that track Betti numbers of decision regions and of data representations,
such as \cite{NZL20}. Persistent homology of the family $\{f_t\le s\}$ in the two parameters $(t,s)$ is the
natural computational tool, and the events of \cref{prop:moves} are the generic singularities of its
bifiltration.
\section*{Exercises}
\begin{exercise}\label{exer:blobs}
Verify \cref{ex:twoblobs}: find the critical points and indices of $(x_1^2-1)^2+x_2^2$ and the topology of
the sublevel sets.
\end{exercise}
\begin{exercise}\label{exer:d1path}
For $d=1$, show that along a generic path the points of the decision boundary are born and die in pairs, and
that the parity of their number is constant if $X$ is a circle.
\end{exercise}
\begin{exercise}\label{exer:chi3}
In $d=3$, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries,
and what are their indices?
\end{exercise}
\begin{exercise}\label{exer:hessgraph}
Prove the formula for the Hessian of $\tau$ used in the proof of \cref{thm:surgery}(b).
\end{exercise}
\chapter{The Loss Landscape}\label{ch:loss}
Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the
loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the
landscape is organized by the sublevel sets of the loss, to which the Morse theory of \DT{Ch.~8} applies
with two caveats established in \cref{ch:symmetry} and below: the loss is rarely a Morse function, and
parameter space is not compact.
\section{The interpolation manifold}
Let $\ev\colon\Theta\to\R^N$ be the evaluation map on $n$ data points, $N=nm$, and
$\loss=\tfrac12|\ev-y|^2$. Write $M_y=\ev^{-1}(y)=\loss^{-1}(0)$.
\begin{theorem}[Interpolating parameters]\label{thm:interp}
\Srf[\DT{Thm.~1.12, Thm.~4.1}; Cooper \cite{Coo21}]
\begin{enumerate}[label=(\alph*)]
\item For almost every $y\in\R^N$, $M_y$ is a submanifold of $\Theta$ of dimension $P-N$, possibly empty. If
$P<N$ it is empty for almost every $y$.
\item If $y$ is a regular value and $\theta\in M_y$, then $T_\theta M_y=\ker J(\theta)$ and
$\nabla^2\loss(\theta)=J^{\mathsf T}J$ has rank $N$: it vanishes on $T_\theta M_y$ and is positive definite on
its orthogonal complement.
\item $M_y$ is non-empty for $y$ in the image of $\ev$, and that image has non-empty interior if and only if
$J$ has rank $N$ somewhere.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) By \DT{Thm.~4.1} almost every $y$ is a regular value of $\ev$, and then \DT{Thm.~1.12} applies; if $P<N$
every point of the image is a critical value. (b) $T_\theta M_y=\ker D\ev_\theta$ by \DT{Thm.~1.12}, and the
Hessian is given by \cref{prop:hess}. (c) If $J$ has rank $N$ at $\theta$ then $\ev$ is open near $\theta$;
conversely if the rank is everywhere less than $N$ the image has measure zero.
\end{proof}
Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in
addition that for sufficiently wide networks $M_y$ is non-empty for generic data. In the overparametrized
regime $P>N$, then, the global minima of the loss are not isolated: they form a manifold whose dimension is
the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it.
In the terminology of Morse theory the loss is a Morse--Bott function near $M_y$. By \cref{prop:hess} its
normal curvature is the spectrum of the tangent kernel.
\begin{proposition}[Convergence near the interpolation manifold]\label{prop:PL}
\Sli[Liu--Zhu--Belkin \cite{LZB22}]
Let $U\subset\Theta$ be an open set on which the smallest eigenvalue of the tangent kernel $K=JJ^{\mathsf T}$
is at least $\mu>0$. Then $|\nabla\loss|^2\ge2\mu\,\loss$ on $U$. A gradient flow line starting at
$\theta_0$ satisfies $\loss(\gamma(t))\le\loss(\theta_0)e^{-2\mu t}$ and has length at most
$\sqrt{2\loss(\theta_0)/\mu}$ for as long as it stays in $U$; if the ball of that radius about $\theta_0$
lies in $U$, the flow line converges to a point of $M_y$.
\end{proposition}
\begin{proof}
$|\nabla\loss|^2=(\ev-y)^{\mathsf T}JJ^{\mathsf T}(\ev-y)\ge\mu|\ev-y|^2=2\mu\loss$. Then
$\frac{d}{dt}\loss=-|\nabla\loss|^2\le-2\mu\loss$, and
$\frac{d}{dt}\sqrt{\loss}=-|\nabla\loss|^2/(2\sqrt\loss)\le-\sqrt{\mu/2}\,|\nabla\loss|$, which integrates to
the bound on the length $\int|\dot\gamma|=\int|\nabla\loss|$.
\end{proof}
The inequality is the quantitative form of transversality: $\mu>0$ says that $\ev$ is a submersion on $U$
with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the
qualitative theory hands over to analysis.
\section{Sublevel sets and barriers}
For $c\in\R$ let $\Theta^c=\{\loss\le c\}$. Two parameters are \term{connected at level $c$} if they lie in
the same component of $\Theta^c$, and the \term{barrier} between them is the infimum of such $c$. This is the
notion studied empirically under the name of mode connectivity \cite{GIP18,DVSH18}.
\begin{proposition}[Barriers]\label{prop:barrier}\Srf[\DT{Cor.~8.7, Prop.~8.10}]
Let $\loss\colon\Theta\to\R$ be a Morse function with compact sublevel sets. Then the components of
$\Theta^c$ change only at critical values: a component is born at each local minimum, and two components
merge at a critical point of index $1$ whose two descending directions lead to different components. The
barrier between two local minima is the level at which their components merge; it is attained at a
critical point of index $1$, and equals the minimum over paths joining them of the maximum of $\loss$ along
the path.
\end{proposition}
So barriers are the death times in the degree-zero barcode of the sublevel functor \DT{Prop.~8.10}, and the
barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a
generic analytic loss with weight decay, $\loss+\tfrac\varepsilon2|\theta|^2$, on an architecture without
continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail
(\cref{prop:notproper}).
\begin{example}[Two parameters]\label{ex:hyperbola}
Let $F(a,b;x)=abx$ with one data point $(x,y)=(1,1)$, so $\loss=\tfrac12(ab-1)^2$. The interpolation manifold
is the hyperbola $ab=1$, of dimension $P-N=1$, with two components exchanged by the symmetry
$(a,b)\mapsto(-a,-b)$. The only other critical point is the origin, a saddle with $\loss=\tfrac12$ and
Hessian eigenvalues $\pm1$. For $c<\tfrac12$ the sublevel set has two components; at $c=\tfrac12$ they merge.
So the barrier between $(1,1)$ and $(-1,-1)$ is $\tfrac12$, attained on the straight segment between them,
and no path does better. After applying the symmetry to one of the two points they coincide. The script
checks the component counts on a grid.
\end{example}
The example is the smallest instance of a general phenomenon. By \cref{prop:invariant} minima come in orbits
of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a
barrier. The barrier between two trained networks should therefore be measured after choosing
representatives of their orbits, which is the content of the conjecture of Entezari et al.\ that barriers
largely disappear once permutations are accounted for \cite{EKSN22}. In the language of \cref{ch:symmetry},
the relevant sublevel sets are those of the function induced on the quotient $\Theta/G$.
Two results from the literature describe the overparametrized case. For a class of deep networks with
piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen
proved that every sublevel set of the loss is connected \cite{Ngu19}. And \c{S}im\c{s}ek et al.\ describe the
critical manifolds that the reducible strata of \cref{prop:fibres} contribute, and show how adding hidden
units connects the $|G|$ copies of a minimum \cite{SGJ21}.
\section{Why the loss is rarely a Morse function}
Three mechanisms make the loss degenerate, and they are independent.
\begin{enumerate}[label=(\roman*)]
\item \emph{Overparametrization.} If $P>N$ the zero-loss minima form the manifold $M_y$ of positive
dimension (\cref{thm:interp}).
\item \emph{Continuous symmetry.} For homogeneous activations every critical point has a Hessian of nullity
at least $H$ (\cref{cor:notmorse}).
\item \emph{Reducible strata.} A critical point of a smaller network gives a positive-dimensional family of
critical points of a larger one, along the fibres of \cref{prop:fibres}; these are typically saddles
\cite{FA00}.
\end{enumerate}
In each case the degeneracy is of Morse--Bott type on an open dense part of the critical set, and the handle
theorem \DT{Thm.~8.6} has a Morse--Bott version in which a critical manifold contributes a bundle of cells
over itself. But the density of Morse functions \DT{Cor.~4.8} is of no direct use: a generic perturbation of
the loss as a function on $\Theta$ is not the loss of any data set, and perturbing the data preserves all
three mechanisms. This is the point of \cref{prin:rich} again. Genericity must be established within the
family one actually has.
\section{Critical points at infinity}
\begin{proposition}\label{prop:notproper}\Sob
For architectures with a bounded activation the loss is not proper, and its infimum need not be attained.
For $F(w;x)=\tanh(wx)$ with the single data point $(1,1)$, $\loss(w)=\tfrac12(\tanh w-1)^2$ has no critical
point, its infimum $0$ is not attained, and every sublevel set $\Theta^c$ with $0<c<2$ is a half-line.
\end{proposition}
\begin{proof}
$\loss'(w)=(\tanh w-1)\operatorname{sech}^2w<0$ for all $w$.
\end{proof}
So the topology of sublevel sets can change with no critical point to account for it, the change being
carried off to infinity, and the regular interval theorem \DT{Thm.~8.4}, which needs compactness, does not
apply. Any use of Morse theory for losses must either restore compactness (weight decay, a bounded
parameter domain) or account for these ends explicitly, as one does for the Palais--Smale condition in
infinite dimensions.
\section*{Notes}
The manifold of global minima was described by Cooper \cite{Coo21}; the Polyak--\L{}ojasiewicz condition in
this setting is due to Liu, Zhu and Belkin. Mode connectivity was observed by Garipov et al.\ and Draxler et
al., and analysed for wide networks by Freeman and Bruna and by Nguyen; permutation alignment is studied in
\cite{EKSN22}. For the singular structure of the landscape at reducible parameters see \cite{FA00,SGJ21} and
Watanabe's theory \cite{Wat09}.
\section*{Exercises}
\begin{exercise}\label{exer:linreg}
For the linear family of \cref{exer:linear} with $P>n$ and features in general position, describe $M_y$ and
the Hessian of the loss.
\end{exercise}
\begin{exercise}\label{exer:hyperbola}
Verify \cref{ex:hyperbola}: find all critical points of $\tfrac12(ab-1)^2$, their Hessians, and the
components of the sublevel sets.
\end{exercise}
\begin{exercise}\label{exer:PL}
In the setting of \cref{prop:PL}, show that the limit point of the flow line lies within
$\sqrt{2\loss(\theta_0)/\mu}$ of $\theta_0$.
\end{exercise}
\begin{exercise}\label{exer:deeplin}
For $F(a_1,\dots,a_k;x)=a_1\cdots a_kx$ with data $(1,1)$, show that $M_y$ has $2^{k-1}$ components and
dimension $k-1$, and that the origin is a degenerate critical point for $k\ge3$.
\end{exercise}
\part{Maps}
\chapter{Embeddings, Isotopy, and Depth}\label{ch:isotopy}
The previous three chapters studied level sets. This one studies the maps themselves, and in particular
what a network does to a data set regarded as a subset of input space. The standing assumption, usually
called the manifold hypothesis \cite{FMN16}, is that the data lie on or near a compact submanifold
$K\subset\R^D$ of dimension $m$ much smaller than $D$, with the classes occupying disjoint closed subsets
$K_+,K_-$. A hidden layer of width $k$ is a map $h_\theta\colon\R^D\to\R^k$, and the question is what
$h_\theta(K)$ can be.
\section{The first layer generically embeds}
\begin{proposition}\label{prop:firstlayer}\Srf[Whitney; \DT{Thm.~4.5, Cor.~4.8, Prop.~3.12}]
Let $K\subset\R^D$ be a compact submanifold of dimension $m$, let $k\ge2m+1$, and let $\sigma$ be a
diffeomorphism of $\R$ onto an open interval. Then for almost every $W\in\R^{k\times D}$ and every $b$, the
map $x\mapsto\sigma(Wx+b)$ restricted to $K$ is an embedding. The set of such $(W,b)$ is open.
\end{proposition}
\begin{proof}
Applying $\sigma$ coordinatewise after a translation is a diffeomorphism onto its image, so the question is
whether $W|_K$ is an embedding. The map $(W,(x,y))\mapsto W(x-y)$ on
$\R^{k\times D}\times(K\times K\setminus\Delta)$ is a submersion to $\R^k$, so by \DT{Thm.~4.5} for almost
every $W$ its restriction to $K\times K\setminus\Delta$, a manifold of dimension $2m<k$, is transverse to
$0$ and hence misses it: $W|_K$ is injective. Likewise $(W,(x,v))\mapsto Wv$ on the unit tangent bundle of
$K$, of dimension $2m-1$, is a submersion, so for almost every $W$ no unit tangent vector is killed: $W|_K$ is
an immersion. An injective immersion of a compact manifold is an embedding. Openness is \DT{Prop.~3.12}.
\end{proof}
So a hidden layer wider than twice the intrinsic dimension of the data loses no topological information for
almost every choice of weights, however small it is compared with the ambient dimension $D$. This is
Whitney's embedding theorem read as a statement about random projections. Narrower layers may fail to be
injective on $K$, and that is where the topology of a representation can change.
\section{What injective networks cannot do}
A classifier ends with a linear map, so the last hidden representation must make the classes linearly
separable. If the network up to that point is injective, its restriction to $K$ is a homeomorphism onto
the image, and separability becomes a constraint on how $K_+$ and $K_-$ sit in the ambient space.
\begin{theorem}[Nested classes]\label{thm:obstruction}\Scl[invariance of domain; cf.\ \cite{Joh19}]
Let $h\colon\R^d\to\R^d$ be continuous and injective, let $\beta\subset\R^d$ be a closed topological ball with
boundary sphere $S$, and let $p$ be an interior point of $\beta$. Then $h(p)$ lies in the convex hull of
$h(S)$. In particular no hyperplane separates $h(p)$ from $h(S)$.
\end{theorem}
\begin{proof}
Suppose a linear functional $\lambda$ satisfies $\lambda>0$ on $h(S)$ and $\lambda(h(p))\le0$. Let $q$ be a
point of the compact set $h(\beta)$ at which $\lambda$ is smallest. Then $\lambda(q)\le0$, so $q\notin h(S)$
and $q\in h(\operatorname{int}\beta)$, which is open in $\R^d$ by invariance of domain. But a linear
functional has no local minimum on an open set.
\end{proof}
\begin{corollary}\label{cor:width}\Scl[\cite{Joh19,HS17}]
A feedforward network on $\R^d$ in which every hidden layer has width $d$, every weight matrix is
invertible, and the activation is injective, cannot separate a point from a sphere surrounding it. To do so
a network must either have a layer of width at least $d+1$ or use a layer that is not injective.
\end{corollary}
Both escapes are used in practice. A wider layer provides the extra dimension into which the inner class can
be lifted. A $\relu$ layer is not injective: it folds space along hyperplanes, and a network of $\relu$
layers can map a nested data set to a separable one by collapsing. Naitzat, Zhitnikov and Lim \cite{NZL20}
measured the Betti numbers of the two classes layer by layer and found that well-trained networks reduce
them steadily, faster with $\relu$ than with smooth invertible activations. In the terms of this chapter:
smooth invertible layers of constant width act by diffeomorphisms and cannot change topology, and the
network must then rely on the few layers where injectivity fails.
\section{Neural ordinary differential equations}
A \term{neural ODE} \cite{CRBD18} is the time-$T$ map $\varphi_\theta$ of the equation
$\dot x=v_\theta(x,t)$, where $v_\theta$ is a network with input $(x,t)$ and output in $\R^d$.
\begin{proposition}\label{prop:node}\Srf[\DT{Thm.~10.2}]
If $v_\theta$ is smooth with bounded velocity, as it is when its last hidden layer has a bounded activation,
then $t\mapsto\varphi^t_\theta$ is a diffeotopy of $\R^d$. Hence $\varphi_\theta$ is a diffeomorphism isotopic
to the identity, it preserves orientation, and for every compact $K\subset\R^d$ the image $\varphi_\theta(K)$
is ambient isotopic to $K$.
\end{proposition}
\begin{corollary}[Obstructions]\label{cor:nodeobst}\Scl[cf.\ Dupont--Doucet--Teh \cite{DDT19}]
A neural ODE on $\R^d$ cannot:
\begin{enumerate}[label=(\alph*)]
\item for $d=1$, reverse the order of two points;
\item map a sphere and a point inside it to linearly separable sets (\cref{thm:obstruction});
\item for $d=3$, map two linked circles to unlinked ones, or a knotted circle to a round one.
\end{enumerate}
\end{corollary}
\begin{proof}
(a) A diffeomorphism of $\R$ isotopic to the identity is increasing. (c) The linking number is the degree of
the Gauss map \DT{Ex.~7.12} and is invariant under isotopy by \DT{Thm.~7.2}; the knot type is an isotopy
invariant by definition.
\end{proof}
These are statements about exact representation. A neural ODE can still approximate a map that violates
them, at the cost of a vector field with large derivatives, and \cite{DDT19} observed that training then
becomes slow and unstable. Their remedy was to append extra coordinates to the state.
\section{Augmentation}
\begin{theorem}[Flows in the stable range]\label{thm:augment}
\Srf[\DT{Thm.~10.9, Thm.~10.3, Thm.~10.2}]
Let $K$ be a compact manifold of dimension $m$ and let $e_0,e_1\colon K\to\R^N$ be embeddings, with
$N\ge2m+2$. Then there is a smooth, compactly supported, time-dependent vector field on $\R^N$ whose
time-one map $\varphi$ satisfies $\varphi\circ e_0=e_1$.
\end{theorem}
\begin{proof}
By \DT{Thm.~10.9} the two embeddings are isotopic. By the isotopy extension theorem \DT{Thm.~10.3} the
isotopy is induced by a diffeotopy of $\R^N$ with compact support, and by \DT{Thm.~10.2} the diffeotopy is
the flow of its velocity field.
\end{proof}
\begin{corollary}[Separation after augmentation]\label{cor:separate}\Sob
Let $K=K_+\sqcup K_-\subset\R^d$ be a compact submanifold of dimension $m$, the union of two disjoint compact
submanifolds, and let $N\ge\max(d+1,2m+2)$. Then there is a compactly supported flow on $\R^N$ whose
time-one map carries $K_+\times\{0\}$ and $K_-\times\{0\}$ into the open half-spaces $\{x_N>0\}$ and
$\{x_N<0\}$. The vector field may be taken to be a neural network with one hidden layer and input
$(x,t)$.
\end{corollary}
\begin{proof}
Take $e_0$ the inclusion of $K$ in $\R^d\times\{0\}\subset\R^N$ and $e_1=e_0\pm(0,\dots,0,1)$ on $K_\pm$, and
apply \cref{thm:augment} to get a field $v$ with flow $\varphi$. For the last statement, approximate $v$ in
$C^1$ on a compact set containing the support, by Hornik's theorem \cite{Hor91}; by Gr\"onwall's inequality
the time-one maps are then close in $C^1$ on compact sets, and the conclusion, being an open condition on
the images of the compact sets $K_\pm$, persists.
\end{proof}
So the obstructions of \cref{cor:nodeobst} disappear once the state space has dimension at least $2m+2$,
where $m$ is the dimension of the data and not of the ambient space. For finite data sets $m=0$ and two
dimensions suffice, in agreement with (a): points on a line cannot be reordered by a flow, points in the
plane can. For curves the bound is $4$, and it is sharp, since circles link in $\R^3$. For connected data
of dimension $m\ge2$ the bound can be lowered to $2m+1$ by the second part of \DT{Thm.~10.9}. Zhang et al.\
\cite{ZGUA20} prove a related statement about maps: every homeomorphism of $\R^d$ can be embedded in a flow on
$\R^{2d}$, and hence approximated by a neural ODE or an invertible residual network acting there. The dimension count of \cref{thm:augment}
depends on the data alone.
Residual networks fit the same description. A block $x\mapsto x+g(x)$ with $g$ of Lipschitz constant less
than one is a diffeomorphism isotopic to the identity (\cref{prop:iresnet}), and a composite of such blocks
is a discretized isotopy. Depth, for such architectures, is time.
\section{The space of representations}
\begin{problem}\label{prob:calculus}\Spg
For a compact data manifold $K$ of dimension $m$ and width $k\ge m+3$, the space $\Emb(K,\R^k)$ of
representations that preserve the topology of the data is described by manifold calculus
\DT{Thm.~10.10}: it is the limit of a tower whose first stage is the space of immersions and whose higher
stages are built from configuration spaces of $K$. Determine which components and which homotopy classes of
this space are reached by the realization map of a given architecture, as a function of width and depth.
\end{problem}
For $k\ge2m+2$ the space is connected and the question is about higher homotopy; in the range
$m+3\le k\le2m+1$ already the set of components is interesting, and \cref{cor:nodeobst}(c) is the first
example outside that range.
\section*{Notes}
The picture of a network as progressively untangling data manifolds goes back at least to expository
writing of Olah in 2014, where \cref{thm:obstruction} for $d=2$ and the role of an extra dimension are
described. Minimal width for universal approximation is treated in \cite{HS17,Joh19}. Neural ODEs are from
\cite{CRBD18}, their limitations and augmentation from \cite{DDT19}, and approximation properties from
\cite{ZGUA20}.
\section*{Exercises}
\begin{exercise}\label{exer:monotone}
Show that the time-one map of a bounded smooth vector field on $\R$ is increasing.
\end{exercise}
\begin{exercise}\label{exer:swap}
Write down a compactly supported vector field on $\R^2$ whose time-one map exchanges the points $(\pm1,0)$.
\end{exercise}
\begin{exercise}\label{exer:link}
Show that the linking number of two disjoint closed curves in $\R^3$ is unchanged when both are moved by a
diffeomorphism isotopic to the identity.
\end{exercise}
\begin{exercise}\label{exer:proj}
Let $K$ be the curve $t\mapsto(\cos t,\sin t,\cos2t,\sin2t,\cos3t)$ in $\R^5$. Show that the projection to
the first two coordinates is an embedding of $K$, and that the projection to the third and fourth is not.
\end{exercise}
\chapter{Degree and Invertibility}\label{ch:degree}
Some architectures are designed so that every member is invertible: normalizing flows, invertible residual
networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both
cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from
\DT{Ch.~7}.
\section{Global invertibility}
\begin{theorem}[Hadamard]\label{thm:hadamard}\Scl
A local diffeomorphism $f\colon\R^d\to\R^d$ is a diffeomorphism if and only if it is proper.
\end{theorem}
\begin{proof}
A proper local diffeomorphism is a covering map, and $\R^d$ is simply connected. In terms of degree: every
point is a regular value, the sign of $\det Df$ is constant, and so the number of preimages of a point is
the absolute value of the degree, which is constant; it is $1$ because a covering of a simply connected
space is trivial.
\end{proof}
A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the
only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded
one.
\begin{proposition}[Invertible residual blocks]\label{prop:iresnet}
\Scl[Banach; Behrmann et al.\ \cite{BGC19}]
Let $g\colon\R^d\to\R^d$ have Lipschitz constant $L<1$ and let $f=\id+g$.
\begin{enumerate}[label=(\alph*)]
\item $f$ is a bijection, its inverse is Lipschitz with constant $(1-L)^{-1}$, and
$x_{j+1}=y-g(x_j)$ converges to $f^{-1}(y)$ with $|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|$.
\item If $g$ is smooth then $f$ is a diffeomorphism with $\det Df>0$.
\item $f_s=\id+sg$, $0\le s\le1$, is a diffeotopy from the identity to $f$. In particular $f$ is proper of
degree $1$.
\end{enumerate}
\end{proposition}
\begin{proof}
(a) $x\mapsto y-g(x)$ is a contraction; and $|f(x)-f(x')|\ge(1-L)|x-x'|$. (b), (c): the eigenvalues of $Dg$
have modulus at most $L$, so $1+sDg$ is invertible for $s\in[0,1]$, and its determinant is continuous,
non-zero, and equal to $1$ at $s=0$.
\end{proof}
Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning
networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common
use all realize diffeomorphisms in the identity component of $\Diff(\R^d)$, and by \DT{Thm.~10.2} every
such diffeomorphism with compact support is conversely the time-one map of a flow.
\section{Topological mismatch}
A normalizing flow models a probability measure $\nu$ on $\R^d$ as the image $h_*\mu$ of a simple measure
$\mu$, usually a Gaussian, under a diffeomorphism $h$.
\begin{proposition}\label{prop:mismatch}\Scl[cf.\ Cornish et al.\ \cite{CCDD20}]
For a homeomorphism $h$ of $\R^d$ and a Borel probability measure $\mu$,
$\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu)$. Hence if the supports of $\mu$ and $\nu$ are not
homeomorphic, no homeomorphism satisfies $h_*\mu=\nu$.
\end{proposition}
A Gaussian has support $\R^d$. A target supported on two disjoint balls, on a sphere, or on any proper
closed subset cannot be represented exactly, and Cornish et al.\ show that a sequence of flows approximating
such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature
are the ones differential topology suggests: a target supported on a submanifold $M^m\subset\R^d$ is modelled
by a flow on $\R^m$ followed by an embedding $\R^m\to\R^d$, and when $M$ is not diffeomorphic to an open
subset of $\R^m$, by several such charts with a discrete index, which is an atlas in the sense of
\DT{Ch.~1}. The minimal number of charts is an invariant of $M$; for a sphere it is two.
\section{Equilibrium layers}
A \term{deep equilibrium layer} outputs a solution $z$ of $z=g_\theta(z,x)$ for a network $g_\theta$. The
existence and number of solutions is a fixed-point problem, and \DT{Prop.~7.11} describes its three aspects:
a generic signed count, a derived fibre product, and a trace.
\begin{theorem}[Equilibria of a bounded layer]\label{thm:deq}\Srf[\DT{Thm.~4.5, Thm.~7.2, Thm.~7.9}]
Let $g(z)=\tanh(Az+u)$ on $\R^k$, with $A\in\R^{k\times k}$ and $u\in\R^k$.
\begin{enumerate}[label=(\alph*)]
\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an
eigenvalue of $Dg$ there. The fixed points are then finite in number.
\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed
points is odd, and at least one exists.
\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with
$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,
which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of
$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed
points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For
$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so
$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to
(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.
At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.
\end{proof}
In one dimension, $z=\tanh(az+u)$ has one solution for $a\le1$, and for $a>1$ and small $|u|$ three, with
signs $+,-,+$: two stable equilibria and an unstable one between them. The script finds the fixed points for
several $(a,u)$ and for a two-dimensional example and checks that the signed count is $1$.
Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the
transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The
exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set $\Delta$ of
\cref{ch:decision}; crossing it creates or destroys a pair of equilibria of opposite sign, as in the
birth--death family of \DT{Thm.~8.12}, and it is there that the output of an equilibrium layer can jump.
\section*{Notes}
The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and
Mazur, Palais). Invertible residual networks are from \cite{BGC19}. The observation that the topology of
the support obstructs normalizing flows, with a quantitative form, is in \cite{CCDD20}. Deep equilibrium
models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone
operator parametrizations) are a separate line of work, and \cref{thm:deq} describes what happens without
them.
\section*{Exercises}
\begin{exercise}\label{exer:banach}
Prove \cref{prop:iresnet}(a).
\end{exercise}
\begin{exercise}\label{exer:det}
Let $g$ be smooth with $\|Dg\|\le L<1$ everywhere. Show $\det(1+Dg)\ge(1-L)^d$.
\end{exercise}
\begin{exercise}\label{exer:deq1}
For $k=1$ and $u=0$, find the fixed points of $z=\tanh(az)$ as a function of $a$ and verify
\cref{thm:deq}(b). What happens at $a=1$?
\end{exercise}
\begin{exercise}\label{exer:supp}
Prove \cref{prop:mismatch}.
\end{exercise}
\part{Assessment}
\chapter{What Was Used, and What Is Missing}\label{ch:coda}
\section{The inputs}
\Cref{tab:used} lists the main statements of this book with the theorems of the companion volume from which
they follow. Every entry in the right-hand column was known by the mid-1970s.
\begin{table}[ht]\centering\small
\begin{tabular}{@{}p{0.47\textwidth}p{0.43\textwidth}@{}}
\toprule
statement here & from \cite{DTHC}\\
\midrule
backpropagation composes (\cref{thm:backprop}) & reverse chain rule, Prop.~2.11\\
generic decision boundaries are hypersurfaces (\cref{thm:boundary}) & Sard, Thm.~4.1; regular values, Thm.~1.12\\
chambers and walls (\cref{thm:chambers,prop:walls}) & fibrewise lemma 4.4; Ehresmann, Thm.~8.5; isotopy extension, Thm.~10.3\\
every hypersurface is realized (\cref{thm:realize}) & tubular neighbourhoods, Thm.~6.2; openness, Thm.~3.2\\
surgery along paths (\cref{thm:trace,thm:surgery}) & neat preimages, Thm.~1.17; handles, Thm.~8.6\\
moves between paths (\cref{prop:moves}) & Cerf, Thm.~8.12\\
interpolation manifold (\cref{thm:interp}) & regular values, Thm.~1.12; Sard, Thm.~4.1\\
barriers (\cref{prop:barrier}) & sublevel functor, Cor.~8.7, Prop.~8.10\\
first layer embeds (\cref{prop:firstlayer}) & parametric transversality, Thm.~4.5\\
neural ODEs are diffeotopies (\cref{prop:node}) & flows, Thm.~10.2\\
augmentation (\cref{thm:augment}) & isotopy in the stable range, Thm.~10.9; Thm.~10.3\\
odd number of equilibria (\cref{thm:deq}) & degree, Thm.~7.2; Thm.~4.5\\
\bottomrule
\end{tabular}
\caption{What is used.}\label{tab:used}
\end{table}
\section{What is and is not new}
Nothing in this book is a new theorem of differential topology, and most of its statements about networks
have appeared in the machine learning literature in some special case, with the references given in the
text: generic smoothness of decision boundaries, the manifold of global minima, the obstruction to neural
ODEs and its removal by augmentation, the mismatch of supports for flows. What the book adds is the uniform
derivation, with the hypothesis that makes each statement true stated as a transversality property of the
architecture (\cref{prin:rich}), and a small number of statements tagged as observations that we have not
found in this form: the description of the moduli of irreducible one-hidden-layer networks as a
configuration space, with its monodromy (\cref{prop:moduli}); the wall structure of the discriminant
(\cref{prop:walls}); the surgery description of training with its Euler characteristic count
(\cref{thm:surgery,prop:chi}) and its moves (\cref{prop:moves}); the dimension count $2m+2$ for
augmentation in terms of the data (\cref{thm:augment,cor:separate}); and the parity of the number of
equilibria (\cref{thm:deq}). Each is a short corollary and is offered as such.
The higher-categorical layer was used as language more than as a tool. The bicategory $\Para$ and the lens
functor organize \cref{ch:families,ch:backprop}; the quotient by symmetries is a stack and not a space for
homogeneous activations (\cref{ch:symmetry}); and a degenerate level set is a derived manifold with a
cobordism class (\cref{ex:cross}). The theorems that do the work are those of \cref{tab:used}.
\section{Limits}
\emph{Smoothness.} The standing assumption excludes $\relu$. For piecewise-linear networks the analogue of
\cref{thm:boundary} is the theorem of Grigsby and Lindsey \cite{GL22}; analogues of
\cref{thm:chambers,thm:surgery} would require stratified Morse theory, and are not proved here.
\emph{Genericity within the family.} ``Almost every parameter'' is with respect to Lebesgue measure on
$\Theta$, so the statements apply to parameters drawn from a distribution with a density, as at a random
initialization. They do not apply to distinguished parameters. In particular, if $g$ is an orthogonal
symmetry of the architecture, the gradient of an invariant loss is equivariant, and so the fixed-point set
of $g$ is invariant under gradient flow: a network initialized with two identical hidden units keeps them
identical for ever, and stays in a reducible stratum of measure zero where none of the generic statements
need hold.
\emph{Paths.} \Cref{thm:surgery} describes generic paths. A training trajectory is a particular path, the
solution of a differential equation, and we have proved nothing about whether it is generic. The honest
statement is \cref{prop:genericpath}: arbitrarily close to it there is a path with the stated behaviour.
\emph{Compactness.} Input domains are assumed compact for the isotopy statements, and parameter space is
never compact, which restricts the use of Morse theory for losses (\cref{prop:notproper}).
\emph{No rates.} With the single exception of \cref{prop:PL}, the statements are qualitative. They say that
a set has measure zero and not how small its neighbourhoods are; that an isotopy exists and not how large
its velocity must be; that a boundary of a given topology is realizable and not at what width. Each of these
has a quantitative counterpart in the literature that the present methods do not reach.
\emph{No learning theory.} Nothing here concerns generalization.
\section{Directions}
\begin{problem}[Piecewise-linear version]\label{prob:PL}\Spg
Prove the analogues of \cref{thm:chambers,thm:surgery} for $\relu$ networks that are generic and transversal
in the sense of \cite{GL22}, with handle attachments replaced by the elementary moves of stratified Morse
theory on the canonical polyhedral complex.
\end{problem}
\begin{problem}[Quantitative discriminant]\label{prob:quant}\Spg
For a given architecture and initialization distribution, estimate the measure of the
$\varepsilon$-neighbourhood of the discriminant $\Delta$ of \cref{ch:decision}, and hence the expected number
of surgeries along a path of given length. \Cref{prop:walls} suggests that $\Delta$ is a hypersurface and
that the answer is governed by its volume.
\end{problem}
\begin{problem}[The stack of networks]\label{prob:stack}\Spg
For homogeneous activations, describe the quotient stack $[\Theta/G]$ of \cref{ch:symmetry} and the critical
locus of a loss on it as a derived object, so that \cref{thm:interp} and the degeneracies listed in
\cref{ch:loss} become statements about a single space with a virtual dimension.
\end{problem}
Together with \cref{prob:calculus}, these are the places where the methods of the companion volume, and in
particular its higher-categorical parts (derived manifolds, stacks, the calculus of embeddings), would be
doing more than providing a vocabulary.
\appendix
\chapter{Dictionary}\label{app:dictionary}
{\small
\begin{longtable}{@{}p{0.25\textwidth}p{0.36\textwidth}p{0.29\textwidth}@{}}
\toprule
Network notion & Differential topology & Where\\
\midrule
\endhead
architecture $F\colon\Theta\times X\to Y$ & plot of the mapping object $[X,Y]$; morphism of $\Para$ & \cref{def:family,prop:plot}\\
composition of layers & composition of plots & \cref{def:para}\\
weight sharing, freezing & $2$-cells of $\Para$ & \cref{exer:sharing}\\
forward-mode derivative & tangent functor & \DT{Prop.~2.1}\\
backpropagation & reverse derivative as a functor to lenses & \cref{thm:backprop}\\
tangent kernel & Gram matrix $JJ^{\mathsf T}$ of the evaluation map & \cref{prop:ntk}\\
functional dimension & rank of the realization map & \cref{def:fdim,prop:rank}\\
permutation, sign, scaling symmetry & group of invertible $2$-cells & \cref{prop:syms}\\
redundant hidden unit & reducible parameter; positive-dimensional fibre & \cref{prop:fibres}\\
space of networks & quotient manifold, or stack & \cref{prop:moduli}\\
decision boundary & preimage of a regular value & \cref{thm:boundary}\\
multi-class boundaries & preimage of a stratification & \cref{prop:multiclass}\\
robust topology of a classifier & chamber of the discriminant complement & \cref{thm:chambers}\\
change of boundary topology & surgery; critical point of the trace & \cref{thm:surgery}\\
training run, seen in input space & embedded cobordism with a Morse function & \cref{thm:trace}\\
two runs with the same endpoints & Cerf moves & \cref{prop:moves}\\
interpolating parameters & fibre of the evaluation map over a regular value & \cref{thm:interp}\\
sharpness at a minimum & normal Hessian $J^{\mathsf T}J$ & \cref{prop:hess}\\
loss barrier, mode connectivity & death level in the degree-zero barcode & \cref{prop:barrier}\\
saturation, diverging weights & failure of properness & \cref{prop:notproper}\\
hidden representation & immersion or embedding of the data manifold & \cref{prop:firstlayer}\\
minimal width & invariance of domain & \cref{thm:obstruction}\\
neural ODE & diffeotopy & \cref{prop:node}\\
augmented neural ODE & isotopy of embeddings in the stable range & \cref{thm:augment}\\
residual block with small branch & diffeomorphism isotopic to the identity & \cref{prop:iresnet}\\
normalizing flow & diffeomorphism; pushforward of measures & \cref{prop:mismatch}\\
equilibrium layer & transverse fixed point; Lefschetz count & \cref{thm:deq}\\
\bottomrule
\end{longtable}
}
\chapter{Computations}\label{app:computations}
The script \texttt{verify-parametrized-networks.py} uses only the Python standard library. It implements small feedforward
networks (widths $(2,3,1)$, $(2,4,4,1)$ and a few others) with forward evaluation, forward-mode
differentiation by dual numbers and reverse-mode differentiation by backpropagation, a Runge--Kutta
integrator, a Jacobi eigenvalue routine and a flood-fill component counter. It performs 49 checks in about
three seconds, each printing \texttt{PASS} or \texttt{FAIL}, and exits with status zero if and only if all
pass. The checks recompute the examples in the text and test numerical instances of the structural
statements on random parameters with a fixed seed. They are not proofs, and a statement about almost every
parameter is tested only at the parameters sampled.
{\small
\begin{longtable}{@{}p{0.13\textwidth}p{0.79\textwidth}@{}}
\toprule
Chapter & What is checked\\
\midrule
\endhead
\ref{ch:families} & parameter counts (\cref{exer:count}); associativity of composition in $\Para$; the
derivative of a weight-tied network (\cref{exer:sharing}); the family $\theta+x^3$ (\cref{exer:notrich})\\
\ref{ch:backprop} & backpropagation against forward mode and finite differences for two architectures, and
for the input covector (\cref{thm:backprop}); the tangent kernel is positive semidefinite of rank $\rk J$;
the output dynamics $\dot u=-K(u-y)$ (\cref{prop:ntk}); the loss gradient; the Hessian as Gauss--Newton plus
curvature term (\cref{exer:hess})\\
\ref{ch:symmetry} & all $48$ symmetries of the $\tanh$ network $(2,3,1)$ and the size of an orbit
(\cref{prop:syms}); the scaling symmetry for $\relu$; rank $P=13$ at a random parameter and ranks $10$ and
$9$ at reducible ones (\cref{thm:ident}, \cref{exer:rankdrop}); for $\relu$, the generators $\xi_j$ in
$\ker J$ and $\rk J\le P-H$ (\cref{prop:relu}); a path of irreducible networks exchanging two hidden units
(\cref{prop:moduli})\\
\ref{ch:decision} & $\partial F/\partial c=1$; degenerate zeros of a one-dimensional network occur only at
the critical values (\cref{thm:boundary}(a)); the two chambers of $x_1^2-x_2^2+c$ (\cref{exer:crossex});
three sectors at a triple point (\cref{exer:triple}); the submersion of \cref{prop:walls}\\
\ref{ch:paths} & component counts, critical points and indices in \cref{ex:twoblobs}; the Euler
characteristic formula of \cref{prop:chi} for $d\le6$; the Hessian of the height function on the trace
(\cref{exer:hessgraph}); births and deaths of boundary points on a circle (\cref{exer:d1path})\\
\ref{ch:loss} & $\rk J=N$ and the dimension $P-N$ of the solution set; the Hessian $J^{\mathsf T}J$ with $4$
positive and $9$ zero eigenvalues (\cref{thm:interp}); the inequality of \cref{prop:PL}; sublevel
components, the barrier and the Hessians of \cref{ex:hyperbola}; \cref{prop:notproper}; \cref{exer:deeplin};
invariance of the loss under the symmetries\\
\ref{ch:isotopy} & random first layers of width $3$ embed a circle in $\R^5$ (\cref{prop:firstlayer}), and a
coordinate projection that does not (\cref{exer:proj}); the convex hull property of \cref{thm:obstruction}
for random injective networks; order preservation on the line; positivity of the Jacobian determinant and
Liouville's formula for a planar neural ODE (\cref{prop:node}); the swap of \cref{exer:swap}; invariance of
the linking number under a neural ODE in $\R^3$ (\cref{cor:nodeobst}); separation of nested circles after
augmentation (\cref{cor:separate})\\
\ref{ch:degree} & convergence of the fixed-point inversion and the determinant bound for a residual block
(\cref{prop:iresnet}, \cref{exer:det}); a non-proper local diffeomorphism (\cref{thm:hadamard}); odd numbers
of equilibria with signed count $1$ in one and two dimensions (\cref{thm:deq})\\
\bottomrule
\end{longtable}
}
\chapter{Solutions to the Exercises}\label{app:solutions}
Solutions to all exercises follow, by chapter. Where a solution relies on a computation, the computation is
also carried out by \texttt{verify-parametrized-networks.py}.
\section*{Chapter 1}
\sol{1.1} $(2,3,1)$: $P=3\cdot3+1\cdot4=13$, $H=3$. $(2,4,4,1)$: $P=4\cdot3+4\cdot5+1\cdot5=37$, $H=8$.
$(d,k,1)$: $P=k(d+1)+(k+1)=k(d+2)+1$, $H=k$.
\sol{1.2} Let $(\Theta,F)\colon X\to Y$, $(\Theta',G)\colon Y\to Z$, $(\Theta'',K)\colon Z\to V$. Both
bracketings have underlying function $(\theta,\theta',\theta'',x)\mapsto K(\theta'',G(\theta',F(\theta,x)))$,
on the parameter spaces $(\Theta\times\Theta')\times\Theta''$ and $\Theta\times(\Theta'\times\Theta'')$
respectively. The reassociation diffeomorphism $r$ satisfies $F_{\text{left}}=F_{\text{right}}\circ(r\times
\id_X)$, so $r$ and $r^{-1}$ are $2$-cells, mutually inverse.
\sol{1.3} The diagonal $\delta\colon\Theta\to\Theta\times\Theta$ gives a $2$-cell
$(\Theta,(F\circ F)\circ(\delta\times\id))\Rightarrow(\Theta\times\Theta,F\circ F)$, and its source is
$(\theta,x)\mapsto F(\theta,F(\theta,x))$, the weight-tied network. By the chain rule, the derivative of
$\theta\mapsto G(\delta(\theta))$ is $\partial_1G+\partial_2G$ evaluated on the diagonal, for
$G(\theta_1,\theta_2)=F(\theta_2,F(\theta_1,x))$.
\sol{1.4} $F(\theta,x)=\theta+x^3$. Every member has the degenerate critical point $x=0$. A function is
Morse when its $1$-jet prolongation is transverse to the set of jets with vanishing derivative; the family
of prolongations $(\theta,x)\mapsto(x,\theta+x^3,3x^2)$ is not transverse to $\{\text{third coordinate}=0\}$,
since the third coordinate has vanishing derivative in both $\theta$ and $x$ at $x=0$. The parameter moves
the value of the function but not its derivative, so the family is rich for conditions on values
(\cref{prin:rich}) and not for conditions on first derivatives. Adding a parameter $\theta_1x$ repairs this:
$\theta_0+\theta_1x+x^3$ is Morse for $\theta_1\ne0$.
\section*{Chapter 2}
\sol{2.1} Let $z=Wx+b$ and $h=\sigma(z)$. The lens of the layer has forward part $(W,b,x)\mapsto h$ and
backward part
$(W,b,x,w)\mapsto\bigl(\delta x^{\mathsf T},\ \delta,\ W^{\mathsf T}\delta\bigr)$ with
$\delta=\sigma'(z)\odot w$: the three components are the covectors for $W$, $b$ and $x$. The output layer
$(a,c;h)\mapsto a^{\mathsf T}h+c$ has backward part $(a,c,h,w)\mapsto(wh,\ w,\ wa)$. The loss
$\frac12(u-y)^2$ has gradient $e=u-y$ in $u$. Composing: the covector entering the output layer is $e$, so
$\nabla_a=e\,h$, $\nabla_c=e$, and the covector passed back is $e\,a$; then
$\delta=\sigma'(z)\odot(e\,a)$, $\nabla_W=\delta x^{\mathsf T}$, $\nabla_b=\delta$.
\sol{2.2} $\ker JJ^{\mathsf T}=\ker J^{\mathsf T}$ since $\langle v,JJ^{\mathsf T}v\rangle=|J^{\mathsf T}v|^2$;
so $\rk K=\rk J^{\mathsf T}=\rk J$. For scalar output $K$ is $n\times n$, invertible if and only if $\rk J=n$,
that is, if and only if the columns of $J$ span $\R^n$; the $k$-th column is the vector
$\bigl(\partial F/\partial\theta_k(\theta,x_i)\bigr)_{i}$.
\sol{2.3} By \DT{Prop.~2.9} the $2$-jet of $\lambda\circ\ev$ at $\theta$ has linear part $C A$ and quadratic part
$C\circ B+E\circ(A\times A)$, where $(A,B)$ is the $2$-jet of $\ev$ (so $A=J$ and $B=(\nabla^2\ev_k)_k$) and
$(C,E)$ that of $\lambda$ (so $C=\nabla\lambda^{\mathsf T}$, $E=\nabla^2\lambda$). The quadratic part is
$\sum_k\partial_k\lambda\,\nabla^2\ev_k+J^{\mathsf T}\nabla^2\lambda\,J$.
\sol{2.4} $\ev(\theta)=\Phi\theta$ with $\Phi$ the $n\times P$ matrix of rows $\phi(x_i)^{\mathsf T}$; so
$J=\Phi$ for all $\theta$ and $K=\Phi\Phi^{\mathsf T}$, the Gram matrix of the features. The fibres of $\ev$
are the affine subspaces $\theta_0+\ker\Phi$, of dimension $P-\rk\Phi$. The realization map is linear with
kernel $\{\theta:\langle\theta,\phi(x)\rangle=0\ \forall x\}$, so the functional dimension is the dimension
of the span of $\phi(X)$.
\section*{Chapter 3}
\sol{3.1} $F(\theta,x)=\sum_ja_j\sigma(\langle w_j,x\rangle+b_j)+c$. (i) Permuting the index $j$ reorders the
sum. (ii) If $\sigma$ is odd, $(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.
(iii) If $\sigma(\lambda s)=\lambda\sigma(s)$ for $\lambda>0$, then
$(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.
\sol{3.2} The columns of $J$ belonging to $(w_j,b_j)$ are
$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i$ and
$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i$; if $a_j=0$ these $d+1$ columns vanish, so
$\rk J\le P-(d+1)$. If $(w_j,b_j)=(w_k,b_k)$, the columns for $a_j$ and $a_k$ are both
$\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i$, so two columns coincide and $\rk J\le P-1$. (In the second
case the columns for $(w_j,b_j)$ and $(w_k,b_k)$ are also proportional, with ratio $a_j/a_k$, so the rank
drops by $d+2$ in all.)
\sol{3.3} $\theta=(w,b,a,c)$ and $\xi=(w,b,-a,0)$. Where $wx+b>0$, $F=a(wx+b)+c$ and
$D_\theta F=(ax,\ a,\ wx+b,\ 1)$, so $D_\theta F\cdot\xi=awx+ab-a(wx+b)=0$. Where $wx+b<0$, $F=c$ and
$D_\theta F=(0,0,0,1)$, again orthogonal to $\xi$. Here $P=4$, $H=1$, so the functional dimension is at most
$3$; it equals $3$ when $a\ne0$ and $w\ne0$, since the function $x\mapsto a\,\relu(wx+b)+c$ is determined by
its breakpoint $-b/w$, the slope $aw$ on the active side, and $c$ (and the side, which is discrete).
\sol{3.4} $(2,3,1)$: one hidden layer of $3$ units, $|G|=2^3\cdot3!=48$. $(2,4,4,1)$: two hidden layers of
$4$ units, $|G|=(2^4\cdot4!)^2=384^2=147456$.
\section*{Chapter 4}
\sol{4.1} By \cref{thm:boundary}(c), for almost every $\theta$ the set $B_\theta$ is a compact neat
$0$-dimensional submanifold of $[-R,R]$: a finite set of interior points at which $f_\theta'\ne0$, with
$f_\theta(\pm R)\ne0$. At each such point $f_\theta$ changes sign. So the number of zeros is even if
$f_\theta(-R)$ and $f_\theta(R)$ have the same sign and odd otherwise.
\sol{4.2} A compact neat $1$-dimensional submanifold of the disc: a finite disjoint union of circles in the
interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such
configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints
by non-crossing arcs.
\sol{4.3} $d_xF=(2x_1,-2x_2)$ vanishes only at $x=0$, where $F=c$. So $\Sigma=\{(0,0)\}$ and $\Delta=\{0\}$.
There are two chambers, $c>0$ and $c<0$. In each the boundary is a hyperbola; for $c<0$ its branches open
along the $x_1$-axis ($x_1^2=x_2^2-c\ge-c$) and for $c>0$ along the $x_2$-axis. The region $\{f<0\}$ is connected for
$c<0$ (it contains the origin) and has two components for $c>0$. (The plane is not compact; the statement about chambers holds here
because the family is a product near infinity up to scaling. Restricted to a disc the same description
holds.)
\sol{4.4} At a triple point $p$, $f_\theta$ is transverse to $A_{\{1,2,3\}}$, the line $y_1=y_2=y_3$ in $\R^3$,
of codimension $2$. So the map $x\mapsto(f_1-f_2,f_2-f_3)$ from the plane to $\R^2$ is a local
diffeomorphism at $p$, and in the coordinates $(u,v)=(f_1-f_2,f_2-f_3)$ the decision regions are
$\{u>0,\ u+v>0\}$, $\{u<0,\ v>0\}$ and $\{v<0,\ u+v<0\}$: three sectors bounded by the rays
$\{u=0,v>0\}$, $\{v=0,u>0\}$ and $\{u+v=0,u<0\}$.
\section*{Chapter 5}
\sol{5.1} $g=(x_1^2-1)^2+x_2^2$ has $\nabla g=(4x_1(x_1^2-1),2x_2)$, vanishing at $(0,0)$ and $(\pm1,0)$. The
Hessian is $\operatorname{diag}(12x_1^2-4,\,2)$: at $(\pm1,0)$ it is $\operatorname{diag}(8,2)$, index $0$,
value $0$; at the origin $\operatorname{diag}(-4,2)$, index $1$, value $1$. The sublevel set $\{g\le t\}$ is
empty for $t<0$, two discs around $(\pm1,0)$ for $0<t<1$ (on the line $x_2=0$ it is
$\{|x_1^2-1|\le\sqrt t\}$, two intervals when $t<1$), and one disc for $t>1$. Its Euler characteristic is
$0,2,1$, consistent with attaching two $0$-cells and then one $1$-cell.
\sol{5.2} Here $B_t$ is a finite set and $W_\gamma$ is a compact $1$-manifold with boundary $B_0\sqcup B_1$.
A critical point of $\tau$ on a $1$-manifold is a local minimum or maximum of $t$ along a curve: two points
of $B_t$ appear together or disappear together (surgery with $\lambda=0$ or $1$ on a $0$-manifold: add or
remove an $S^0$). Between critical times the points move continuously. If $X$ is a circle, $f_t$ changes
sign at each point of $B_t$ and returns to its original sign after one circuit, so $\#B_t$ is even for every
regular $t$.
\sol{5.3} Two. By \cref{prop:chi} with $d=3$, the Euler characteristic of the region changes by
$\pm(-1)^k$ at each surgery; the solid torus has $\chi=0$ and needs at least one $0$-handle to be non-empty,
so at least one more handle of odd index is needed. A $0$-handle (a ball appears, $k=0$) followed by a
$1$-handle attached to it (the ball becomes a solid torus, $k=1$) suffices. The boundary goes from empty to
a sphere to a torus.
\sol{5.4} Write $\Phi=\Phi_\gamma$ and let $t(x)$ be defined by $\Phi(t(x),x)=0$ near $x_i$, with
$t(x_i)=t_i$. Differentiating once, $\Phi_t\,\partial_jt+\partial_j\Phi=0$, so $\partial_jt(x_i)=0$ because
$d_x\Phi=0$ there. Differentiating again and evaluating at $x_i$, where the terms containing first
derivatives of $t$ vanish, gives $\Phi_t\,\partial_j\partial_kt+\partial_j\partial_k\Phi=0$. Hence
$\nabla^2t(x_i)=-\nabla^2_x\Phi(t_i,x_i)/\Phi_t(t_i,x_i)$.
\section*{Chapter 6}
\sol{6.1} $\ev(\theta)=\Phi\theta$ with $\Phi$ of rank $n$. Every $y$ is a regular value;
$M_y=\theta^*+\ker\Phi$ is an affine subspace of dimension $P-n$, where $\theta^*$ is any solution, for
instance the minimum-norm one $\Phi^{\mathsf T}(\Phi\Phi^{\mathsf T})^{-1}y$. The Hessian is the constant
matrix $\Phi^{\mathsf T}\Phi$, of rank $n$, with kernel $\ker\Phi$.
\sol{6.2} $\nabla\loss=(ab-1)(b,a)$, which vanishes when $ab=1$ or $(a,b)=0$. The Hessian is
$\bigl(\begin{smallmatrix}b^2&2ab-1\\2ab-1&a^2\end{smallmatrix}\bigr)$. On $ab=1$ it is
$\bigl(\begin{smallmatrix}b^2&1\\1&a^2\end{smallmatrix}\bigr)$, of determinant $a^2b^2-1=0$ and trace
$a^2+b^2>0$: rank $1$, with kernel spanned by $(a,-b)$, the tangent to the hyperbola. At the origin it is
$\bigl(\begin{smallmatrix}0&-1\\-1&0\end{smallmatrix}\bigr)$, with eigenvalues $\pm1$: a saddle of index
$1$ with value $\frac12$. The sublevel set is $\{1-\sqrt{2c}\le ab\le1+\sqrt{2c}\}$. For $c<\frac12$ the
lower bound is positive, and the set lies in the two quadrants $ab>0$, one component in each. For
$c\ge\frac12$ it contains the axes and is connected. Any path from $(1,1)$ to $(-1,-1)$ must cross the set
$ab=0$, where $\loss=\frac12$.
\sol{6.3} The flow line $\gamma$ has length at most $\ell=\sqrt{2\loss(\theta_0)/\mu}$ while it stays in $U$.
If the closed ball of radius $\ell$ about $\theta_0$ lies in $U$, the flow line cannot leave $U$: at the
first exit time it would be at distance at most its length, which is at most $\ell$, from $\theta_0$,
hence still inside. So the flow line is defined for all time, has finite length, and therefore converges; the
limit $\theta_\infty$ satisfies $|\theta_\infty-\theta_0|\le\ell$ and $\loss(\theta_\infty)=\lim
\loss(\gamma(t))=0$.
\sol{6.4} $\loss=\frac12(\prod a_i-1)^2$ and $M_y=\{\prod a_i=1\}$, the preimage of the regular value $1$ of
the product map (its gradient at a point with all $a_i\ne0$ is non-zero); so $\dim M_y=k-1$. On $M_y$ every
$a_i\ne0$ and the number of negative $a_i$ is even; each admissible sign pattern gives one component (a
graph over the positive orthant in the first $k-1$ coordinates), and there are $2^{k-1}$ sign patterns with
an even number of minus signs. The gradient is $(\prod a_i-1)\bigl(\prod_{j\ne i}a_j\bigr)_i$, which vanishes
at the origin for $k\ge2$; the Hessian at the origin has entries
$-\partial_i\partial_j\prod a_l$, which are zero for $k\ge3$. So the origin is a critical point with zero
Hessian.
\section*{Chapter 7}
\sol{7.1} Let $\varphi^t$ be the flow. If $x<y$ then $\varphi^t(x)\ne\varphi^t(y)$ for all $t$ by uniqueness
of solutions, so $\varphi^t(y)-\varphi^t(x)$ is continuous in $t$, never zero, and positive at $t=0$; hence
positive at $t=1$.
\sol{7.2} Let $\rho\colon[0,\infty)\to[0,1]$ be smooth with $\rho=1$ on $[0,2]$ and $\rho=0$ on $[3,\infty)$,
and let $v(x)=\pi\rho(|x|)\,(-x_2,x_1)$. The flow preserves $|x|$, since $v$ is tangent to circles, and on
$\{|x|\le2\}$ it is rotation with angular speed $\pi$. At time one the points $(\pm1,0)$ have been rotated
by the angle $\pi$ and are exchanged.
\sol{7.3} Let $\varphi_t$ be the diffeotopy and $K_t=\varphi_t(K)$, $L_t=\varphi_t(L)$; they are disjoint for
every $t$ because $\varphi_t$ is injective. The Gauss maps
$G_t\colon K\times L\to S^2$, $G_t(x,y)=(\varphi_t(y)-\varphi_t(x))/|\varphi_t(y)-\varphi_t(x)|$, form a
homotopy, and the degree is a homotopy invariant by \DT{Thm.~7.2}.
\sol{7.4} The first projection is $t\mapsto(\cos t,\sin t)$, the standard embedding of the circle. The second
is $t\mapsto(\cos2t,\sin2t)$, which takes the same value at $t$ and $t+\pi$; it is an immersion but not
injective, a double cover of the circle.
\section*{Chapter 8}
\sol{8.1} For fixed $y$ the map $T(x)=y-g(x)$ satisfies $|T(x)-T(x')|\le L|x-x'|$, so it has a unique fixed
point $x^*$, which is the unique solution of $x+g(x)=y$, and $|x_j-x^*|=|T^j(x_0)-T^j(x^*)|\le L^j|x_0-x^*|$.
So $f$ is a bijection. And $|f(x)-f(x')|\ge|x-x'|-|g(x)-g(x')|\ge(1-L)|x-x'|$, which says that $f^{-1}$ is
Lipschitz with constant $(1-L)^{-1}$.
\sol{8.2} The singular values of $1+Dg$ are at least $1-L$, since $|(1+Dg)v|\ge|v|-|Dg\,v|\ge(1-L)|v|$. The
absolute value of the determinant is the product of the singular values, so it is at least $(1-L)^d$; and
the determinant is positive by \cref{prop:iresnet}(b).
\sol{8.3} $z=0$ is always a fixed point, with $1-g'(0)=1-a$. For $a\le1$ it is the only one: $\tanh(az)$ is
strictly concave for $z>0$ with slope $a\le1$ at $0$, so $\tanh(az)<z$ for $z>0$, and symmetrically for
$z<0$. For $a>1$, $\tanh(az)>z$ for small $z>0$ and $\tanh(az)<1<z$ for $z>1$, so there is a fixed point
$z^*\in(0,1)$, unique by concavity, and $-z^*$ by symmetry. At $z^*$ the graph of $\tanh(az)$ crosses the
diagonal from above, so $g'(z^*)<1$ and the sign is $+$; at $0$ the sign of $1-a$ is $-$. The signed count is
$1-1+1=1$. For $a<1$ it is the single $+1$. At $a=1$ the fixed point $0$ is degenerate ($1-g'(0)=0$): this
is the exceptional parameter at which the pair $\pm z^*$ is born, a pitchfork, which is not a generic
birth--death because of the symmetry $z\mapsto-z$ present when $u=0$.
\sol{8.4} A point $y$ lies in $\operatorname{supp}(h_*\mu)$ if and only if every open neighbourhood $V$ of $y$
has $\mu(h^{-1}(V))>0$. As $h$ is a homeomorphism, the sets $h^{-1}(V)$ are exactly the open neighbourhoods of
$h^{-1}(y)$; so the condition is $h^{-1}(y)\in\operatorname{supp}\mu$. If $h_*\mu=\nu$ then
$h$ restricts to a homeomorphism from $\operatorname{supp}\mu$ to $\operatorname{supp}\nu$.
\backmatter
\chapter{References}
\begingroup\small
\renewcommand{\chapter}[2]{}
\begin{thebibliography}{CGGWZ22}\setlength{\itemsep}{1pt}
\newcommand{\bi}[1]{\bibitem[#1]{#1}}
\bi{BGC19} J.~Behrmann, W.~Grathwohl, R.~T.~Q.~Chen, D.~Duvenaud, J.-H.~Jacobsen, Invertible residual networks, in \emph{ICML 2019}, PMLR 97, 573--582.
\bi{BS14} M.~Bianchini, F.~Scarselli, On the complexity of neural network classifiers: a comparison between shallow and deep architectures, \emph{IEEE Trans.\ Neural Netw.\ Learn.\ Syst.} 25 (2014), 1553--1565.
\bi{CCDD20} R.~Cornish, A.~Caterini, G.~Deligiannidis, A.~Doucet, Relaxing bijectivity constraints with continuously indexed normalising flows, in \emph{ICML 2020}, PMLR 119.
\bi{CCG20} R.~Cockett, G.~Cruttwell, J.~Gallagher, J.-S.~P.~Lemay, B.~MacAdam, G.~Plotkin, D.~Pronk, Reverse derivative categories, in \emph{Computer Science Logic 2020}, LIPIcs 152, 18:1--18:16.
\bi{CGGWZ22} G.~S.~H.~Cruttwell, B.~Gavranovi\'c, N.~Ghani, P.~Wilson, F.~Zanasi, Categorical foundations of gradient-based learning, in \emph{ESOP 2022}, Lecture Notes in Computer Science 13240, Springer, 1--28.
\bi{Coo21} Y.~Cooper, Global minima of overparameterized neural networks, \emph{SIAM J.~Math.\ Data Sci.} 3 (2021), 676--691.
\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, in \emph{NeurIPS 2018}.
\bi{DDT19} E.~Dupont, A.~Doucet, Y.~W.~Teh, Augmented neural ODEs, in \emph{NeurIPS 2019}.
\bi{DTHC} Anonymous, \emph{Differential Topology in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.
\bi{DVSH18} F.~Draxler, K.~Veschgini, M.~Salmhofer, F.~Hamprecht, Essentially no barriers in neural network energy landscape, in \emph{ICML 2018}, PMLR 80.
\bi{EKSN22} R.~Entezari, H.~Sedghi, O.~Saukh, B.~Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, in \emph{ICLR 2022}.
\bi{FA00} K.~Fukumizu, S.~Amari, Local minima and plateaus in hierarchical structures of multilayer perceptrons, \emph{Neural Networks} 13 (2000), 317--327.
\bi{Fef94} C.~Fefferman, Reconstructing a neural net from its output, \emph{Rev.\ Mat.\ Iberoamericana} 10 (1994), 507--555.
\bi{FMN16} C.~Fefferman, S.~Mitter, H.~Narayanan, Testing the manifold hypothesis, \emph{J.~Amer.\ Math.\ Soc.} 29 (2016), 983--1049.
\bi{FST19} B.~Fong, D.~I.~Spivak, R.~Tuy\'eras, Backprop as functor: a compositional perspective on supervised learning, in \emph{LICS 2019}, IEEE, 2019.
\bi{Fuk96} K.~Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network, \emph{Neural Networks} 9 (1996), 871--879.
\bi{GIP18} T.~Garipov, P.~Izmailov, D.~Podoprikhin, D.~Vetrov, A.~G.~Wilson, Loss surfaces, mode connectivity, and fast ensembling of DNNs, in \emph{NeurIPS 2018}.
\bi{GL22} J.~E.~Grigsby, K.~Lindsey, On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks, \emph{SIAM J.~Appl.\ Algebra Geom.} 6 (2022), 216--242.
\bi{GLMW22} J.~E.~Grigsby, K.~Lindsey, R.~Meyerhoff, C.~Wu, Functional dimension of feedforward ReLU neural networks, preprint, arXiv:2209.04036.
\bi{Hir76} M.~W.~Hirsch, \emph{Differential Topology}, Graduate Texts in Mathematics 33, Springer, 1976.
\bi{Hor91} K.~Hornik, Approximation capabilities of multilayer feedforward networks, \emph{Neural Networks} 4 (1991), 251--257.
\bi{HS17} B.~Hanin, M.~Sellke, Approximating continuous functions by ReLU nets of minimal width, preprint, arXiv:1710.11278.
\bi{JGH18} A.~Jacot, F.~Gabriel, C.~Hongler, Neural tangent kernel: convergence and generalization in neural networks, in \emph{NeurIPS 2018}.
\bi{Joh19} J.~Johnson, Deep, skinny neural networks are not universal approximators, in \emph{ICLR 2019}.
\bi{LZB22} C.~Liu, L.~Zhu, M.~Belkin, Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, \emph{Appl.\ Comput.\ Harmon.\ Anal.} 59 (2022), 85--116.
\bi{Ngu19} Q.~Nguyen, On connected sublevel sets in deep learning, in \emph{ICML 2019}, PMLR 97.
\bi{NZL20} G.~Naitzat, A.~Zhitnikov, L.-H.~Lim, Topology of deep neural networks, \emph{J.~Mach.\ Learn.\ Res.} 21 (2020), no.~184, 1--40.
\bi{PL20} M.~Phuong, C.~H.~Lampert, Functional vs.\ parametric equivalence of ReLU networks, in \emph{ICLR 2020}.
\bi{SGJ21} B.~\c{S}im\c{s}ek, F.~Ged, A.~Jacot, F.~Spadaro, C.~Hongler, W.~Gerstner, J.~Brea, Geometry of the loss landscape in overparameterized neural networks: symmetries and invariances, in \emph{ICML 2021}, PMLR 139.
\bi{Sus92} H.~J.~Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, \emph{Neural Networks} 5 (1992), 589--593.
\bi{Wat09} S.~Watanabe, \emph{Algebraic Geometry and Statistical Learning Theory}, Cambridge University Press, 2009.
\bi{ZGUA20} H.~Zhang, X.~Gao, J.~Unterman, T.~Arodz, Approximation capabilities of neural ODEs and invertible residual networks, in \emph{ICML 2020}, PMLR 119.
\end{thebibliography}
\endgroup
\end{document}