Independent scholarship · Open accessThe monthly digest

Full manuscript · browser edition

Parametrized Neural Networks and Their Representations in Differential Topology

Continuous, single-column reading. Equations, tables, proofs, references and appendices stay in document order.

Contents · 84 sections

image

About this text. This monograph applies the results of Differential Topology in Higher Categories [9], the higher-categorical treatment of M. W. Hirsch’s Differential Topology [20] in the same series, to smooth families of maps defined by neural network architectures. It is written to be read after that volume and cites it by theorem number, in the form [9, 4.5]. It is an independent work and is not affiliated with or endorsed by the publisher of Hirsch’s book.

Status of results. Every numbered statement carries a tag: classical (standard mathematics, attributed), literature (established in the machine learning or mathematical literature, attributed), reformulation (a known result restated or specialized, with the derivation given here), observation (an elementary consequence we have not found stated in this form), or programme (a direction, not a result). The statements tagged reformulation and observation are corollaries of classical differential topology applied to parametrized families; none is claimed as a new theorem of differential topology, and none says anything quantitative about training or generalization.

Authorship and assistance. The text was drafted and typeset with a large language model (Anthropic’s Claude) under the direction of the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall, mathematical consistency checks, computation, and typesetting. The numerical examples and finite instances of the structural claims are recomputed by the accompanying script verify-parametrized-networks.py (3), which uses small networks implemented in pure Python. Proofs and bibliographic details have not been independently refereed; corrections are welcome.

Open access. Rights remain with the author.

Preface

A neural network architecture is a smooth map F ⁣:Θ×X→YF\colon\Theta\times X\to Y: a family of maps from inputs to outputs, parametrized by weights. Differential topology, in the form Hirsch gave it, is the study of what is true of a generic member of a family of maps and of what happens along a generic path in one. The fit is close enough that most of the qualitative questions asked about networks have answers that are corollaries of theorems proved before 1976, once the questions are phrased as questions about families.

Is the decision boundary of a classifier a manifold? For almost every value of the parameters, yes, by the parametric transversality theorem, provided the last layer has a bias. How does the boundary change during training? By ambient isotopy, except at isolated times, at each of which it undergoes one surgery whose type is the index of a critical point; this is Morse theory on the trace of the boundary. What is the set of parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary differential equation separate two linked circles? Not in dimension three, because its time-one map is ambient isotopic to the identity; in dimension four always, because there any two embeddings of a one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the companion volume, and this book supplies the paragraphs.

The higher-categorical layer of the companion volume enters in three places. A parametrized network is a plot of the mapping object [X,Y][X,Y] in smooth sets, and composition of layers is composition in a category of parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an architecture make the parameter space an atlas for a stack whose points are networks up to relabelling, and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is what one sees of them from nearby.

The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a rate of convergence or a generalization error. Activation functions are assumed smooth except where stated, and the piecewise-linear case, which needs a different transversality theory, is treated only by reference. What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks the examples.

Introduction

What is assumed

The companion volume [9], at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of Hirsch’s book. From machine learning, only the definition of a feedforward network and of gradient descent.

How the book is organized

Part I sets up families, their derivatives and their symmetries. Part II studies level sets: of the network in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss in parameter space (the interpolation manifold). Part III studies the maps themselves: when they are embeddings, when they are isotopic, when they are invertible. Part IV is an assessment. gives the dictionary between the two subjects in one table.

Conventions

Manifolds and maps are smooth. A family is a smooth map F ⁣:Θ×X→YF\colon\Theta\times X\to Y; we write fθ=F(θ,−)f_\theta=F(\theta,-). P=dim⁡ΘP=\dim\Theta is the number of parameters, d=dim⁡Xd=\dim X the input dimension, nn the number of data points. “Almost every” refers to Lebesgue measure on a Euclidean parameter space, and “generic” means belonging to a residual set whose complement has measure zero. An activation function σ ⁣:R→R\sigma\colon\mathbb{R}\to\mathbb{R} is smooth unless it is explicitly ReLU(s)=max⁡(s,0)\mathop{\mathrm{ReLU}}(s)=\max(s,0). Citations of the form [9, 4.5] refer to numbered statements of the companion volume.

Families

Families of Maps

Parametrized maps

Definition 1.1. A parametrized map from XX to YY is a pair (Θ,F)(\Theta,F) of a manifold Θ\Theta and a smooth map F ⁣:Θ×X→YF\colon\Theta\times X\to Y. Its members are the maps fθ=F(θ,−)f_\theta=F(\theta,-), and its realization map is θ↦fθ\theta\mapsto f_\theta, from Θ\Theta to C∞(X,Y)C^\infty(X,Y).

By [9, Thm. 1.6] a parametrized map is exactly a plot of the mapping object [X,Y][X,Y] in smooth sets, with domain Θ\Theta; the realization map is the function it induces on points. Everything in this book is a statement about plots of mapping objects. The topologies on C∞(X,Y)C^\infty(X,Y) play no direct role, since parameter spaces are finite-dimensional; what is used is the relation between properties of the total map FF and properties of its members.

Definition 1.2. Para\mathsf{Para} is the bicategory whose objects are manifolds, whose morphisms X→YX\to Y are parametrized maps, and whose 22-cells (Θ′,F′)⇒(Θ,F)(\Theta',F')\Rightarrow(\Theta,F) are smooth maps r ⁣:Θ′→Θr\colon\Theta'\to\Theta with F′=F∘(r×idX)F'=F\circ(r\times\mathrm{id}_X). The composite of (Θ,F) ⁣:X→Y(\Theta,F)\colon X\to Y and (Θ′,G) ⁣:Y→Z(\Theta',G)\colon Y\to Z is (Θ×Θ′, (θ,θ′,x)↦G(θ′,F(θ,x))).\bigl(\Theta\times\Theta',\ (\theta,\theta',x)\mapsto G(\theta',F(\theta,x))\bigr).

Composition is associative up to the canonical reassociation of products, and the identity of XX is parametrized by a point. The construction is due to Fong, Spivak and Tuyéras [15] and is developed in [5]. A 22-cell is a reparametrization: sharing weights between layers is the 22-cell induced by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.

Proposition 1.3. [reformulation; cf. [9, Thm. 1.6]; [15]] Composition in Para\mathsf{Para} is the internal composition [X,Y]×[Y,Z]→[X,Z][X,Y]\times[Y,Z]\to[X,Z] of smooth sets, evaluated on plots: the composite of two parametrized maps is the plot Θ×Θ′→[X,Y]×[Y,Z]→[X,Z]\Theta\times\Theta'\to[X,Y]\times[Y,Z]\to[X,Z]. Hence Para\mathsf{Para} is the bicategory of plots of the category of manifolds enriched in smooth sets.

Proof. The internal composition is the map of sheaves which on plots with domain UU sends (F,G)(F,G) to (u,x)↦G(u,F(u,x))(u,x)\mapsto G(u,F(u,x)). Precomposing with the product plot, whose components depend on separate factors of U=Θ×Θ′U=\Theta\times\Theta', gives the formula of 1.2. ◻

Architectures

Definition 1.4. A feedforward architecture with widths (n0,…,nL)(n_0,\dots,n_L) and activation σ\sigma is the composite in Para\mathsf{Para} of the layers ℓi ⁣:Rni−1→Rni,ℓi(Wi,bi;x)=σ(Wix+bi)(i<L),ℓL ⁣:RnL−1→RnL,ℓL(WL,bL;x)=WLx+bL,\begin{aligned} \ell_i&\colon\mathbb{R}^{n_{i-1}}\to\mathbb{R}^{n_i}, & \ell_i(W_i,b_i;x)&=\sigma(W_ix+b_i)\qquad(i<L),\\ \ell_L&\colon\mathbb{R}^{n_{L-1}}\to\mathbb{R}^{n_L}, & \ell_L(W_L,b_L;x)&=W_Lx+b_L, \end{aligned} with σ\sigma applied to each coordinate. Its parameter space is Θ=∏i(Rni×ni−1×Rni)\Theta=\prod_i(\mathbb{R}^{n_i\times n_{i-1}}\times\mathbb{R}^{n_i}), of dimension P=∑ini(ni−1+1)P=\sum_in_i(n_{i-1}+1). The number of hidden units is H=n1+⋯+nL−1H=n_1+\dots+n_{L-1}.

If σ\sigma is smooth the family is smooth, and real-analytic if σ\sigma is (as tanh⁡\tanh, the logistic function and softplus are). For σ=ReLU\sigma=\mathop{\mathrm{ReLU}} the family is continuous and piecewise polynomial in (θ,x)(\theta,x), and it is not a morphism of Para\mathsf{Para} as defined; statements about it are made separately. Convolutional and attention layers are parametrized maps in the same sense, with 22-cells expressing weight sharing. Nothing below depends on the feedforward form except where an explicit hypothesis on the last or first layer is made.

The image of the realization map, the set F={fθ}\mathcal F=\{f_\theta\} of functions the architecture can compute, is a subset of C∞(X,Y)C^\infty(X,Y). With the plots that factor locally through Θ\Theta it is a diffeological space, the quotient of Θ\Theta by the relation fθ=fθ′f_\theta=f_{\theta'}. It is in general not a manifold; 3 describes its singularities.

Data, loss, and training

Fix inputs x1,…,xn∈Xx_1,\dots,x_n\in X and targets y1,…,yn∈Y=Rmy_1,\dots,y_n\in Y=\mathbb{R}^m. The evaluation map is ev ⁣:Θ→Yn=Rnm,ev(θ)=(fθ(x1),…,fθ(xn)),\mathrm{ev}\colon\Theta\to Y^n=\mathbb{R}^{nm},\qquad\mathrm{ev}(\theta)=\bigl(f_\theta(x_1),\dots,f_\theta(x_n)\bigr), and the squared loss is L(θ)=12∣ev(θ)−y∣2\mathcal{L}(\theta)=\tfrac12|\mathrm{ev}(\theta)-y|^2. More generally a loss is L=λ∘ev\mathcal{L}=\lambda\circ\mathrm{ev} for a smooth λ ⁣:Rnm→R\lambda\colon\mathbb{R}^{nm}\to\mathbb{R}. A training path is a smooth curve γ ⁣:[0,T]→Θ\gamma\colon[0,T]\to\Theta; gradient flow is the curve with γ˙=−∇L(γ)\dot\gamma=-\nabla\mathcal{L}(\gamma), for the Euclidean metric on Θ\Theta. A path in Θ\Theta is a plot [0,T]→[X,Y][0,T]\to[X,Y]: a one-parameter family of maps. When every member is an embedding of a submanifold K⊂XK\subset X it is an isotopy of KK in the sense of [9, Def. 10.1].

Three kinds of question organize the rest of the book.

  1. What does a generic member fθf_\theta look like? (Transversality and Decision Boundaries, Embeddings, Isotopy, and Depth, Degree and Invertibility)

  2. What happens to the members along a generic path? (2)

  3. What does the set of parameters satisfying a condition, such as L(θ)=0\mathcal{L}(\theta)=0, look like? (Symmetries and the Moduli of Networks, The Loss Landscape)

In each case the answer comes from the same mechanism. A condition on a member, or on a parameter, is the condition that a point lies in the preimage of a submanifold under a map built from FF; the preimage is a manifold when the map is transverse to the submanifold; and transversality of the map built from FF can be checked on the architecture, usually by exhibiting one direction in parameter space, a bias, in which the map moves freely.

Principle 1.5 (Richness). [reformulation; cf. [9, Thm. 4.5]] Let F ⁣:Θ×X→YF\colon\Theta\times X\to Y be a family and A⊂YA\subset Y a submanifold. If FF is transverse to AA, in particular if FF is a submersion, then fθf_\theta is transverse to AA for almost every θ\theta. A feedforward architecture is a submersion onto Y=RnLY=\mathbb{R}^{n_L} because of the last-layer bias: ∂F/∂bL=id\partial F/\partial b_L=\mathrm{id}.

This is [9, Thm. 4.5] with its hypothesis verified. It is worth stating separately because the hypothesis is about the family and not about function space. A statement that a property is generic among all smooth maps (for instance, that Morse functions are dense) says nothing about a given architecture, whose realization is a thin subset of all maps. Genericity transfers to an architecture only through a transversality property of FF itself, and each use of 1.5 below names the parameters that provide it.

The dictionary in brief

network notion object of differential topology Ch.
architecture plot of a mapping object; morphism of Para\mathsf{Para} 1
backpropagation functoriality of the reverse derivative 2
weight-space symmetry group action with invariant realization 3
decision boundary preimage of a regular value 1
change of topology in training handle attachment on the trace 2
interpolating parameters fibre of the evaluation map 3
barrier between minima death level in the sublevel barcode 3
hidden representation of data immersion or embedding of a manifold 1
neural ODE, residual network diffeotopy; isotopy of embeddings 1
invertible network diffeomorphism of degree one 2
equilibrium of an implicit layer transverse fixed point 2

Notes

The categorical treatment of learners as parametrized maps with update and request functions begins with [15]; the bicategory Para\mathsf{Para} and its combination with lenses and reverse derivatives is in [5]. The point of view that a network is a family in the sense of singularity theory, to which transversality applies, is implicit in much work on loss landscapes, and explicit for piecewise-linear networks in [18].

Exercises

Exercise 1.1. Compute PP and HH for the architectures with widths (2,3,1)(2,3,1), (2,4,4,1)(2,4,4,1) and (d,k,1)(d,k,1).

Exercise 1.2. Verify that composition in Para\mathsf{Para} is associative up to the reassociation (Θ×Θ′)×Θ′′≅Θ×(Θ′×Θ′′)(\Theta\times\Theta')\times\Theta''\cong\Theta\times(\Theta'\times\Theta''), and that this isomorphism is a 22-cell in both directions.

Exercise 1.3. Let (Θ,F) ⁣:X→X(\Theta,F)\colon X\to X be a layer. Express the weight-tied network fθ∘fθf_\theta\circ f_\theta as the source of a 22-cell into the composite (Θ×Θ,F∘F)(\Theta\times\Theta,F\circ F), and show that its derivative with respect to θ\theta is the sum of the two partial derivatives of the untied network on the diagonal.

Exercise 1.4. Give an example of a family F ⁣:R×R→RF\colon\mathbb{R}\times\mathbb{R}\to\mathbb{R} no member of which is a Morse function, although Morse functions are dense in C∞(R,R)C^\infty(\mathbb{R},\mathbb{R}). Which hypothesis of 1.5 fails for the relevant jet condition?

Derivatives

Two derivatives of a family matter: with respect to the input, which governs the geometry of each member, and with respect to the parameters, which governs training. Both are instances of the tangent functor of [9, Ch. 2], and the algorithm that computes the second efficiently is the functoriality of its transpose.

Backpropagation

Recall from [9, Def. 2.10] the category Lens\mathsf{Lens} and from [9, Prop. 2.11] the functor f↦(f,R[f])f\mapsto(f,R[f]), R[f](x,w)=(Dfx)TwR[f](x,w)=(Df_x)^{\mathsf T}w, from Euclidean spaces to lenses. For a parametrized map F ⁣:Θ×X→YF\colon\Theta\times X\to Y the reverse derivative has two components, R[F](θ,x,w)=(RΘ[F](θ,x,w), RX[F](θ,x,w))=((DθF)Tw, (DxF)Tw),R[F](\theta,x,w)=\bigl(R_\Theta[F](\theta,x,w),\ R_X[F](\theta,x,w)\bigr) =\bigl((D_\theta F)^{\mathsf T}w,\ (D_xF)^{\mathsf T}w\bigr), the first being the update direction for the parameters and the second the covector passed back to the previous layer.

Theorem 2.1 (Backpropagation is a functor). [literature; Fong–Spivak–Tuyéras [15]; Cruttwell et al. [5]] The assignment (Θ,F)↦(Θ,(F,R[F]))(\Theta,F)\mapsto(\Theta,(F,R[F])) is a functor of bicategories from Para\mathsf{Para} over Euclidean spaces to parametrized lenses. In particular, for a composite of layers F=ℓL∘⋯∘ℓ1F=\ell_L\circ\dots\circ\ell_1 the component RΘi[F]R_{\Theta_i}[F] is obtained by composing the lenses of the layers: a forward pass computing the activations xi=ℓi(θi,xi−1)x_i=\ell_i(\theta_i,x_{i-1}), and a backward pass computing wi−1=RX[ℓi](θi,xi−1,wi)w_{i-1}=R_X[\ell_i](\theta_i, x_{i-1},w_i) and emitting RΘ[ℓi](θi,xi−1,wi)R_\Theta[\ell_i](\theta_i,x_{i-1},w_i).

Proof. Composition in Para\mathsf{Para} is composition of maps after taking products with identities, and RR is a product-preserving functor to lenses by [9, Prop. 2.11]. ◻

The gradient of a loss L=λ∘ev\mathcal{L}=\lambda\circ\mathrm{ev} is R[ev](θ,∇λ)R[\mathrm{ev}](\theta,\nabla\lambda). The theorem says the cost of the gradient is that of one forward and one backward pass, whatever the number of parameters, which is the practical content of reverse-mode differentiation. The script implements both modes for a small network and checks them against each other and against finite differences.

The Jacobian of the evaluation map

Definition 2.2. For data x1,…,xnx_1,\dots,x_n the parameter Jacobian is J(θ)=DevθJ(\theta)=D\mathrm{ev}_\theta, an nm×Pnm\times P matrix. The tangent kernel is the nm×nmnm\times nm matrix K(θ)=J(θ)J(θ)TK(\theta)=J(\theta)J(\theta)^{\mathsf T}.

Proposition 2.3. [classical; Jacot–Gabriel–Hongler [23]] K(θ)K(\theta) is symmetric positive semidefinite of rank rkJ(θ)\mathop{\mathrm{rk}}J(\theta). Along gradient flow of L=λ∘ev\mathcal{L}=\lambda\circ\mathrm{ev} the outputs u(t)=ev(γ(t))u(t)=\mathrm{ev}(\gamma(t)) satisfy u˙=−K(γ)∇λ(u)\dot u=-K(\gamma)\nabla\lambda(u).

Proof. u˙=Jγ˙=−J∇L=−JJT∇λ\dot u=J\dot\gamma=-J\nabla\mathcal{L}=-JJ^{\mathsf T}\nabla\lambda. ◻

So the tangent kernel is the Riemannian metric that the Euclidean metric of Θ\Theta induces, by pushing forward along ev\mathrm{ev}, on the cotangent directions of output space; where JJ is onto, KK is invertible and the outputs follow a gradient flow for the metric K−1K^{-1}.

Rank

The rank of JJ is the first invariant of a parameter. It is lower semicontinuous, and for analytic architectures its generic value is attained almost everywhere.

Proposition 2.4. [classical; ] Let FF be real-analytic in θ\theta on a connected Θ\Theta, and let rr be the maximum of rkJ(θ)\mathop{\mathrm{rk}}J(\theta). Then {θ:rkJ(θ)=r}\{\theta:\mathop{\mathrm{rk}}J(\theta)=r\} is open and dense and its complement has measure zero. On it the fibres of ev\mathrm{ev} are submanifolds of dimension P−rP-r, and ev\mathrm{ev} is locally a submersion onto an rr-dimensional submanifold of Rnm\mathbb{R}^{nm}.

Proof. The complement is the common zero set of the r×rr\times r minors of JJ, which are analytic and not all identically zero. The zero set of a non-zero analytic function on a connected domain is closed with empty interior and measure zero. The rest is the constant rank theorem. ◻

Definition 2.5. The functional dimension of the architecture at θ\theta is the supremum over all finite sets of inputs of rkJ(θ)\mathop{\mathrm{rk}}J(\theta): the rank of the differential of the realization map [19].

The functional dimension is at most PP, and 3 shows when it is smaller. The strata {rkJ=s}\{\mathop{\mathrm{rk}}J=s\} for s<rs<r are where the fibres of ev\mathrm{ev} jump in dimension. They contain the parameters at which hidden units are redundant, and they are responsible for most of what is singular in the loss landscape (3).

Second derivatives

By the composition law for 22-jets [9, Prop. 2.9], the Hessian of a composite λ∘ev\lambda\circ\mathrm{ev} is ∇2L(θ)=JT ∇2λ J+∑k∂λ∂uk ∇2evk(θ),\nabla^2\mathcal{L}(\theta)=J^{\mathsf T}\,\nabla^2\lambda\,J+\sum_k\frac{\partial\lambda}{\partial u_k}\, \nabla^2\mathrm{ev}_k(\theta), a Gauss–Newton term and a term weighted by the gradient of λ\lambda in output space.

Proposition 2.6. [classical; ] At a parameter where ∇λ(ev(θ))=0\nabla\lambda(\mathrm{ev}(\theta))=0, in particular at a zero of the squared loss, ∇2L=JT∇2λ J\nabla^2\mathcal{L}=J^{\mathsf T}\nabla^2\lambda\,J. For the squared loss this is JTJJ^{\mathsf T}J, positive semidefinite of rank rkJ\mathop{\mathrm{rk}}J, with kernel ker⁡J\ker J.

The non-zero eigenvalues of JTJJ^{\mathsf T}J and of the tangent kernel JJTJJ^{\mathsf T} coincide. So at an interpolating parameter the curvature of the loss in the directions normal to the solution set is the spectrum of the tangent kernel.

Notes

Reverse-mode differentiation has been rediscovered several times; Griewank’s history is the standard reference. The compositional formulation is from [15] and [5], and the axiomatics of reverse derivatives from [4]. The tangent kernel was introduced in [23]. Functional dimension for ReLU\mathop{\mathrm{ReLU}} networks is studied in [19].

Exercises

Exercise 2.1. Write out the lens of a layer ℓ(W,b;x)=σ(Wx+b)\ell(W,b;x)=\sigma(Wx+b), and compose two of them to obtain the gradient of 12(aTσ(Wx+b)+c−y)2\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2 with respect to (W,b,a,c)(W,b,a,c).

Exercise 2.2. Show that rkK(θ)=rkJ(θ)\mathop{\mathrm{rk}}K(\theta)=\mathop{\mathrm{rk}}J(\theta) and that, for nn data points and scalar output, K(θ)K(\theta) is invertible if and only if the functions ∂F/∂θk(θ,⋅)\partial F/\partial\theta_k(\theta,\cdot), k=1,…,Pk=1,\dots,P, restricted to the data, span Rn\mathbb{R}^n.

Exercise 2.3. Derive the formula for ∇2(λ∘ev)\nabla^2(\lambda\circ\mathrm{ev}) from the composition of 22-jets.

Exercise 2.4. For the linear family F(θ,x)=⟨θ,ϕ(x)⟩F(\theta,x)=\langle\theta,\phi(x)\rangle with a fixed feature map ϕ ⁣:X→RP\phi\colon X\to\mathbb{R}^P, compute JJ, KK, the fibres of ev\mathrm{ev}, and the functional dimension.

Symmetries and the Moduli of Networks

The realization map θ↦fθ\theta\mapsto f_\theta is never injective. Its fibres are the first thing to understand about an architecture, since every function on parameter space that is defined in terms of the network’s input–output behaviour, such as the loss, is constant on them. This chapter describes the fibres for networks with one hidden layer, where the theory is complete, and draws the consequences that hold in general.

Symmetries

Definition 3.1. A symmetry of a family F ⁣:Θ×X→YF\colon\Theta\times X\to Y is a diffeomorphism gg of Θ\Theta with F(gθ,x)=F(θ,x)F(g\theta,x)=F(\theta,x) for all θ,x\theta,x: an invertible 22-cell of (Θ,F)(\Theta,F) in Para\mathsf{Para}.

For a hidden unit jj in layer ii of a feedforward architecture, call its incoming parameters the jj-th row of WiW_i and the jj-th entry of bib_i, and its outgoing parameters the jj-th column of Wi+1W_{i+1}.

Proposition 3.2. [classical; ] Every feedforward architecture has the following symmetries.

  1. Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.

  2. If σ\sigma is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.

  3. If σ\sigma is positively homogeneous of degree one, as ReLU\mathop{\mathrm{ReLU}} is: for each hidden unit and λ>0\lambda>0, multiplication of the incoming parameters by λ\lambda and of the outgoing ones by λ−1\lambda^{-1}.

For σ=tanh⁡\sigma=\tanh these generate a finite group GG of order ∏i=1L−12nini!\prod_{i=1}^{L-1}2^{n_i}n_i!.

One hidden layer

Consider F(θ,x)=∑j=1Haj σ(⟨wj,x⟩+bj)+cF(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c on X=RdX=\mathbb{R}^d, with θ=(wj,bj,aj)j≤H\theta=(w_j,b_j,a_j)_{j\le H} and cc, so P=H(d+2)+1P=H(d+2)+1, and σ=tanh⁡\sigma=\tanh.

Definition 3.3. The parameter θ\theta is irreducible if every aj≠0a_j\ne0, every wj≠0w_j\ne0, and (wj,bj)≠±(wk,bk)(w_j,b_j)\ne\pm(w_k,b_k) for j≠kj\ne k. The set of irreducible parameters is Θirr\Theta^{\mathrm{irr}}.

Theorem 3.4 (Identifiability). [literature; Sussmann [30]; Fukumizu [16]] Let θ\theta be irreducible. Then fθ′=fθf_{\theta'}=f_\theta for a parameter θ′\theta' with the same number of hidden units if and only if θ′=gθ\theta'=g\theta for some g∈Gg\in G. Moreover θ\theta is irreducible if and only if the functions ∂F/∂θk(θ,⋅)\partial F/\partial\theta_k(\theta,\cdot), k=1,…,Pk=1,\dots,P, are linearly independent; so the functional dimension is PP exactly on Θirr\Theta^{\mathrm{irr}}.

For deep networks with tanh⁡\tanh the corresponding statement holds for generic parameters, by a theorem of Fefferman [13]; for ReLU\mathop{\mathrm{ReLU}} networks see [28]. The complement of Θirr\Theta^{\mathrm{irr}} is a finite union of proper linear subspaces, so Θirr\Theta^{\mathrm{irr}} is open and dense, and GG acts freely on it.

Proposition 3.5 (The moduli of irreducible networks). [observation] The quotient Θirr/G\Theta^{\mathrm{irr}}/G is a manifold of dimension PP on which the realization map is injective. Let Z={(w,b,a):w≠0, a>0}Z=\{(w,b,a):w\ne0,\ a>0\} and Z′={(w,b):w≠0}/±Z'=\{(w,b):w\ne0\}/\pm. Then Θirr/G ≅ {sets of H points of Z with pairwise distinct images in Z′}×R.\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in } Z'\bigr\}\times\mathbb{R}. If d≥2d\ge2 it is connected and its fundamental group maps onto the symmetric group SHS_H.

Proof. A free action of a finite group has a manifold quotient. Using the sign change to make each aja_j positive identifies Θirr/(Z/2)H\Theta^{\mathrm{irr}}/(\mathbb{Z}/2)^H with the ordered HH-tuples in ZZ having distinct images in Z′Z', times the line of the output bias; dividing by SHS_H forgets the order. For d≥2d\ge2, ZZ is connected, and the tuples with a coincidence in Z′Z' form a closed subset of codimension d+1≥2d+1\ge2 in ZHZ^H, so the space of ordered tuples is connected. A connected covering with group SHS_H gives a surjection from the fundamental group of the base. ◻

So there are loops of networks, along which the function computed changes continuously and returns to itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged roles. An explicit one moves (w1,b1,a1)(w_1,b_1,a_1) to (w2,b2,a2)(w_2,b_2,a_2) and back along two different arcs. The space of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.

Proposition 3.6 (Fibres at reducible parameters). [classical; cf. [30], [16], [29]] At a reducible parameter the fibre of the realization map has positive dimension:

  1. if aj=0a_j=0, the fibre contains the affine space of all values of (wj,bj)(w_j,b_j), of dimension d+1d+1;

  2. if (wj,bj)=(wk,bk)(w_j,b_j)=(w_k,b_k), it contains the line aj+ak=consta_j+a_k=\mathrm{const};

  3. if wj=0w_j=0, it contains the surface ajσ(bj)+c=consta_j\sigma(b_j)+c=\mathrm{const} in the coordinates (aj,bj,c)(a_j,b_j,c).

Each case is a way for a network with HH hidden units to compute a function that needs fewer. The set of parameters in ΘH\Theta_H computing a given function of an irreducible network with H′<HH'<H units is a union of such families, permuted by GG; it is the subject of a detailed combinatorial study in [29]. At these parameters the rank of JJ drops, by 3.4. The image F\mathcal F of the realization map is therefore a PP-dimensional manifold Θirr/G\Theta^{\mathrm{irr}}/G together with a singular set, of lower dimension, onto which the reducible strata of Θ\Theta are collapsed.

Homogeneous activations

Proposition 3.7. [literature; cf. [19]] Let σ\sigma be positively homogeneous of degree one. For each hidden unit jj let ξj\xi_j be the vector field on Θ\Theta equal to the incoming parameters of jj in the incoming coordinates, minus the outgoing parameters in the outgoing coordinates, and zero elsewhere. Then DθF(θ,x) ξj(θ)=0D_\theta F(\theta,x)\,\xi_j(\theta)=0 wherever FF is differentiable. Consequently the functional dimension of the architecture is at most P−HP-H at every parameter.

Proof. ξj\xi_j is the generator of the one-parameter group of symmetries of 3.2(iii); differentiate F(gλθ,x)=F(θ,x)F(g_\lambda\theta,x)=F(\theta,x) at λ=1\lambda=1. The ξj\xi_j for different units have disjoint supports in the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and the bound holds on the closure of that set by semicontinuity of rank. ◻

The script confirms both statements on small networks: for tanh⁡\tanh the Jacobian has full rank PP at a random parameter and loses rank at reducible ones; for ReLU\mathop{\mathrm{ReLU}} its rank never exceeds P−HP-H and the fields ξj\xi_j lie in its kernel.

For ReLU\mathop{\mathrm{ReLU}} the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space: the orbit of a unit with zero outgoing weight is not closed, since (λw,λb,0)→(0,0,0)(\lambda w,\lambda b,0)\to(0,0,0) as λ→0\lambda\to0. The correct object is the quotient stack, in which such points are remembered with their stabilizers. This is the same reason that vector bundles in [9, Ch. 5] formed a stack and not a space, and the loss is a function on it.

Invariant functions

Proposition 3.8. [classical; ] Let L ⁣:Θ→R\mathcal{L}\colon\Theta\to\mathbb{R} be invariant under a group GG of symmetries. Then the critical set of L\mathcal{L} is a union of GG-orbits. If GG contains a one-parameter group with generator ξ\xi, then dL(ξ)=0d\mathcal{L}(\xi)=0 identically and, at every critical point, ξ\xi lies in the kernel of the Hessian. If moreover ∇λ\nabla\lambda vanishes at ev(θ)\mathrm{ev}(\theta) for a loss L=λ∘ev\mathcal{L}=\lambda\circ\mathrm{ev}, then the whole fibre of ev\mathrm{ev} through θ\theta is critical.

Proof. Differentiating L(gsθ)=L(θ)\mathcal{L}(g_s\theta)=\mathcal{L}(\theta) in ss gives dLθ(ξ)=0d\mathcal{L}_\theta(\xi)=0 for all θ\theta; differentiating that identity in a direction vv at a critical point gives ∇2L(ξ,v)+dL(Dvξ)=∇2L(ξ,v)=0\nabla^2\mathcal{L}(\xi,v)+d\mathcal{L}(D_v\xi)=\nabla^2\mathcal{L}(\xi,v)=0. The last statement is ∇L=JT∇λ\nabla\mathcal{L}=J^{\mathsf T}\nabla\lambda. ◻

Corollary 3.9. [observation] For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the form λ∘ev\lambda\circ\mathrm{ev} is a Morse function on any open set containing a critical point at which some hidden unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least the number of such units.

So Morse theory in the form of [9, Ch. 8] never applies directly to the loss of a ReLU\mathop{\mathrm{ReLU}} network. It can apply to the function induced on a slice transverse to the orbits, or in Morse–Bott form, with critical manifolds in place of critical points. For tanh⁡\tanh networks there is no continuous symmetry and no such obstruction on Θirr\Theta^{\mathrm{irr}}; the degeneracies there come from the reducible strata and from overparametrization (3).

Notes

Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann and by Albertini and Sontag, and for deep tanh⁡\tanh networks by Fefferman. The relation between reducibility and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of Watanabe’s singular learning theory [31]. The use of permutation symmetry to compare trained networks is discussed in 3.

Exercises

Exercise 3.1. Verify the three symmetries of 3.2 for a network with one hidden layer.

Exercise 3.2. For the one-hidden-layer tanh⁡\tanh network, show that if aj=0a_j=0 then rkJ(θ)≤P−(d+1)\mathop{\mathrm{rk}}J(\theta)\le P-(d+1) for every set of inputs, and if (wj,bj)=(wk,bk)(w_j,b_j)=(w_k,b_k) then rkJ(θ)≤P−1\mathop{\mathrm{rk}}J(\theta)\le P-1.

Exercise 3.3. For F(θ,x)=a ReLU(wx+b)+cF(\theta,x)=a\,\mathop{\mathrm{ReLU}}(wx+b)+c with d=1d=1, write down ξ\xi and verify DθF⋅ξ=0D_\theta F\cdot\xi=0 by hand. What is the functional dimension?

Exercise 3.4. Compute the order of GG for the tanh⁡\tanh architectures with widths (2,3,1)(2,3,1) and (2,4,4,1)(2,4,4,1).

Level Sets

Transversality and Decision Boundaries

A binary classifier is a function fθ ⁣:X→Rf_\theta\colon X\to\mathbb{R} whose sign is the predicted class, and its decision boundary is Bθ=fθ−1(0)B_\theta=f_\theta^{-1}(0). This chapter answers question (Q1) of 1 for decision boundaries: for almost every parameter the boundary is a smooth hypersurface; its isotopy type is constant on the components of the complement of a closed null set of parameters; and every isotopy type of compact hypersurface occurs for wide enough networks.

The boundary is generically a hypersurface

Theorem 1.1 (Generic decision boundaries). [reformulation; cf. [9, Thm. 4.1, Thm. 1.12]] Let F ⁣:Θ×X→RF\colon\Theta\times X\to\mathbb{R} be a family of the form F((θ′,c),x)=G(θ′,x)+cF((\theta',c),x)=G(\theta',x)+c.

  1. For every θ′\theta', the set of cc for which 00 fails to be a regular value of f(θ′,c)f_{(\theta',c)} has measure zero.

  2. For almost every θ\theta, the decision boundary BθB_\theta is a smooth hypersurface in XX, closed as a subset and possibly empty, with trivial normal bundle oriented by ∇fθ\nabla f_\theta; and X∖BθX\setminus B_\theta is the disjoint union of the open sets {fθ>0}\{f_\theta>0\} and {fθ<0}\{f_\theta<0\}.

  3. If XX is a compact manifold with boundary, then for almost every θ\theta the boundary BθB_\theta is moreover a neat submanifold: it meets ∂X\partial X transversally, in ∂Bθ\partial B_\theta.

Proof. (a) 00 is a regular value of G(θ′,⋅)+cG(\theta',\cdot)+c exactly when −c-c is a regular value of G(θ′,⋅)G(\theta',\cdot), and the critical values of a smooth function have measure zero by [9, Thm. 4.1]. (b) follows from (a) by Fubini’s theorem and [9, Thm. 1.12]. (c) Apply (a) also to the restriction to ∂X\partial X and use [9, Thm. 1.17]. ◻

Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever smooth activation it uses. Part (a) is stronger than the conclusion of 1.5, since it holds for each value of the other parameters separately; it says that a non-smooth decision boundary can always be repaired by an arbitrarily small change of the output bias alone. The level sets fθ−1(s)f_\theta^{-1}(s) for other thresholds ss are covered by the same statement.

Example 1.2. Let f(x)=x12−x22+cf(x)=x_1^2-x_2^2+c. For c≠0c\ne0 the boundary is a hyperbola, a 11-manifold with two components. For c=0c=0 it is a pair of crossing lines, not a manifold. In the language of [9, Ch. 4], the boundary at c=0c=0 is a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.

Several classes

A classifier with kk classes is a map fθ ⁣:X→Rkf_\theta\colon X\to\mathbb{R}^k, and the predicted class is the index of the largest coordinate. For a non-empty I⊂{1,…,k}I\subset\{1,\dots,k\} let AI⊂RkA_I\subset\mathbb{R}^k be the set of points whose largest coordinate is attained exactly on II. Then AIA_I is an open subset of the linear subspace {yi=yj:i,j∈I}\{y_i=y_j:i,j\in I\}, a submanifold of codimension ∣I∣−1|I|-1, and the AIA_I partition Rk\mathbb{R}^k.

Proposition 1.3. [reformulation; cf. [9, Thm. 4.5]] Let F ⁣:Θ×X→RkF\colon\Theta\times X\to\mathbb{R}^k be a family whose last layer is affine with a bias. For almost every θ\theta, fθf_\theta is transverse to every AIA_I. Then the decision region of class ii is the open set fθ−1(A{i})f_\theta^{-1}(A_{\{i\}}); the set where exactly the classes in II tie is a submanifold of codimension ∣I∣−1|I|-1; and no more than d+1d+1 classes tie anywhere.

Proof. FF is a submersion, so [9, Thm. 4.5] applies to each of the finitely many AIA_I, and a finite intersection of sets of full measure has full measure. A submanifold of codimension greater than dd is empty. ◻

In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time at isolated points, like the cells of a foam, and four regions never meet at a point.

Chambers

Let XX be compact, and for simplicity without boundary. Define Σ={(θ,x):F(θ,x)=0, dxF(θ,x)=0}⊂Θ×X,Δ=π(Σ)⊂Θ,\Sigma=\{(\theta,x):F(\theta,x)=0,\ d_xF(\theta,x)=0\}\subset\Theta\times X, \qquad\Delta=\pi(\Sigma)\subset\Theta , the set of parameters for which 00 is not a regular value: the discriminant of the family.

Theorem 1.4 (Chambers). [reformulation; cf. [9, Lem. 4.4, Thm. 8.5, Thm. 10.3]] Let FF be as in 1.1 with XX compact. Then Δ\Delta is closed and of measure zero. If γ\gamma is a path in Θ∖Δ\Theta\setminus\Delta, there is a diffeotopy φt\varphi_t of XX with φt(Bγ(0))=Bγ(t)\varphi_t(B_{\gamma(0)})=B_{\gamma(t)}. In particular the diffeomorphism type of the pair (X,Bθ)(X,B_\theta) is constant on each connected component of Θ∖Δ\Theta\setminus\Delta.

Proof. Σ\Sigma is closed and π∣Σ\pi|_\Sigma is proper because XX is compact, so Δ\Delta is closed; its complement has full measure by 1.1. Over Θ∖Δ\Theta\setminus\Delta the set W=F−1(0)W=F^{-1}(0) is a submanifold of Θ×X\Theta\times X, since dxF≠0d_xF\ne0 there, and by [9, Lem. 4.4] the projection W→Θ∖ΔW\to\Theta\setminus\Delta is a submersion; it is proper. By Ehresmann’s theorem [9, Thm. 8.5] it is a fibre bundle. Pulled back along γ\gamma it is trivial, which gives an isotopy of embeddings Bγ(0)→XB_{\gamma(0)}\to X with image Bγ(t)B_{\gamma(t)} at time tt, and the isotopy extends to a diffeotopy of XX by [9, Thm. 10.3]. ◻

The components of Θ∖Δ\Theta\setminus\Delta are the chambers of the architecture. The theorem says that a classifier has a well-defined topology (the number of components of its decision boundary and of its decision regions, their genera, how they are nested) which does not change under small changes of the parameters, and can change only when the parameter crosses Δ\Delta. What happens at a crossing is the subject of the next chapter. The symmetry group of 3 permutes the chambers.

For manifolds with boundary the same holds with Δ\Delta enlarged by the parameters for which BθB_\theta is tangent to ∂X\partial X.

Proposition 1.5 (Walls). [observation] Consider the one-hidden-layer architecture F(θ,x)=∑j≤Hajσ(⟨wj,x⟩+bj)+cF(\theta,x)=\sum_{j\le H}a_j\sigma(\langle w_j,x\rangle+b_j)+c on Rd\mathbb{R}^d with σ′>0\sigma'>0. At every point where the matrix WW with rows wjw_j has rank dd, the map Φ(θ,x)=(F,dxF)∈R×Rd\Phi(\theta,x)=(F,d_xF)\in\mathbb{R}\times\mathbb{R}^d is a submersion. Hence, over the open set of such parameters, Σ\Sigma is a submanifold of dimension P−1P-1, and Δ\Delta is the image of a manifold of dimension P−1P-1.

Proof. dxF=WT(a⊙σ′(z))d_xF=W^{\mathsf T}(a\odot\sigma'(z)) with z=Wx+bz=Wx+b. A variation δa\delta a changes dxFd_xF by WT(δa⊙σ′(z))W^{\mathsf T}(\delta a\odot\sigma'(z)), and δa↦δa⊙σ′(z)\delta a\mapsto\delta a\odot\sigma'(z) is a linear isomorphism since σ′>0\sigma'>0; so if WTW^{\mathsf T} is onto, every variation of dxFd_xF is produced. The resulting change in FF is compensated by δc\delta c. ◻

So the discriminant is no larger than a hypersurface, and the picture of parameter space is that of open chambers separated by walls of codimension one. The proposition is stated for the simplest architecture; the argument needs only that the parameters can move the value and the input-gradient of the network independently at each point, which holds for deeper networks at parameters where the layer Jacobians have full rank.

Every hypersurface is a decision boundary

Theorem 1.6 (Realization). [reformulation; cf. [9, Thm. 6.2]; Hornik [21]] Let K⊂RdK\subset\mathbb{R}^d be a compact domain with smooth boundary and B⊂int⁡KB\subset\operatorname{int}K a closed hypersurface. Let σ\sigma be smooth, bounded and non-constant. Then for all sufficiently large HH there is a non-empty open set of parameters θ\theta of the one-hidden-layer architecture with HH units for which Bθ∩KB_\theta\cap K is carried to BB by a diffeomorphism of KK isotopic to the identity.

Proof. A closed hypersurface in Rd\mathbb{R}^d separates and so has a trivial normal bundle; using a tubular neighbourhood [9, Thm. 6.2] one constructs a smooth g ⁣:K→Rg\colon K\to\mathbb{R} with g−1(0)=Bg^{-1}(0)=B, 00 a regular value, and g≠0g\ne0 on ∂K\partial K. Since BB is compact, there is ε>0\varepsilon>0 such that every hh with ∥h−g∥C1(K)<ε\|h-g\|_{C^1(K)}<\varepsilon has 00 as a regular value and does not vanish on ∂K\partial K. By Hornik’s theorem, networks with one hidden layer are dense in C1(K)C^1(K), so some fθ0f_{\theta_0} is within ε\varepsilon of gg, and so is fθf_\theta for θ\theta near θ0\theta_0, the realization map being continuous into C1(K)C^1(K). The family (1−s)g+sfθ(1-s)g+sf_\theta, s∈[0,1]s\in[0,1], stays within ε\varepsilon of gg, and the argument of 1.4 applied to it gives the diffeotopy. ◻

So there is no topological restriction on decision boundaries beyond those satisfied by every compact hypersurface in Rd\mathbb{R}^d: it is orientable, it separates, and it bounds (2.6). The theorem is silent on how large HH must be. Upper bounds for the Betti numbers of decision regions in terms of the architecture, which give lower bounds for HH, are due to Bianchini and Scarselli [2].

Piecewise-linear networks

For σ=ReLU\sigma=\mathop{\mathrm{ReLU}} a network is piecewise linear, its decision boundary is a polyhedral complex, and the smooth theory does not apply as it stands. The counterpart of 1.1 is due to Grigsby and Lindsey [18]. They define what it means for a ReLU\mathop{\mathrm{ReLU}} network to be generic and transversal, in terms of the bent hyperplane arrangement formed by the zero sets of its units, and prove that almost all ReLU\mathop{\mathrm{ReLU}} networks have both properties. As an application they show that for the architecture with one hidden layer of d+1d+1 units, a decision region has at most one bounded connected component: a restriction on topology coming from the architecture, of the kind 1.6 rules out for unbounded width.

Notes

That decision boundaries are generically manifolds is folklore, and is usually justified by Sard’s theorem in the form (a). The statement about chambers is the standard structure theory of discriminants in singularity theory, applied to this family. For topological complexity of decision regions see [2], and for an empirical study of how networks change the topology of data, [27].

Exercises

Exercise 1.1. Let d=1d=1 and X=[−R,R]X=[-R,R]. Show that for almost every θ\theta the decision boundary is a finite set at which fθf_\theta changes sign, and that the parity of its cardinality is determined by the signs of fθ(±R)f_\theta(\pm R).

Exercise 1.2. Let XX be a closed disc in the plane. Describe the generic decision boundary up to diffeomorphism.

Exercise 1.3. For the family F(c,x)=x12−x22+cF(c,x)=x_1^2-x_2^2+c on the plane, determine Σ\Sigma and Δ\Delta, and describe the chambers.

Exercise 1.4. For a three-class classifier in the plane satisfying the conclusion of 1.3, show that near a point where all three classes tie the decision regions look like three sectors meeting at a point.

Topology Change along Paths: Surgery and Cobordism

By 1.4 the topology of a decision boundary can change during training only when the parameter crosses the discriminant. This chapter answers question (Q2): for a generic path the crossings are isolated, and at each of them the boundary changes by one surgery, determined by the index of a critical point. The union of the boundaries along the path is a cobordism, and the surgeries are the critical points of its height function. This is the Morse theory of [9, Ch. 8] applied to a manifold that the path itself creates.

The trace of a path

Let F ⁣:Θ×X→RF\colon\Theta\times X\to\mathbb{R} be a family as in 1.1, with XX compact of dimension dd, and let γ ⁣:[0,1]→Θ\gamma\colon[0,1]\to\Theta be a path. Write ft=fγ(t)f_t=f_{\gamma(t)} and Bt=ft−1(0)B_t=f_t^{-1}(0), and define Φγ ⁣:[0,1]×X→R,Φγ(t,x)=F(γ(t),x);Wγ=Φγ−1(0)⊂[0,1]×X.\Phi_\gamma\colon[0,1]\times X\to\mathbb{R},\quad\Phi_\gamma(t,x)=F(\gamma(t),x);\qquad W_\gamma=\Phi_\gamma^{-1}(0)\subset[0,1]\times X .

Theorem 2.1 (The trace is a cobordism). [reformulation; cf. [9, Thm. 1.17, Thm. 4.5]] If 00 is a regular value of Φγ\Phi_\gamma and of f0f_0 and f1f_1, then WγW_\gamma is a compact neat hypersurface in [0,1]×X[0,1]\times X with ∂Wγ=B0⊔B1\partial W_\gamma=B_0\sqcup B_1: a cobordism from B0B_0 to B1B_1 embedded in [0,1]×X[0,1]\times X. For a path with endpoints outside Δ\Delta, this holds after an arbitrarily small perturbation of the bias component of γ\gamma fixed at the endpoints.

Proof. The first statement is [9, Thm. 1.17]. For the second, replace the bias component c(t)c(t) by c(t)+s β(t)c(t)+s\,\beta(t) with β>0\beta>0 on (0,1)(0,1) and vanishing with all derivatives at the ends. The map (s,t,x)↦Φγ(t,x)+sβ(t)(s,t,x)\mapsto\Phi_\gamma(t,x)+s\beta(t) is a submersion on {0<t<1}\{0<t<1\}, so by [9, Thm. 4.5] zero is a regular value for almost every ss; near t=0,1t=0,1 it is one already because f0,f1f_0,f_1 have 00 as a regular value. ◻

The times at which γ\gamma meets Δ\Delta are, by [9, Lem. 4.4], exactly the critical values of the height function τ ⁣:Wγ→[0,1]\tau\colon W_\gamma\to[0,1], τ(t,x)=t\tau(t,x)=t. At such a point (t0,x0)(t_0,x_0) we have dxft0(x0)=0d_xf_{t_0}(x_0)=0: x0x_0 is a critical point of ft0f_{t_0} with critical value 00.

Surgery

Theorem 2.2 (One surgery at a time). [reformulation; cf. [9, Thm. 8.6, Cor. 8.7]] Let γ\gamma be a path for which WγW_\gamma is a manifold and τ\tau is a Morse function with distinct critical values t1<⋯<trt_1<\dots<t_r in (0,1)(0,1). Then:

  1. On each interval of [0,1]∖{t1,…,tr}[0,1]\setminus\{t_1,\dots,t_r\} the boundary BtB_t moves by an ambient isotopy of XX.

  2. At each tit_i there is exactly one point xi∈Btix_i\in B_{t_i} at which BtiB_{t_i} fails to be a manifold. It is a non-degenerate critical point of ftif_{t_i} with ∂tΦγ(ti,xi)≠0\partial_t\Phi_\gamma(t_i,x_i)\ne0, and near (ti,xi)(t_i,x_i) there are coordinates on XX in which Φγ(t,x)=±(t−ti)−x12−⋯−xk2+xk+12+⋯+xd2,\Phi_\gamma(t,x)=\pm(t-t_i)-x_1^2-\dots-x_k^2+x_{k+1}^2+\dots+x_d^2 , with kk the index of xix_i as a critical point of ftif_{t_i}.

  3. Bti+εB_{t_i+\varepsilon} is obtained from Bti−εB_{t_i-\varepsilon} by a surgery: a copy of Sλ−1×Dd−λS^{\lambda-1}\times D^{d-\lambda} is removed and Dλ×Sd−λ−1D^\lambda\times S^{d-\lambda-1} is glued in, where λ\lambda is the index of τ\tau at (ti,xi)(t_i,x_i); λ=k\lambda=k or d−kd-k according to the sign in (b). Correspondingly the region {ft≤0}\{f_t\le0\} gains or loses a handle of index kk.

Proof. (a) is 1.4. For (b): at a critical point of τ\tau on WγW_\gamma the tangent space of WγW_\gamma is {dt=0}\{dt=0\}, so dxΦγ=0d_x\Phi_\gamma=0 and, WγW_\gamma being a manifold, ∂tΦγ≠0\partial_t\Phi_\gamma\ne0. Solving Φγ=0\Phi_\gamma=0 for tt as a function of xx near the point exhibits WγW_\gamma as a graph over XX on which τ\tau is the function t(x)t(x), whose Hessian at xix_i is −∇2fti(xi)/∂tΦγ-\nabla^2f_{t_i}(x_i)/\partial_t\Phi_\gamma. So τ\tau is non-degenerate there exactly when xix_i is a non-degenerate critical point of ftif_{t_i}, and the normal form follows from the Morse lemma with parameters. (c) is [9, Thm. 8.6] for τ\tau on WγW_\gamma: the level sets of τ\tau are the BtB_t, and passing a critical point of index λ\lambda changes the level set by the stated surgery. ◻

In words: the topology of a decision boundary changes exactly when a critical point of the network function crosses the threshold, and the change is read off from the index of the critical point and the direction of crossing. lists the cases for inputs of dimension two and three.

Elementary changes of topology of a decision region, by the index of the critical point crossing zero from above. Crossing in the other direction reverses each.
dd index kk of xix_i effect on the region {ft≤0}\{f_t\le0\} when the critical value becomes negative
22 00 (minimum) a new disc appears; the boundary gains a circle
22 11 (saddle) two parts of the region join, or one part meets itself; two boundary arcs reconnect, so the number of boundary circles changes by ±1\pm1
22 22 (maximum) a hole in the region is filled; a boundary circle disappears
33 00 a new ball appears; the boundary gains a sphere
33 11 a tube joins two parts of the region, or adds a handle to one
33 22 a tunnel through the region is closed by a disc, or a cavity is split in two
33 33 a cavity is filled; a boundary sphere disappears

Proposition 2.3. [observation] In the situation of 2.2(c), χ(Bti+ε)−χ(Bti−ε)\chi(B_{t_i+\varepsilon})-\chi(B_{t_i-\varepsilon}) equals 2(−1)λ2(-1)^\lambda if dd is odd and 00 if dd is even, and the Euler characteristic of the region {ft≤0}\{f_t\le0\} changes by ±(−1)k\pm(-1)^k.

Proof. The change is χ(Dλ×Sd−λ−1)−χ(Sλ−1×Dd−λ)=χ(Sd−λ−1)−χ(Sλ−1)=(−1)d−λ−1−(−1)λ−1\chi(D^\lambda\times S^{d-\lambda-1})-\chi(S^{\lambda-1}\times D^{d-\lambda}) =\chi(S^{d-\lambda-1})-\chi(S^{\lambda-1})=(-1)^{d-\lambda-1}-(-1)^{\lambda-1}. The second statement is additivity of χ\chi under attaching a kk-cell. ◻

Example 2.4. Let F(t,x)=(x12−1)2+x22−tF(t,x)=(x_1^2-1)^2+x_2^2-t on the plane. The function (x12−1)2+x22(x_1^2-1)^2+x_2^2 has minima at (±1,0)(\pm1,0) with value 00 and a saddle at the origin with value 11. For t<0t<0 the region {F≤0}\{F\le0\} is empty. At t=0t=0 two discs are born (two critical points cross at the same time, which is not generic; a small asymmetry separates them). At t=1t=1 the saddle crosses and the two discs merge. The numbers of components are 00, 22, 11, and the Euler characteristics of the region 00, 22, 11. The script counts components on a grid.

Genericity of paths

Are the hypotheses of 2.2 generic? For paths in the space of all functions, yes: this is the content of [9, Thm. 8.12]. For paths in a given architecture one needs enough parameters to move critical points and critical values independently.

Proposition 2.5. [reformulation; cf. [9, Thm. 4.6, Thm. 8.12]] For the architecture of 1.5, with H≥dH\ge d, let γ\gamma be a path with endpoints in Θ∖Δ\Theta\setminus\Delta along which WW has rank dd. Then arbitrarily close to γ\gamma, with the same endpoints, there is a path satisfying the hypotheses of 2.2.

Sketch. By 1.5 the map (θ,x)↦(F,dxF)(\theta,x)\mapsto(F,d_xF) is a submersion along the path, so the family of perturbations of γ\gamma by compactly supported variations of (a,c)(a,c) induces a family of maps (t,x)↦(Φ,dxΦ)(t,x)\mapsto(\Phi,d_x\Phi) transverse to the zero section; by [9, Thm. 4.5] a generic member has Σ\Sigma pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at those points, and distinctness of the times, are further conditions of positive codimension on the 22-jet, which a variation of WW and bb moves freely; [9, Thm. 4.6] applies. ◻

A training path produced by gradient descent is one particular path, and nothing guarantees that it is generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the typical exception (2.4): a symmetric network crossing a wall has several critical points crossing together.

What cobordism does and does not see

Proposition 2.6. [classical; ] Let d≥2d\ge2 and let B=f−1(0)⊂RdB=f^{-1}(0)\subset\mathbb{R}^d be compact with 00 a regular value. Then BB is the boundary of a compact region of Rd\mathbb{R}^d; it is orientable, its normal bundle is trivial, and TB⊕RTB\oplus\mathbb{R} is trivial. In particular BB is null-cobordant, all its Stiefel–Whitney numbers vanish, and no closed non-orientable surface is a decision boundary in R3\mathbb{R}^3.

Proof. Choose a closed ball DD containing BB in its interior. Its boundary sphere is connected and ff does not vanish on it, so ff has a constant sign there, say positive. Then {f≤0}∩D\{f\le0\}\cap D is a compact manifold with boundary BB. The normal bundle is trivialized by ∇f\nabla f, and TB⊕ν=TRd∣BTB\oplus\nu=T\mathbb{R}^d|_B is trivial. ◻

So the cobordism class of a decision boundary carries no information: it is always zero, as [9, Thm. 4.16] predicts for a fibre of a map to a contractible space. What carries information is finer: the diffeomorphism type of the pair (X,Bt)(X,B_t) in each chamber, and the sequence of surgeries along a path.

That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between generic paths meets a short list of codimension-two events.

Proposition 2.7 (Moves). [reformulation; cf. Cerf; [9, Thm. 8.12]] Let γ0,γ1\gamma_0,\gamma_1 be generic paths with the same endpoints in an architecture rich enough for jet transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or deleting a pair of consecutive surgeries of indices λ,λ+1\lambda,\lambda+1 that cancel (a birth–death point of ftf_t on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that touches zero and returns).

The proposition is the two-parameter analogue of 2.2, and is the form Cerf’s theorem takes for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and that anything invariant under them, for instance the parity of the number of surgeries of each index modulo the cancellation of adjacent pairs, depends only on the endpoints.

Notes

The description of how level sets change in one-parameter families is classical Morse and Cerf theory; see Milnor’s lectures on the h-cobordism theorem. Its use for decision boundaries during training appears in empirical form in studies that track Betti numbers of decision regions and of data representations, such as [27]. Persistent homology of the family {ft≤s}\{f_t\le s\} in the two parameters (t,s)(t,s) is the natural computational tool, and the events of 2.7 are the generic singularities of its bifiltration.

Exercises

Exercise 2.1. Verify 2.4: find the critical points and indices of (x12−1)2+x22(x_1^2-1)^2+x_2^2 and the topology of the sublevel sets.

Exercise 2.2. For d=1d=1, show that along a generic path the points of the decision boundary are born and die in pairs, and that the parity of their number is constant if XX is a circle.

Exercise 2.3. In d=3d=3, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries, and what are their indices?

Exercise 2.4. Prove the formula for the Hessian of τ\tau used in the proof of 2.2(b).

The Loss Landscape

Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the landscape is organized by the sublevel sets of the loss, to which the Morse theory of [9, Ch. 8] applies with two caveats established in 3 and below: the loss is rarely a Morse function, and parameter space is not compact.

The interpolation manifold

Let ev ⁣:Θ→RN\mathrm{ev}\colon\Theta\to\mathbb{R}^N be the evaluation map on nn data points, N=nmN=nm, and L=12∣ev−y∣2\mathcal{L}=\tfrac12|\mathrm{ev}-y|^2. Write My=ev−1(y)=L−1(0)M_y=\mathrm{ev}^{-1}(y)=\mathcal{L}^{-1}(0).

Theorem 3.1 (Interpolating parameters). [reformulation; cf. [9, Thm. 1.12, Thm. 4.1]; Cooper [6]]

  1. For almost every y∈RNy\in\mathbb{R}^N, MyM_y is a submanifold of Θ\Theta of dimension P−NP-N, possibly empty. If P<NP<N it is empty for almost every yy.

  2. If yy is a regular value and θ∈My\theta\in M_y, then TθMy=ker⁡J(θ)T_\theta M_y=\ker J(\theta) and ∇2L(θ)=JTJ\nabla^2\mathcal{L}(\theta)=J^{\mathsf T}J has rank NN: it vanishes on TθMyT_\theta M_y and is positive definite on its orthogonal complement.

  3. MyM_y is non-empty for yy in the image of ev\mathrm{ev}, and that image has non-empty interior if and only if JJ has rank NN somewhere.

Proof. (a) By [9, Thm. 4.1] almost every yy is a regular value of ev\mathrm{ev}, and then [9, Thm. 1.12] applies; if P<NP<N every point of the image is a critical value. (b) TθMy=ker⁡DevθT_\theta M_y=\ker D\mathrm{ev}_\theta by [9, Thm. 1.12], and the Hessian is given by 2.6. (c) If JJ has rank NN at θ\theta then ev\mathrm{ev} is open near θ\theta; conversely if the rank is everywhere less than NN the image has measure zero. ◻

Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in addition that for sufficiently wide networks MyM_y is non-empty for generic data. In the overparametrized regime P>NP>N, then, the global minima of the loss are not isolated: they form a manifold whose dimension is the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it. In the terminology of Morse theory the loss is a Morse–Bott function near MyM_y. By 2.6 its normal curvature is the spectrum of the tangent kernel.

Proposition 3.2 (Convergence near the interpolation manifold). [literature; Liu–Zhu–Belkin [25]] Let U⊂ΘU\subset\Theta be an open set on which the smallest eigenvalue of the tangent kernel K=JJTK=JJ^{\mathsf T} is at least μ>0\mu>0. Then ∣∇L∣2≥2μ L|\nabla\mathcal{L}|^2\ge2\mu\,\mathcal{L} on UU. A gradient flow line starting at θ0\theta_0 satisfies L(γ(t))≤L(θ0)e−2μt\mathcal{L}(\gamma(t))\le\mathcal{L}(\theta_0)e^{-2\mu t} and has length at most 2L(θ0)/μ\sqrt{2\mathcal{L}(\theta_0)/\mu} for as long as it stays in UU; if the ball of that radius about θ0\theta_0 lies in UU, the flow line converges to a point of MyM_y.

Proof. ∣∇L∣2=(ev−y)TJJT(ev−y)≥μ∣ev−y∣2=2μL|\nabla\mathcal{L}|^2=(\mathrm{ev}-y)^{\mathsf T}JJ^{\mathsf T}(\mathrm{ev}-y)\ge\mu|\mathrm{ev}-y|^2=2\mu\mathcal{L}. Then ddtL=−∣∇L∣2≤−2μL\frac{d}{dt}\mathcal{L}=-|\nabla\mathcal{L}|^2\le-2\mu\mathcal{L}, and ddtL=−∣∇L∣2/(2L)≤−μ/2 ∣∇L∣\frac{d}{dt}\sqrt{\mathcal{L}}=-|\nabla\mathcal{L}|^2/(2\sqrt\mathcal{L})\le-\sqrt{\mu/2}\,|\nabla\mathcal{L}|, which integrates to the bound on the length ∫∣γ˙∣=∫∣∇L∣\int|\dot\gamma|=\int|\nabla\mathcal{L}|. ◻

The inequality is the quantitative form of transversality: μ>0\mu>0 says that ev\mathrm{ev} is a submersion on UU with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the qualitative theory hands over to analysis.

Sublevel sets and barriers

For c∈Rc\in\mathbb{R} let Θc={L≤c}\Theta^c=\{\mathcal{L}\le c\}. Two parameters are connected at level cc if they lie in the same component of Θc\Theta^c, and the barrier between them is the infimum of such cc. This is the notion studied empirically under the name of mode connectivity [17, 10].

Proposition 3.3 (Barriers). [reformulation; cf. [9, Cor. 8.7, Prop. 8.10]] Let L ⁣:Θ→R\mathcal{L}\colon\Theta\to\mathbb{R} be a Morse function with compact sublevel sets. Then the components of Θc\Theta^c change only at critical values: a component is born at each local minimum, and two components merge at a critical point of index 11 whose two descending directions lead to different components. The barrier between two local minima is the level at which their components merge; it is attained at a critical point of index 11, and equals the minimum over paths joining them of the maximum of L\mathcal{L} along the path.

So barriers are the death times in the degree-zero barcode of the sublevel functor [9, Prop. 8.10], and the barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a generic analytic loss with weight decay, L+ε2∣θ∣2\mathcal{L}+\tfrac\varepsilon2|\theta|^2, on an architecture without continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail (3.5).

Example 3.4 (Two parameters). Let F(a,b;x)=abxF(a,b;x)=abx with one data point (x,y)=(1,1)(x,y)=(1,1), so L=12(ab−1)2\mathcal{L}=\tfrac12(ab-1)^2. The interpolation manifold is the hyperbola ab=1ab=1, of dimension P−N=1P-N=1, with two components exchanged by the symmetry (a,b)↦(−a,−b)(a,b)\mapsto(-a,-b). The only other critical point is the origin, a saddle with L=12\mathcal{L}=\tfrac12 and Hessian eigenvalues ±1\pm1. For c<12c<\tfrac12 the sublevel set has two components; at c=12c=\tfrac12 they merge. So the barrier between (1,1)(1,1) and (−1,−1)(-1,-1) is 12\tfrac12, attained on the straight segment between them, and no path does better. After applying the symmetry to one of the two points they coincide. The script checks the component counts on a grid.

The example is the smallest instance of a general phenomenon. By 3.8 minima come in orbits of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a barrier. The barrier between two trained networks should therefore be measured after choosing representatives of their orbits, which is the content of the conjecture of Entezari et al. that barriers largely disappear once permutations are accounted for [11]. In the language of 3, the relevant sublevel sets are those of the function induced on the quotient Θ/G\Theta/G.

Two results from the literature describe the overparametrized case. For a class of deep networks with piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen proved that every sublevel set of the loss is connected [26]. And Şimşek et al. describe the critical manifolds that the reducible strata of 3.6 contribute, and show how adding hidden units connects the ∣G∣|G| copies of a minimum [29].

Why the loss is rarely a Morse function

Three mechanisms make the loss degenerate, and they are independent.

  1. Overparametrization. If P>NP>N the zero-loss minima form the manifold MyM_y of positive dimension (3.1).

  2. Continuous symmetry. For homogeneous activations every critical point has a Hessian of nullity at least HH (3.9).

  3. Reducible strata. A critical point of a smaller network gives a positive-dimensional family of critical points of a larger one, along the fibres of 3.6; these are typically saddles [12].

In each case the degeneracy is of Morse–Bott type on an open dense part of the critical set, and the handle theorem [9, Thm. 8.6] has a Morse–Bott version in which a critical manifold contributes a bundle of cells over itself. But the density of Morse functions [9, Cor. 4.8] is of no direct use: a generic perturbation of the loss as a function on Θ\Theta is not the loss of any data set, and perturbing the data preserves all three mechanisms. This is the point of 1.5 again. Genericity must be established within the family one actually has.

Critical points at infinity

Proposition 3.5. [observation] For architectures with a bounded activation the loss is not proper, and its infimum need not be attained. For F(w;x)=tanh⁡(wx)F(w;x)=\tanh(wx) with the single data point (1,1)(1,1), L(w)=12(tanh⁡w−1)2\mathcal{L}(w)=\tfrac12(\tanh w-1)^2 has no critical point, its infimum 00 is not attained, and every sublevel set Θc\Theta^c with 0<c<20<c<2 is a half-line.

Proof. L′(w)=(tanh⁡w−1)sech⁡2w<0\mathcal{L}'(w)=(\tanh w-1)\operatorname{sech}^2w<0 for all ww. ◻

So the topology of sublevel sets can change with no critical point to account for it, the change being carried off to infinity, and the regular interval theorem [9, Thm. 8.4], which needs compactness, does not apply. Any use of Morse theory for losses must either restore compactness (weight decay, a bounded parameter domain) or account for these ends explicitly, as one does for the Palais–Smale condition in infinite dimensions.

Notes

The manifold of global minima was described by Cooper [6]; the Polyak–Łojasiewicz condition in this setting is due to Liu, Zhu and Belkin. Mode connectivity was observed by Garipov et al. and Draxler et al., and analysed for wide networks by Freeman and Bruna and by Nguyen; permutation alignment is studied in [11]. For the singular structure of the landscape at reducible parameters see [12, 29] and Watanabe’s theory [31].

Exercises

Exercise 3.1. For the linear family of 2.4 with P>nP>n and features in general position, describe MyM_y and the Hessian of the loss.

Exercise 3.2. Verify 3.4: find all critical points of 12(ab−1)2\tfrac12(ab-1)^2, their Hessians, and the components of the sublevel sets.

Exercise 3.3. In the setting of 3.2, show that the limit point of the flow line lies within 2L(θ0)/μ\sqrt{2\mathcal{L}(\theta_0)/\mu} of θ0\theta_0.

Exercise 3.4. For F(a1,…,ak;x)=a1⋯akxF(a_1,\dots,a_k;x)=a_1\cdots a_kx with data (1,1)(1,1), show that MyM_y has 2k−12^{k-1} components and dimension k−1k-1, and that the origin is a degenerate critical point for k≥3k\ge3.

Maps

Embeddings, Isotopy, and Depth

The previous three chapters studied level sets. This one studies the maps themselves, and in particular what a network does to a data set regarded as a subset of input space. The standing assumption, usually called the manifold hypothesis [14], is that the data lie on or near a compact submanifold K⊂RDK\subset\mathbb{R}^D of dimension mm much smaller than DD, with the classes occupying disjoint closed subsets K+,K−K_+,K_-. A hidden layer of width kk is a map hθ ⁣:RD→Rkh_\theta\colon\mathbb{R}^D\to\mathbb{R}^k, and the question is what hθ(K)h_\theta(K) can be.

The first layer generically embeds

Proposition 1.1. [reformulation; cf. Whitney; [9, Thm. 4.5, Cor. 4.8, Prop. 3.12]] Let K⊂RDK\subset\mathbb{R}^D be a compact submanifold of dimension mm, let k≥2m+1k\ge2m+1, and let σ\sigma be a diffeomorphism of R\mathbb{R} onto an open interval. Then for almost every W∈Rk×DW\in\mathbb{R}^{k\times D} and every bb, the map x↦σ(Wx+b)x\mapsto\sigma(Wx+b) restricted to KK is an embedding. The set of such (W,b)(W,b) is open.

Proof. Applying σ\sigma coordinatewise after a translation is a diffeomorphism onto its image, so the question is whether W∣KW|_K is an embedding. The map (W,(x,y))↦W(x−y)(W,(x,y))\mapsto W(x-y) on Rk×D×(K×K∖Δ)\mathbb{R}^{k\times D}\times(K\times K\setminus\Delta) is a submersion to Rk\mathbb{R}^k, so by [9, Thm. 4.5] for almost every WW its restriction to K×K∖ΔK\times K\setminus\Delta, a manifold of dimension 2m<k2m<k, is transverse to 00 and hence misses it: W∣KW|_K is injective. Likewise (W,(x,v))↦Wv(W,(x,v))\mapsto Wv on the unit tangent bundle of KK, of dimension 2m−12m-1, is a submersion, so for almost every WW no unit tangent vector is killed: W∣KW|_K is an immersion. An injective immersion of a compact manifold is an embedding. Openness is [9, Prop. 3.12]. ◻

So a hidden layer wider than twice the intrinsic dimension of the data loses no topological information for almost every choice of weights, however small it is compared with the ambient dimension DD. This is Whitney’s embedding theorem read as a statement about random projections. Narrower layers may fail to be injective on KK, and that is where the topology of a representation can change.

What injective networks cannot do

A classifier ends with a linear map, so the last hidden representation must make the classes linearly separable. If the network up to that point is injective, its restriction to KK is a homeomorphism onto the image, and separability becomes a constraint on how K+K_+ and K−K_- sit in the ambient space.

Theorem 1.2 (Nested classes). [classical; invariance of domain; cf. [24]] Let h ⁣:Rd→Rdh\colon\mathbb{R}^d\to\mathbb{R}^d be continuous and injective, let β⊂Rd\beta\subset\mathbb{R}^d be a closed topological ball with boundary sphere SS, and let pp be an interior point of β\beta. Then h(p)h(p) lies in the convex hull of h(S)h(S). In particular no hyperplane separates h(p)h(p) from h(S)h(S).

Proof. Suppose a linear functional λ\lambda satisfies λ>0\lambda>0 on h(S)h(S) and λ(h(p))≤0\lambda(h(p))\le0. Let qq be a point of the compact set h(β)h(\beta) at which λ\lambda is smallest. Then λ(q)≤0\lambda(q)\le0, so q∉h(S)q\notin h(S) and q∈h(int⁡β)q\in h(\operatorname{int}\beta), which is open in Rd\mathbb{R}^d by invariance of domain. But a linear functional has no local minimum on an open set. ◻

Corollary 1.3. [classical; [24, 22]] A feedforward network on Rd\mathbb{R}^d in which every hidden layer has width dd, every weight matrix is invertible, and the activation is injective, cannot separate a point from a sphere surrounding it. To do so a network must either have a layer of width at least d+1d+1 or use a layer that is not injective.

Both escapes are used in practice. A wider layer provides the extra dimension into which the inner class can be lifted. A ReLU\mathop{\mathrm{ReLU}} layer is not injective: it folds space along hyperplanes, and a network of ReLU\mathop{\mathrm{ReLU}} layers can map a nested data set to a separable one by collapsing. Naitzat, Zhitnikov and Lim [27] measured the Betti numbers of the two classes layer by layer and found that well-trained networks reduce them steadily, faster with ReLU\mathop{\mathrm{ReLU}} than with smooth invertible activations. In the terms of this chapter: smooth invertible layers of constant width act by diffeomorphisms and cannot change topology, and the network must then rely on the few layers where injectivity fails.

Neural ordinary differential equations

A neural ODE [7] is the time-TT map φθ\varphi_\theta of the equation x˙=vθ(x,t)\dot x=v_\theta(x,t), where vθv_\theta is a network with input (x,t)(x,t) and output in Rd\mathbb{R}^d.

Proposition 1.4. [reformulation; cf. [9, Thm. 10.2]] If vθv_\theta is smooth with bounded velocity, as it is when its last hidden layer has a bounded activation, then t↦φθtt\mapsto\varphi^t_\theta is a diffeotopy of Rd\mathbb{R}^d. Hence φθ\varphi_\theta is a diffeomorphism isotopic to the identity, it preserves orientation, and for every compact K⊂RdK\subset\mathbb{R}^d the image φθ(K)\varphi_\theta(K) is ambient isotopic to KK.

Corollary 1.5 (Obstructions). [classical; cf. Dupont–Doucet–Teh [8]] A neural ODE on Rd\mathbb{R}^d cannot:

  1. for d=1d=1, reverse the order of two points;

  2. map a sphere and a point inside it to linearly separable sets (1.2);

  3. for d=3d=3, map two linked circles to unlinked ones, or a knotted circle to a round one.

Proof. (a) A diffeomorphism of R\mathbb{R} isotopic to the identity is increasing. (c) The linking number is the degree of the Gauss map [9, Ex. 7.12] and is invariant under isotopy by [9, Thm. 7.2]; the knot type is an isotopy invariant by definition. ◻

These are statements about exact representation. A neural ODE can still approximate a map that violates them, at the cost of a vector field with large derivatives, and [8] observed that training then becomes slow and unstable. Their remedy was to append extra coordinates to the state.

Augmentation

Theorem 1.6 (Flows in the stable range). [reformulation; cf. [9, Thm. 10.9, Thm. 10.3, Thm. 10.2]] Let KK be a compact manifold of dimension mm and let e0,e1 ⁣:K→RNe_0,e_1\colon K\to\mathbb{R}^N be embeddings, with N≥2m+2N\ge2m+2. Then there is a smooth, compactly supported, time-dependent vector field on RN\mathbb{R}^N whose time-one map φ\varphi satisfies φ∘e0=e1\varphi\circ e_0=e_1.

Proof. By [9, Thm. 10.9] the two embeddings are isotopic. By the isotopy extension theorem [9, Thm. 10.3] the isotopy is induced by a diffeotopy of RN\mathbb{R}^N with compact support, and by [9, Thm. 10.2] the diffeotopy is the flow of its velocity field. ◻

Corollary 1.7 (Separation after augmentation). [observation] Let K=K+⊔K−⊂RdK=K_+\sqcup K_-\subset\mathbb{R}^d be a compact submanifold of dimension mm, the union of two disjoint compact submanifolds, and let N≥max⁡(d+1,2m+2)N\ge\max(d+1,2m+2). Then there is a compactly supported flow on RN\mathbb{R}^N whose time-one map carries K+×{0}K_+\times\{0\} and K−×{0}K_-\times\{0\} into the open half-spaces {xN>0}\{x_N>0\} and {xN<0}\{x_N<0\}. The vector field may be taken to be a neural network with one hidden layer and input (x,t)(x,t).

Proof. Take e0e_0 the inclusion of KK in Rd×{0}⊂RN\mathbb{R}^d\times\{0\}\subset\mathbb{R}^N and e1=e0±(0,…,0,1)e_1=e_0\pm(0,\dots,0,1) on K±K_\pm, and apply 1.6 to get a field vv with flow φ\varphi. For the last statement, approximate vv in C1C^1 on a compact set containing the support, by Hornik’s theorem [21]; by Grönwall’s inequality the time-one maps are then close in C1C^1 on compact sets, and the conclusion, being an open condition on the images of the compact sets K±K_\pm, persists. ◻

So the obstructions of 1.5 disappear once the state space has dimension at least 2m+22m+2, where mm is the dimension of the data and not of the ambient space. For finite data sets m=0m=0 and two dimensions suffice, in agreement with (a): points on a line cannot be reordered by a flow, points in the plane can. For curves the bound is 44, and it is sharp, since circles link in R3\mathbb{R}^3. For connected data of dimension m≥2m\ge2 the bound can be lowered to 2m+12m+1 by the second part of [9, Thm. 10.9]. Zhang et al. [32] prove a related statement about maps: every homeomorphism of Rd\mathbb{R}^d can be embedded in a flow on R2d\mathbb{R}^{2d}, and hence approximated by a neural ODE or an invertible residual network acting there. The dimension count of 1.6 depends on the data alone.

Residual networks fit the same description. A block x↦x+g(x)x\mapsto x+g(x) with gg of Lipschitz constant less than one is a diffeomorphism isotopic to the identity (2.2), and a composite of such blocks is a discretized isotopy. Depth, for such architectures, is time.

The space of representations

Open Problem 1.8. [programme] For a compact data manifold KK of dimension mm and width k≥m+3k\ge m+3, the space Emb(K,Rk)\mathop{\mathrm{Emb}}(K,\mathbb{R}^k) of representations that preserve the topology of the data is described by manifold calculus [9, Thm. 10.10]: it is the limit of a tower whose first stage is the space of immersions and whose higher stages are built from configuration spaces of KK. Determine which components and which homotopy classes of this space are reached by the realization map of a given architecture, as a function of width and depth.

For k≥2m+2k\ge2m+2 the space is connected and the question is about higher homotopy; in the range m+3≤k≤2m+1m+3\le k\le2m+1 already the set of components is interesting, and 1.5(c) is the first example outside that range.

Notes

The picture of a network as progressively untangling data manifolds goes back at least to expository writing of Olah in 2014, where 1.2 for d=2d=2 and the role of an extra dimension are described. Minimal width for universal approximation is treated in [22, 24]. Neural ODEs are from [7], their limitations and augmentation from [8], and approximation properties from [32].

Exercises

Exercise 1.1. Show that the time-one map of a bounded smooth vector field on R\mathbb{R} is increasing.

Exercise 1.2. Write down a compactly supported vector field on R2\mathbb{R}^2 whose time-one map exchanges the points (±1,0)(\pm1,0).

Exercise 1.4. Let KK be the curve t↦(cos⁡t,sin⁡t,cos⁡2t,sin⁡2t,cos⁡3t)t\mapsto(\cos t,\sin t,\cos2t,\sin2t,\cos3t) in R5\mathbb{R}^5. Show that the projection to the first two coordinates is an embedding of KK, and that the projection to the third and fourth is not.

Degree and Invertibility

Some architectures are designed so that every member is invertible: normalizing flows, invertible residual networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from [9, Ch. 7].

Global invertibility

Theorem 2.1 (Hadamard). [classical; ] A local diffeomorphism f ⁣:Rd→Rdf\colon\mathbb{R}^d\to\mathbb{R}^d is a diffeomorphism if and only if it is proper.

Proof. A proper local diffeomorphism is a covering map, and Rd\mathbb{R}^d is simply connected. In terms of degree: every point is a regular value, the sign of det⁡Df\det Df is constant, and so the number of preimages of a point is the absolute value of the degree, which is constant; it is 11 because a covering of a simply connected space is trivial. ◻

A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded one.

Proposition 2.2 (Invertible residual blocks). [classical; Banach; Behrmann et al. [1]] Let g ⁣:Rd→Rdg\colon\mathbb{R}^d\to\mathbb{R}^d have Lipschitz constant L<1L<1 and let f=id+gf=\mathrm{id}+g.

  1. ff is a bijection, its inverse is Lipschitz with constant (1−L)−1(1-L)^{-1}, and xj+1=y−g(xj)x_{j+1}=y-g(x_j) converges to f−1(y)f^{-1}(y) with ∣xj−f−1(y)∣≤Lj∣x0−f−1(y)∣|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|.

  2. If gg is smooth then ff is a diffeomorphism with det⁡Df>0\det Df>0.

  3. fs=id+sgf_s=\mathrm{id}+sg, 0≤s≤10\le s\le1, is a diffeotopy from the identity to ff. In particular ff is proper of degree 11.

Proof. (a) x↦y−g(x)x\mapsto y-g(x) is a contraction; and ∣f(x)−f(x′)∣≥(1−L)∣x−x′∣|f(x)-f(x')|\ge(1-L)|x-x'|. (b), (c): the eigenvalues of DgDg have modulus at most LL, so 1+sDg1+sDg is invertible for s∈[0,1]s\in[0,1], and its determinant is continuous, non-zero, and equal to 11 at s=0s=0. ◻

Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common use all realize diffeomorphisms in the identity component of Diff(Rd)\mathop{\mathrm{Diff}}(\mathbb{R}^d), and by [9, Thm. 10.2] every such diffeomorphism with compact support is conversely the time-one map of a flow.

Topological mismatch

A normalizing flow models a probability measure ν\nu on Rd\mathbb{R}^d as the image h∗μh_*\mu of a simple measure μ\mu, usually a Gaussian, under a diffeomorphism hh.

Proposition 2.3. [classical; cf. Cornish et al. [3]] For a homeomorphism hh of Rd\mathbb{R}^d and a Borel probability measure μ\mu, supp⁡(h∗μ)=h(supp⁡μ)\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu). Hence if the supports of μ\mu and ν\nu are not homeomorphic, no homeomorphism satisfies h∗μ=νh_*\mu=\nu.

A Gaussian has support Rd\mathbb{R}^d. A target supported on two disjoint balls, on a sphere, or on any proper closed subset cannot be represented exactly, and Cornish et al. show that a sequence of flows approximating such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature are the ones differential topology suggests: a target supported on a submanifold Mm⊂RdM^m\subset\mathbb{R}^d is modelled by a flow on Rm\mathbb{R}^m followed by an embedding Rm→Rd\mathbb{R}^m\to\mathbb{R}^d, and when MM is not diffeomorphic to an open subset of Rm\mathbb{R}^m, by several such charts with a discrete index, which is an atlas in the sense of [9, Ch. 1]. The minimal number of charts is an invariant of MM; for a sphere it is two.

Equilibrium layers

A deep equilibrium layer outputs a solution zz of z=gθ(z,x)z=g_\theta(z,x) for a network gθg_\theta. The existence and number of solutions is a fixed-point problem, and [9, Prop. 7.11] describes its three aspects: a generic signed count, a derived fibre product, and a trace.

Theorem 2.4 (Equilibria of a bounded layer). [reformulation; cf. [9, Thm. 4.5, Thm. 7.2, Thm. 7.9]] Let g(z)=tanh⁡(Az+u)g(z)=\tanh(Az+u) on Rk\mathbb{R}^k, with A∈Rk×kA\in\mathbb{R}^{k\times k} and u∈Rku\in\mathbb{R}^k.

  1. For every AA and almost every uu, every fixed point of gg is non-degenerate: 11 is not an eigenvalue of DgDg there. The fixed points are then finite in number.

  2. In that case ∑g(z)=zsign⁡det⁡(1−Dg(z))=1\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1. In particular the number of fixed points is odd, and at least one exists.

  3. Each non-degenerate fixed point is locally a smooth function of (A,u)(A,u), with dz=(1−Dzg)−1 d(A,u)gdz=(1-D_zg)^{-1}\,d_{(A,u)}g.

Proof. (a) The map Ψ(u,z)=z−tanh⁡(Az+u)\Psi(u,z)=z-\tanh(Az+u) has DuΨ=−diag⁡(sech⁡2(Az+u))D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u)), which is invertible; so Ψ\Psi is a submersion and by [9, Thm. 4.5] zero is a regular value of Ψ(u,⋅)\Psi(u,\cdot) for almost every uu. Regularity at a zero zz means that 1−Dg(z)1-Dg(z) is invertible. All fixed points lie in the open cube (−1,1)k(-1,1)^k; being isolated in a compact set they are finite. (b) For 0≤s≤10\le s\le1 the map gs(z)=tanh⁡(s(Az+u))g_s(z)=\tanh(s(Az+u)) sends the closed cube Q=[−1,1]kQ=[-1,1]^k into its interior, so id−gs\mathrm{id}-g_s has no zero on ∂Q\partial Q, and the degree at 00 of id−gs ⁣:(Q,∂Q)→(Rk,Rk∖0)\mathrm{id}-g_s\colon(Q,\partial Q)\to (\mathbb{R}^k,\mathbb{R}^k\setminus0) is independent of ss by [9, Thm. 7.2]. At s=0s=0 it is the degree of the identity. At s=1s=1, zero being a regular value, it is the stated sum. (c) is the implicit function theorem. ◻

In one dimension, z=tanh⁡(az+u)z=\tanh(az+u) has one solution for a≤1a\le1, and for a>1a>1 and small ∣u∣|u| three, with signs +,−,++,-,+: two stable equilibria and an unstable one between them. The script finds the fixed points for several (a,u)(a,u) and for a two-dimensional example and checks that the signed count is 11.

Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set Δ\Delta of 1; crossing it creates or destroys a pair of equilibria of opposite sign, as in the birth–death family of [9, Thm. 8.12], and it is there that the output of an equilibrium layer can jump.

Notes

The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and Mazur, Palais). Invertible residual networks are from [1]. The observation that the topology of the support obstructs normalizing flows, with a quantitative form, is in [3]. Deep equilibrium models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone operator parametrizations) are a separate line of work, and 2.4 describes what happens without them.

Exercises

Exercise 2.1. Prove 2.2(a).

Exercise 2.2. Let gg be smooth with ∥Dg∥≤L<1\|Dg\|\le L<1 everywhere. Show det⁡(1+Dg)≥(1−L)d\det(1+Dg)\ge(1-L)^d.

Exercise 2.3. For k=1k=1 and u=0u=0, find the fixed points of z=tanh⁡(az)z=\tanh(az) as a function of aa and verify 2.4(b). What happens at a=1a=1?

Exercise 2.4. Prove 2.3.

Assessment

What Was Used, and What Is Missing

The inputs

lists the main statements of this book with the theorems of the companion volume from which they follow. Every entry in the right-hand column was known by the mid-1970s.

What is used.
statement here from [9]
backpropagation composes (2.1) reverse chain rule, Prop. 2.11
generic decision boundaries are hypersurfaces (1.1) Sard, Thm. 4.1; regular values, Thm. 1.12
chambers and walls (Theorem 1.4, Proposition 1.5) fibrewise lemma 4.4; Ehresmann, Thm. 8.5; isotopy extension, Thm. 10.3
every hypersurface is realized (1.6) tubular neighbourhoods, Thm. 6.2; openness, Thm. 3.2
surgery along paths (Theorem 2.1, Theorem 2.2) neat preimages, Thm. 1.17; handles, Thm. 8.6
moves between paths (2.7) Cerf, Thm. 8.12
interpolation manifold (3.1) regular values, Thm. 1.12; Sard, Thm. 4.1
barriers (3.3) sublevel functor, Cor. 8.7, Prop. 8.10
first layer embeds (1.1) parametric transversality, Thm. 4.5
neural ODEs are diffeotopies (1.4) flows, Thm. 10.2
augmentation (1.6) isotopy in the stable range, Thm. 10.9; Thm. 10.3
odd number of equilibria (2.4) degree, Thm. 7.2; Thm. 4.5

What is and is not new

Nothing in this book is a new theorem of differential topology, and most of its statements about networks have appeared in the machine learning literature in some special case, with the references given in the text: generic smoothness of decision boundaries, the manifold of global minima, the obstruction to neural ODEs and its removal by augmentation, the mismatch of supports for flows. What the book adds is the uniform derivation, with the hypothesis that makes each statement true stated as a transversality property of the architecture (1.5), and a small number of statements tagged as observations that we have not found in this form: the description of the moduli of irreducible one-hidden-layer networks as a configuration space, with its monodromy (3.5); the wall structure of the discriminant (1.5); the surgery description of training with its Euler characteristic count (Theorem 2.2, Proposition 2.3) and its moves (2.7); the dimension count 2m+22m+2 for augmentation in terms of the data (Theorem 1.6, Corollary 1.7); and the parity of the number of equilibria (2.4). Each is a short corollary and is offered as such.

The higher-categorical layer was used as language more than as a tool. The bicategory Para\mathsf{Para} and the lens functor organize Families of Maps, Derivatives; the quotient by symmetries is a stack and not a space for homogeneous activations (3); and a degenerate level set is a derived manifold with a cobordism class (1.2). The theorems that do the work are those of 1.1.

Limits

Smoothness. The standing assumption excludes ReLU\mathop{\mathrm{ReLU}}. For piecewise-linear networks the analogue of 1.1 is the theorem of Grigsby and Lindsey [18]; analogues of Theorem 1.4, Theorem 2.2 would require stratified Morse theory, and are not proved here.

Genericity within the family. “Almost every parameter” is with respect to Lebesgue measure on Θ\Theta, so the statements apply to parameters drawn from a distribution with a density, as at a random initialization. They do not apply to distinguished parameters. In particular, if gg is an orthogonal symmetry of the architecture, the gradient of an invariant loss is equivariant, and so the fixed-point set of gg is invariant under gradient flow: a network initialized with two identical hidden units keeps them identical for ever, and stays in a reducible stratum of measure zero where none of the generic statements need hold.

Paths. describes generic paths. A training trajectory is a particular path, the solution of a differential equation, and we have proved nothing about whether it is generic. The honest statement is 2.5: arbitrarily close to it there is a path with the stated behaviour.

Compactness. Input domains are assumed compact for the isotopy statements, and parameter space is never compact, which restricts the use of Morse theory for losses (3.5).

No rates. With the single exception of 3.2, the statements are qualitative. They say that a set has measure zero and not how small its neighbourhoods are; that an isotopy exists and not how large its velocity must be; that a boundary of a given topology is realizable and not at what width. Each of these has a quantitative counterpart in the literature that the present methods do not reach.

No learning theory. Nothing here concerns generalization.

Directions

Open Problem 1.1 (Piecewise-linear version). [programme] Prove the analogues of Theorem 1.4, Theorem 2.2 for ReLU\mathop{\mathrm{ReLU}} networks that are generic and transversal in the sense of [18], with handle attachments replaced by the elementary moves of stratified Morse theory on the canonical polyhedral complex.

Open Problem 1.2 (Quantitative discriminant). [programme] For a given architecture and initialization distribution, estimate the measure of the ε\varepsilon-neighbourhood of the discriminant Δ\Delta of 1, and hence the expected number of surgeries along a path of given length. suggests that Δ\Delta is a hypersurface and that the answer is governed by its volume.

Open Problem 1.3 (The stack of networks). [programme] For homogeneous activations, describe the quotient stack [Θ/G][\Theta/G] of 3 and the critical locus of a loss on it as a derived object, so that 3.1 and the degeneracies listed in 3 become statements about a single space with a virtual dimension.

Together with 1.8, these are the places where the methods of the companion volume, and in particular its higher-categorical parts (derived manifolds, stacks, the calculus of embeddings), would be doing more than providing a vocabulary.

Dictionary

Network notion Differential topology Where
architecture F ⁣:Θ×X→YF\colon\Theta\times X\to Y plot of the mapping object [X,Y][X,Y]; morphism of Para\mathsf{Para} Definition 1.1, Proposition 1.3
composition of layers composition of plots 1.2
weight sharing, freezing 22-cells of Para\mathsf{Para} 1.3
forward-mode derivative tangent functor [9, Prop. 2.1]
backpropagation reverse derivative as a functor to lenses 2.1
tangent kernel Gram matrix JJTJJ^{\mathsf T} of the evaluation map 2.3
functional dimension rank of the realization map Definition 2.5, Proposition 2.4
permutation, sign, scaling symmetry group of invertible 22-cells 3.2
redundant hidden unit reducible parameter; positive-dimensional fibre 3.6
space of networks quotient manifold, or stack 3.5
decision boundary preimage of a regular value 1.1
multi-class boundaries preimage of a stratification 1.3
robust topology of a classifier chamber of the discriminant complement 1.4
change of boundary topology surgery; critical point of the trace 2.2
training run, seen in input space embedded cobordism with a Morse function 2.1
two runs with the same endpoints Cerf moves 2.7
interpolating parameters fibre of the evaluation map over a regular value 3.1
sharpness at a minimum normal Hessian JTJJ^{\mathsf T}J 2.6
loss barrier, mode connectivity death level in the degree-zero barcode 3.3
saturation, diverging weights failure of properness 3.5
hidden representation immersion or embedding of the data manifold 1.1
minimal width invariance of domain 1.2
neural ODE diffeotopy 1.4
augmented neural ODE isotopy of embeddings in the stable range 1.6
residual block with small branch diffeomorphism isotopic to the identity 2.2
normalizing flow diffeomorphism; pushforward of measures 2.3
equilibrium layer transverse fixed point; Lefschetz count 2.4

Computations

The script verify-parametrized-networks.py uses only the Python standard library. It implements small feedforward networks (widths (2,3,1)(2,3,1), (2,4,4,1)(2,4,4,1) and a few others) with forward evaluation, forward-mode differentiation by dual numbers and reverse-mode differentiation by backpropagation, a Runge–Kutta integrator, a Jacobi eigenvalue routine and a flood-fill component counter. It performs 49 checks in about three seconds, each printing PASS or FAIL, and exits with status zero if and only if all pass. The checks recompute the examples in the text and test numerical instances of the structural statements on random parameters with a fixed seed. They are not proofs, and a statement about almost every parameter is tested only at the parameters sampled.

Chapter What is checked
1 parameter counts (1.1); associativity of composition in Para\mathsf{Para}; the derivative of a weight-tied network (1.3); the family θ+x3\theta+x^3 (1.4)
2 backpropagation against forward mode and finite differences for two architectures, and for the input covector (2.1); the tangent kernel is positive semidefinite of rank rkJ\mathop{\mathrm{rk}}J; the output dynamics u˙=−K(u−y)\dot u=-K(u-y) (2.3); the loss gradient; the Hessian as Gauss–Newton plus curvature term (2.3)
3 all 4848 symmetries of the tanh⁡\tanh network (2,3,1)(2,3,1) and the size of an orbit (3.2); the scaling symmetry for ReLU\mathop{\mathrm{ReLU}}; rank P=13P=13 at a random parameter and ranks 1010 and 99 at reducible ones (3.4, 3.2); for ReLU\mathop{\mathrm{ReLU}}, the generators ξj\xi_j in ker⁡J\ker J and rkJ≤P−H\mathop{\mathrm{rk}}J\le P-H (3.7); a path of irreducible networks exchanging two hidden units (3.5)
1 ∂F/∂c=1\partial F/\partial c=1; degenerate zeros of a one-dimensional network occur only at the critical values (1.1(a)); the two chambers of x12−x22+cx_1^2-x_2^2+c (1.3); three sectors at a triple point (1.4); the submersion of 1.5
2 component counts, critical points and indices in 2.4; the Euler characteristic formula of 2.3 for d≤6d\le6; the Hessian of the height function on the trace (2.4); births and deaths of boundary points on a circle (2.2)
3 rkJ=N\mathop{\mathrm{rk}}J=N and the dimension P−NP-N of the solution set; the Hessian JTJJ^{\mathsf T}J with 44 positive and 99 zero eigenvalues (3.1); the inequality of 3.2; sublevel components, the barrier and the Hessians of 3.4; 3.5; 3.4; invariance of the loss under the symmetries
1 random first layers of width 33 embed a circle in R5\mathbb{R}^5 (1.1), and a coordinate projection that does not (1.4); the convex hull property of 1.2 for random injective networks; order preservation on the line; positivity of the Jacobian determinant and Liouville’s formula for a planar neural ODE (1.4); the swap of 1.2; invariance of the linking number under a neural ODE in R3\mathbb{R}^3 (1.5); separation of nested circles after augmentation (1.7)
2 convergence of the fixed-point inversion and the determinant bound for a residual block (2.2, 2.2); a non-proper local diffeomorphism (2.1); odd numbers of equilibria with signed count 11 in one and two dimensions (2.4)

Solutions to the Exercises

Solutions to all exercises follow, by chapter. Where a solution relies on a computation, the computation is also carried out by verify-parametrized-networks.py.

Chapter 1

1.1. (2,3,1)(2,3,1): P=3⋅3+1⋅4=13P=3\cdot3+1\cdot4=13, H=3H=3. (2,4,4,1)(2,4,4,1): P=4⋅3+4⋅5+1⋅5=37P=4\cdot3+4\cdot5+1\cdot5=37, H=8H=8. (d,k,1)(d,k,1): P=k(d+1)+(k+1)=k(d+2)+1P=k(d+1)+(k+1)=k(d+2)+1, H=kH=k.

1.2. Let (Θ,F) ⁣:X→Y(\Theta,F)\colon X\to Y, (Θ′,G) ⁣:Y→Z(\Theta',G)\colon Y\to Z, (Θ′′,K) ⁣:Z→V(\Theta'',K)\colon Z\to V. Both bracketings have underlying function (θ,θ′,θ′′,x)↦K(θ′′,G(θ′,F(θ,x)))(\theta,\theta',\theta'',x)\mapsto K(\theta'',G(\theta',F(\theta,x))), on the parameter spaces (Θ×Θ′)×Θ′′(\Theta\times\Theta')\times\Theta'' and Θ×(Θ′×Θ′′)\Theta\times(\Theta'\times\Theta'') respectively. The reassociation diffeomorphism rr satisfies Fleft=Fright∘(r×idX)F_{\text{left}}=F_{\text{right}}\circ(r\times \mathrm{id}_X), so rr and r−1r^{-1} are 22-cells, mutually inverse.

1.3. The diagonal δ ⁣:Θ→Θ×Θ\delta\colon\Theta\to\Theta\times\Theta gives a 22-cell (Θ,(F∘F)∘(δ×id))⇒(Θ×Θ,F∘F)(\Theta,(F\circ F)\circ(\delta\times\mathrm{id}))\Rightarrow(\Theta\times\Theta,F\circ F), and its source is (θ,x)↦F(θ,F(θ,x))(\theta,x)\mapsto F(\theta,F(\theta,x)), the weight-tied network. By the chain rule, the derivative of θ↦G(δ(θ))\theta\mapsto G(\delta(\theta)) is ∂1G+∂2G\partial_1G+\partial_2G evaluated on the diagonal, for G(θ1,θ2)=F(θ2,F(θ1,x))G(\theta_1,\theta_2)=F(\theta_2,F(\theta_1,x)).

1.4. F(θ,x)=θ+x3F(\theta,x)=\theta+x^3. Every member has the degenerate critical point x=0x=0. A function is Morse when its 11-jet prolongation is transverse to the set of jets with vanishing derivative; the family of prolongations (θ,x)↦(x,θ+x3,3x2)(\theta,x)\mapsto(x,\theta+x^3,3x^2) is not transverse to {third coordinate=0}\{\text{third coordinate}=0\}, since the third coordinate has vanishing derivative in both θ\theta and xx at x=0x=0. The parameter moves the value of the function but not its derivative, so the family is rich for conditions on values (1.5) and not for conditions on first derivatives. Adding a parameter θ1x\theta_1x repairs this: θ0+θ1x+x3\theta_0+\theta_1x+x^3 is Morse for θ1≠0\theta_1\ne0.

Chapter 2

2.1. Let z=Wx+bz=Wx+b and h=σ(z)h=\sigma(z). The lens of the layer has forward part (W,b,x)↦h(W,b,x)\mapsto h and backward part (W,b,x,w)↦(δxT, δ, WTδ)(W,b,x,w)\mapsto\bigl(\delta x^{\mathsf T},\ \delta,\ W^{\mathsf T}\delta\bigr) with δ=σ′(z)⊙w\delta=\sigma'(z)\odot w: the three components are the covectors for WW, bb and xx. The output layer (a,c;h)↦aTh+c(a,c;h)\mapsto a^{\mathsf T}h+c has backward part (a,c,h,w)↦(wh, w, wa)(a,c,h,w)\mapsto(wh,\ w,\ wa). The loss 12(u−y)2\frac12(u-y)^2 has gradient e=u−ye=u-y in uu. Composing: the covector entering the output layer is ee, so ∇a=e h\nabla_a=e\,h, ∇c=e\nabla_c=e, and the covector passed back is e ae\,a; then δ=σ′(z)⊙(e a)\delta=\sigma'(z)\odot(e\,a), ∇W=δxT\nabla_W=\delta x^{\mathsf T}, ∇b=δ\nabla_b=\delta.

2.2. ker⁡JJT=ker⁡JT\ker JJ^{\mathsf T}=\ker J^{\mathsf T} since ⟨v,JJTv⟩=∣JTv∣2\langle v,JJ^{\mathsf T}v\rangle=|J^{\mathsf T}v|^2; so rkK=rkJT=rkJ\mathop{\mathrm{rk}}K=\mathop{\mathrm{rk}}J^{\mathsf T}=\mathop{\mathrm{rk}}J. For scalar output KK is n×nn\times n, invertible if and only if rkJ=n\mathop{\mathrm{rk}}J=n, that is, if and only if the columns of JJ span Rn\mathbb{R}^n; the kk-th column is the vector (∂F/∂θk(θ,xi))i\bigl(\partial F/\partial\theta_k(\theta,x_i)\bigr)_{i}.

2.3. By [9, Prop. 2.9] the 22-jet of λ∘ev\lambda\circ\mathrm{ev} at θ\theta has linear part CAC A and quadratic part C∘B+E∘(A×A)C\circ B+E\circ(A\times A), where (A,B)(A,B) is the 22-jet of ev\mathrm{ev} (so A=JA=J and B=(∇2evk)kB=(\nabla^2\mathrm{ev}_k)_k) and (C,E)(C,E) that of λ\lambda (so C=∇λTC=\nabla\lambda^{\mathsf T}, E=∇2λE=\nabla^2\lambda). The quadratic part is ∑k∂kλ ∇2evk+JT∇2λ J\sum_k\partial_k\lambda\,\nabla^2\mathrm{ev}_k+J^{\mathsf T}\nabla^2\lambda\,J.

2.4. ev(θ)=Φθ\mathrm{ev}(\theta)=\Phi\theta with Φ\Phi the n×Pn\times P matrix of rows ϕ(xi)T\phi(x_i)^{\mathsf T}; so J=ΦJ=\Phi for all θ\theta and K=ΦΦTK=\Phi\Phi^{\mathsf T}, the Gram matrix of the features. The fibres of ev\mathrm{ev} are the affine subspaces θ0+ker⁡Φ\theta_0+\ker\Phi, of dimension P−rkΦP-\mathop{\mathrm{rk}}\Phi. The realization map is linear with kernel {θ:⟨θ,ϕ(x)⟩=0 ∀x}\{\theta:\langle\theta,\phi(x)\rangle=0\ \forall x\}, so the functional dimension is the dimension of the span of ϕ(X)\phi(X).

Chapter 3

3.1. F(θ,x)=∑jajσ(⟨wj,x⟩+bj)+cF(\theta,x)=\sum_ja_j\sigma(\langle w_j,x\rangle+b_j)+c. (i) Permuting the index jj reorders the sum. (ii) If σ\sigma is odd, (−aj)σ(−⟨wj,x⟩−bj)=ajσ(⟨wj,x⟩+bj)(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j). (iii) If σ(λs)=λσ(s)\sigma(\lambda s)=\lambda\sigma(s) for λ>0\lambda>0, then (aj/λ)σ(λ⟨wj,x⟩+λbj)=ajσ(⟨wj,x⟩+bj)(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j).

3.2. The columns of JJ belonging to (wj,bj)(w_j,b_j) are (ajσ′(⟨wj,xi⟩+bj) xi)i\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i and (ajσ′(⟨wj,xi⟩+bj))i\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i; if aj=0a_j=0 these d+1d+1 columns vanish, so rkJ≤P−(d+1)\mathop{\mathrm{rk}}J\le P-(d+1). If (wj,bj)=(wk,bk)(w_j,b_j)=(w_k,b_k), the columns for aja_j and aka_k are both (σ(⟨wj,xi⟩+bj))i\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i, so two columns coincide and rkJ≤P−1\mathop{\mathrm{rk}}J\le P-1. (In the second case the columns for (wj,bj)(w_j,b_j) and (wk,bk)(w_k,b_k) are also proportional, with ratio aj/aka_j/a_k, so the rank drops by d+2d+2 in all.)

3.3. θ=(w,b,a,c)\theta=(w,b,a,c) and ξ=(w,b,−a,0)\xi=(w,b,-a,0). Where wx+b>0wx+b>0, F=a(wx+b)+cF=a(wx+b)+c and DθF=(ax, a, wx+b, 1)D_\theta F=(ax,\ a,\ wx+b,\ 1), so DθF⋅ξ=awx+ab−a(wx+b)=0D_\theta F\cdot\xi=awx+ab-a(wx+b)=0. Where wx+b<0wx+b<0, F=cF=c and DθF=(0,0,0,1)D_\theta F=(0,0,0,1), again orthogonal to ξ\xi. Here P=4P=4, H=1H=1, so the functional dimension is at most 33; it equals 33 when a≠0a\ne0 and w≠0w\ne0, since the function x↦a ReLU(wx+b)+cx\mapsto a\,\mathop{\mathrm{ReLU}}(wx+b)+c is determined by its breakpoint −b/w-b/w, the slope awaw on the active side, and cc (and the side, which is discrete).

3.4. (2,3,1)(2,3,1): one hidden layer of 33 units, ∣G∣=23⋅3!=48|G|=2^3\cdot3!=48. (2,4,4,1)(2,4,4,1): two hidden layers of 44 units, ∣G∣=(24⋅4!)2=3842=147456|G|=(2^4\cdot4!)^2=384^2=147456.

Chapter 4

4.1. By 1.1(c), for almost every θ\theta the set BθB_\theta is a compact neat 00-dimensional submanifold of [−R,R][-R,R]: a finite set of interior points at which fθ′≠0f_\theta'\ne0, with fθ(±R)≠0f_\theta(\pm R)\ne0. At each such point fθf_\theta changes sign. So the number of zeros is even if fθ(−R)f_\theta(-R) and fθ(R)f_\theta(R) have the same sign and odd otherwise.

4.2. A compact neat 11-dimensional submanifold of the disc: a finite disjoint union of circles in the interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints by non-crossing arcs.

4.3. dxF=(2x1,−2x2)d_xF=(2x_1,-2x_2) vanishes only at x=0x=0, where F=cF=c. So Σ={(0,0)}\Sigma=\{(0,0)\} and Δ={0}\Delta=\{0\}. There are two chambers, c>0c>0 and c<0c<0. In each the boundary is a hyperbola; for c<0c<0 its branches open along the x1x_1-axis (x12=x22−c≥−cx_1^2=x_2^2-c\ge-c) and for c>0c>0 along the x2x_2-axis. The region {f<0}\{f<0\} is connected for c<0c<0 (it contains the origin) and has two components for c>0c>0. (The plane is not compact; the statement about chambers holds here because the family is a product near infinity up to scaling. Restricted to a disc the same description holds.)

4.4. At a triple point pp, fθf_\theta is transverse to A{1,2,3}A_{\{1,2,3\}}, the line y1=y2=y3y_1=y_2=y_3 in R3\mathbb{R}^3, of codimension 22. So the map x↦(f1−f2,f2−f3)x\mapsto(f_1-f_2,f_2-f_3) from the plane to R2\mathbb{R}^2 is a local diffeomorphism at pp, and in the coordinates (u,v)=(f1−f2,f2−f3)(u,v)=(f_1-f_2,f_2-f_3) the decision regions are {u>0, u+v>0}\{u>0,\ u+v>0\}, {u<0, v>0}\{u<0,\ v>0\} and {v<0, u+v<0}\{v<0,\ u+v<0\}: three sectors bounded by the rays {u=0,v>0}\{u=0,v>0\}, {v=0,u>0}\{v=0,u>0\} and {u+v=0,u<0}\{u+v=0,u<0\}.

Chapter 5

5.1. g=(x12−1)2+x22g=(x_1^2-1)^2+x_2^2 has ∇g=(4x1(x12−1),2x2)\nabla g=(4x_1(x_1^2-1),2x_2), vanishing at (0,0)(0,0) and (±1,0)(\pm1,0). The Hessian is diag⁡(12x12−4, 2)\operatorname{diag}(12x_1^2-4,\,2): at (±1,0)(\pm1,0) it is diag⁡(8,2)\operatorname{diag}(8,2), index 00, value 00; at the origin diag⁡(−4,2)\operatorname{diag}(-4,2), index 11, value 11. The sublevel set {g≤t}\{g\le t\} is empty for t<0t<0, two discs around (±1,0)(\pm1,0) for 0<t<10<t<1 (on the line x2=0x_2=0 it is {∣x12−1∣≤t}\{|x_1^2-1|\le\sqrt t\}, two intervals when t<1t<1), and one disc for t>1t>1. Its Euler characteristic is 0,2,10,2,1, consistent with attaching two 00-cells and then one 11-cell.

5.2. Here BtB_t is a finite set and WγW_\gamma is a compact 11-manifold with boundary B0⊔B1B_0\sqcup B_1. A critical point of τ\tau on a 11-manifold is a local minimum or maximum of tt along a curve: two points of BtB_t appear together or disappear together (surgery with λ=0\lambda=0 or 11 on a 00-manifold: add or remove an S0S^0). Between critical times the points move continuously. If XX is a circle, ftf_t changes sign at each point of BtB_t and returns to its original sign after one circuit, so #Bt\#B_t is even for every regular tt.

5.3. Two. By 2.3 with d=3d=3, the Euler characteristic of the region changes by ±(−1)k\pm(-1)^k at each surgery; the solid torus has χ=0\chi=0 and needs at least one 00-handle to be non-empty, so at least one more handle of odd index is needed. A 00-handle (a ball appears, k=0k=0) followed by a 11-handle attached to it (the ball becomes a solid torus, k=1k=1) suffices. The boundary goes from empty to a sphere to a torus.

5.4. Write Φ=Φγ\Phi=\Phi_\gamma and let t(x)t(x) be defined by Φ(t(x),x)=0\Phi(t(x),x)=0 near xix_i, with t(xi)=tit(x_i)=t_i. Differentiating once, Φt ∂jt+∂jΦ=0\Phi_t\,\partial_jt+\partial_j\Phi=0, so ∂jt(xi)=0\partial_jt(x_i)=0 because dxΦ=0d_x\Phi=0 there. Differentiating again and evaluating at xix_i, where the terms containing first derivatives of tt vanish, gives Φt ∂j∂kt+∂j∂kΦ=0\Phi_t\,\partial_j\partial_kt+\partial_j\partial_k\Phi=0. Hence ∇2t(xi)=−∇x2Φ(ti,xi)/Φt(ti,xi)\nabla^2t(x_i)=-\nabla^2_x\Phi(t_i,x_i)/\Phi_t(t_i,x_i).

Chapter 6

6.1. ev(θ)=Φθ\mathrm{ev}(\theta)=\Phi\theta with Φ\Phi of rank nn. Every yy is a regular value; My=θ∗+ker⁡ΦM_y=\theta^*+\ker\Phi is an affine subspace of dimension P−nP-n, where θ∗\theta^* is any solution, for instance the minimum-norm one ΦT(ΦΦT)−1y\Phi^{\mathsf T}(\Phi\Phi^{\mathsf T})^{-1}y. The Hessian is the constant matrix ΦTΦ\Phi^{\mathsf T}\Phi, of rank nn, with kernel ker⁡Φ\ker\Phi.

6.2. ∇L=(ab−1)(b,a)\nabla\mathcal{L}=(ab-1)(b,a), which vanishes when ab=1ab=1 or (a,b)=0(a,b)=0. The Hessian is (b22ab−12ab−1a2)\bigl(\begin{smallmatrix}b^2&2ab-1\\2ab-1&a^2\end{smallmatrix}\bigr). On ab=1ab=1 it is (b211a2)\bigl(\begin{smallmatrix}b^2&1\\1&a^2\end{smallmatrix}\bigr), of determinant a2b2−1=0a^2b^2-1=0 and trace a2+b2>0a^2+b^2>0: rank 11, with kernel spanned by (a,−b)(a,-b), the tangent to the hyperbola. At the origin it is (0−1−10)\bigl(\begin{smallmatrix}0&-1\\-1&0\end{smallmatrix}\bigr), with eigenvalues ±1\pm1: a saddle of index 11 with value 12\frac12. The sublevel set is {1−2c≤ab≤1+2c}\{1-\sqrt{2c}\le ab\le1+\sqrt{2c}\}. For c<12c<\frac12 the lower bound is positive, and the set lies in the two quadrants ab>0ab>0, one component in each. For c≥12c\ge\frac12 it contains the axes and is connected. Any path from (1,1)(1,1) to (−1,−1)(-1,-1) must cross the set ab=0ab=0, where L=12\mathcal{L}=\frac12.

6.3. The flow line γ\gamma has length at most ℓ=2L(θ0)/μ\ell=\sqrt{2\mathcal{L}(\theta_0)/\mu} while it stays in UU. If the closed ball of radius ℓ\ell about θ0\theta_0 lies in UU, the flow line cannot leave UU: at the first exit time it would be at distance at most its length, which is at most ℓ\ell, from θ0\theta_0, hence still inside. So the flow line is defined for all time, has finite length, and therefore converges; the limit θ∞\theta_\infty satisfies ∣θ∞−θ0∣≤ℓ|\theta_\infty-\theta_0|\le\ell and L(θ∞)=lim⁡L(γ(t))=0\mathcal{L}(\theta_\infty)=\lim \mathcal{L}(\gamma(t))=0.

6.4. L=12(∏ai−1)2\mathcal{L}=\frac12(\prod a_i-1)^2 and My={∏ai=1}M_y=\{\prod a_i=1\}, the preimage of the regular value 11 of the product map (its gradient at a point with all ai≠0a_i\ne0 is non-zero); so dim⁡My=k−1\dim M_y=k-1. On MyM_y every ai≠0a_i\ne0 and the number of negative aia_i is even; each admissible sign pattern gives one component (a graph over the positive orthant in the first k−1k-1 coordinates), and there are 2k−12^{k-1} sign patterns with an even number of minus signs. The gradient is (∏ai−1)(∏j≠iaj)i(\prod a_i-1)\bigl(\prod_{j\ne i}a_j\bigr)_i, which vanishes at the origin for k≥2k\ge2; the Hessian at the origin has entries −∂i∂j∏al-\partial_i\partial_j\prod a_l, which are zero for k≥3k\ge3. So the origin is a critical point with zero Hessian.

Chapter 7

7.1. Let φt\varphi^t be the flow. If x<yx<y then φt(x)≠φt(y)\varphi^t(x)\ne\varphi^t(y) for all tt by uniqueness of solutions, so φt(y)−φt(x)\varphi^t(y)-\varphi^t(x) is continuous in tt, never zero, and positive at t=0t=0; hence positive at t=1t=1.

7.2. Let ρ ⁣:[0,∞)→[0,1]\rho\colon[0,\infty)\to[0,1] be smooth with ρ=1\rho=1 on [0,2][0,2] and ρ=0\rho=0 on [3,∞)[3,\infty), and let v(x)=πρ(∣x∣) (−x2,x1)v(x)=\pi\rho(|x|)\,(-x_2,x_1). The flow preserves ∣x∣|x|, since vv is tangent to circles, and on {∣x∣≤2}\{|x|\le2\} it is rotation with angular speed π\pi. At time one the points (±1,0)(\pm1,0) have been rotated by the angle π\pi and are exchanged.

7.3. Let φt\varphi_t be the diffeotopy and Kt=φt(K)K_t=\varphi_t(K), Lt=φt(L)L_t=\varphi_t(L); they are disjoint for every tt because φt\varphi_t is injective. The Gauss maps Gt ⁣:K×L→S2G_t\colon K\times L\to S^2, Gt(x,y)=(φt(y)−φt(x))/∣φt(y)−φt(x)∣G_t(x,y)=(\varphi_t(y)-\varphi_t(x))/|\varphi_t(y)-\varphi_t(x)|, form a homotopy, and the degree is a homotopy invariant by [9, Thm. 7.2].

7.4. The first projection is t↦(cos⁡t,sin⁡t)t\mapsto(\cos t,\sin t), the standard embedding of the circle. The second is t↦(cos⁡2t,sin⁡2t)t\mapsto(\cos2t,\sin2t), which takes the same value at tt and t+πt+\pi; it is an immersion but not injective, a double cover of the circle.

Chapter 8

8.1. For fixed yy the map T(x)=y−g(x)T(x)=y-g(x) satisfies ∣T(x)−T(x′)∣≤L∣x−x′∣|T(x)-T(x')|\le L|x-x'|, so it has a unique fixed point x∗x^*, which is the unique solution of x+g(x)=yx+g(x)=y, and ∣xj−x∗∣=∣Tj(x0)−Tj(x∗)∣≤Lj∣x0−x∗∣|x_j-x^*|=|T^j(x_0)-T^j(x^*)|\le L^j|x_0-x^*|. So ff is a bijection. And ∣f(x)−f(x′)∣≥∣x−x′∣−∣g(x)−g(x′)∣≥(1−L)∣x−x′∣|f(x)-f(x')|\ge|x-x'|-|g(x)-g(x')|\ge(1-L)|x-x'|, which says that f−1f^{-1} is Lipschitz with constant (1−L)−1(1-L)^{-1}.

8.2. The singular values of 1+Dg1+Dg are at least 1−L1-L, since ∣(1+Dg)v∣≥∣v∣−∣Dg v∣≥(1−L)∣v∣|(1+Dg)v|\ge|v|-|Dg\,v|\ge(1-L)|v|. The absolute value of the determinant is the product of the singular values, so it is at least (1−L)d(1-L)^d; and the determinant is positive by 2.2(b).

8.3. z=0z=0 is always a fixed point, with 1−g′(0)=1−a1-g'(0)=1-a. For a≤1a\le1 it is the only one: tanh⁡(az)\tanh(az) is strictly concave for z>0z>0 with slope a≤1a\le1 at 00, so tanh⁡(az)<z\tanh(az)<z for z>0z>0, and symmetrically for z<0z<0. For a>1a>1, tanh⁡(az)>z\tanh(az)>z for small z>0z>0 and tanh⁡(az)<1<z\tanh(az)<1<z for z>1z>1, so there is a fixed point z∗∈(0,1)z^*\in(0,1), unique by concavity, and −z∗-z^* by symmetry. At z∗z^* the graph of tanh⁡(az)\tanh(az) crosses the diagonal from above, so g′(z∗)<1g'(z^*)<1 and the sign is ++; at 00 the sign of 1−a1-a is −-. The signed count is 1−1+1=11-1+1=1. For a<1a<1 it is the single +1+1. At a=1a=1 the fixed point 00 is degenerate (1−g′(0)=01-g'(0)=0): this is the exceptional parameter at which the pair ±z∗\pm z^* is born, a pitchfork, which is not a generic birth–death because of the symmetry z↦−zz\mapsto-z present when u=0u=0.

8.4. A point yy lies in supp⁡(h∗μ)\operatorname{supp}(h_*\mu) if and only if every open neighbourhood VV of yy has μ(h−1(V))>0\mu(h^{-1}(V))>0. As hh is a homeomorphism, the sets h−1(V)h^{-1}(V) are exactly the open neighbourhoods of h−1(y)h^{-1}(y); so the condition is h−1(y)∈supp⁡μh^{-1}(y)\in\operatorname{supp}\mu. If h∗μ=νh_*\mu=\nu then hh restricts to a homeomorphism from supp⁡μ\operatorname{supp}\mu to supp⁡ν\operatorname{supp}\nu.

References

References

  1. J. Behrmann, W. Grathwohl, R. T. Q. Chen, D. Duvenaud, J.-H. Jacobsen, Invertible residual networks, in ICML 2019, PMLR 97, 573–582.

  2. M. Bianchini, F. Scarselli, On the complexity of neural network classifiers: a comparison between shallow and deep architectures, IEEE Trans. Neural Netw. Learn. Syst. 25 (2014), 1553–1565.

  3. R. Cornish, A. Caterini, G. Deligiannidis, A. Doucet, Relaxing bijectivity constraints with continuously indexed normalising flows, in ICML 2020, PMLR 119.

  4. R. Cockett, G. Cruttwell, J. Gallagher, J.-S. P. Lemay, B. MacAdam, G. Plotkin, D. Pronk, Reverse derivative categories, in Computer Science Logic 2020, LIPIcs 152, 18:1–18:16.

  5. G. S. H. Cruttwell, B. Gavranović, N. Ghani, P. Wilson, F. Zanasi, Categorical foundations of gradient-based learning, in ESOP 2022, Lecture Notes in Computer Science 13240, Springer, 1–28.

  6. Y. Cooper, Global minima of overparameterized neural networks, SIAM J. Math. Data Sci. 3 (2021), 676–691.

  7. R. T. Q. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud, Neural ordinary differential equations, in NeurIPS 2018.

  8. E. Dupont, A. Doucet, Y. W. Teh, Augmented neural ODEs, in NeurIPS 2019.

  9. Anonymous, Differential Topology in Higher Categories, Higher Categorical Semantics, 2026. Companion volume.

  10. F. Draxler, K. Veschgini, M. Salmhofer, F. Hamprecht, Essentially no barriers in neural network energy landscape, in ICML 2018, PMLR 80.

  11. R. Entezari, H. Sedghi, O. Saukh, B. Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, in ICLR 2022.

  12. K. Fukumizu, S. Amari, Local minima and plateaus in hierarchical structures of multilayer perceptrons, Neural Networks 13 (2000), 317–327.

  13. C. Fefferman, Reconstructing a neural net from its output, Rev. Mat. Iberoamericana 10 (1994), 507–555.

  14. C. Fefferman, S. Mitter, H. Narayanan, Testing the manifold hypothesis, J. Amer. Math. Soc. 29 (2016), 983–1049.

  15. B. Fong, D. I. Spivak, R. Tuyéras, Backprop as functor: a compositional perspective on supervised learning, in LICS 2019, IEEE, 2019.

  16. K. Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network, Neural Networks 9 (1996), 871–879.

  17. T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, A. G. Wilson, Loss surfaces, mode connectivity, and fast ensembling of DNNs, in NeurIPS 2018.

  18. J. E. Grigsby, K. Lindsey, On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks, SIAM J. Appl. Algebra Geom. 6 (2022), 216–242.

  19. J. E. Grigsby, K. Lindsey, R. Meyerhoff, C. Wu, Functional dimension of feedforward ReLU neural networks, preprint, arXiv:2209.04036.

  20. M. W. Hirsch, Differential Topology, Graduate Texts in Mathematics 33, Springer, 1976.

  21. K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks 4 (1991), 251–257.

  22. B. Hanin, M. Sellke, Approximating continuous functions by ReLU nets of minimal width, preprint, arXiv:1710.11278.

  23. A. Jacot, F. Gabriel, C. Hongler, Neural tangent kernel: convergence and generalization in neural networks, in NeurIPS 2018.

  24. J. Johnson, Deep, skinny neural networks are not universal approximators, in ICLR 2019.

  25. C. Liu, L. Zhu, M. Belkin, Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, Appl. Comput. Harmon. Anal. 59 (2022), 85–116.

  26. Q. Nguyen, On connected sublevel sets in deep learning, in ICML 2019, PMLR 97.

  27. G. Naitzat, A. Zhitnikov, L.-H. Lim, Topology of deep neural networks, J. Mach. Learn. Res. 21 (2020), no. 184, 1–40.

  28. M. Phuong, C. H. Lampert, Functional vs. parametric equivalence of ReLU networks, in ICLR 2020.

  29. B. Şimşek, F. Ged, A. Jacot, F. Spadaro, C. Hongler, W. Gerstner, J. Brea, Geometry of the loss landscape in overparameterized neural networks: symmetries and invariances, in ICML 2021, PMLR 139.

  30. H. J. Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, Neural Networks 5 (1992), 589–593.

  31. S. Watanabe, Algebraic Geometry and Statistical Learning Theory, Cambridge University Press, 2009.

  32. H. Zhang, X. Gao, J. Unterman, T. Arodz, Approximation capabilities of neural ODEs and invertible residual networks, in ICML 2020, PMLR 119.

Complete flattened LaTeX source
\documentclass[11pt,openany]{book}
\usepackage[T1]{fontenc}

\usepackage[utf8]{inputenc}

\usepackage[sc,osf]{mathpazo}

\linespread{1.05}
\usepackage[scaled=0.9]{helvet}

\usepackage[paperwidth=7in,paperheight=10in,inner=0.95in,outer=0.85in,top=0.95in,bottom=1.0in,headsep=14pt]{geometry}

\usepackage{amsmath,amssymb,amsthm,mathtools,stmaryrd}

\usepackage{microtype}

\usepackage{xcolor}

\definecolor{accent}{HTML}{7A1F2B}
\definecolor{ink2}{HTML}{4A5560}
\definecolor{fwd}{HTML}{1F4E79}
\definecolor{bwd}{HTML}{A8432C}
\usepackage{tikz,tikz-cd}

\usetikzlibrary{arrows.meta,positioning,calc,decorations.markings,shapes.geometric,backgrounds,fit}
\usepackage{booktabs,array,tabularx,longtable}

\usepackage{enumitem}

\setlist{itemsep=2pt,topsep=4pt}
\usepackage{titlesec}

\usepackage{fancyhdr}

\usepackage[font=small,labelfont={bf,sf},margin=1.5em]{caption}

\usepackage[colorlinks=true,linkcolor=accent,citecolor=accent,urlcolor=accent,
  pdftitle={Parametrized Neural Networks and Their Representations in Differential Topology},
  pdfsubject={Differential topology of parametrized families of maps, applied to neural networks},
  pdfkeywords={neural networks, transversality, decision boundaries, loss landscapes, isotopy, neural ODEs, symmetries, differential topology}]{hyperref}

\usepackage[nameinlink,capitalise,noabbrev]{cleveref}


\titleformat{\chapter}[display]{\normalfont\Large\scshape\color{accent}}{\chaptertitlename\ \thechapter}{4pt}{\normalfont\huge\bfseries\color{black}}
\titlespacing*{\chapter}{0pt}{10pt}{26pt}
\titleformat{\section}{\normalfont\large\bfseries}{\thesection}{0.7em}{}
\titleformat{\subsection}{\normalfont\normalsize\bfseries\itshape}{\thesubsection}{0.7em}{}
\titleformat{\part}[display]{\normalfont\centering\Large\scshape\color{accent}}{\partname\ \thepart}{12pt}{\normalfont\Huge\bfseries\color{black}}
\pagestyle{fancy}\fancyhf{}
\fancyhead[LE]{\small\thepage\quad\textsc{\nouppercase{\leftmark}}}
\fancyhead[RO]{\small\textsc{\nouppercase{\rightmark}}\quad\thepage}
\renewcommand{\headrulewidth}{0pt}
\renewcommand{\chaptermark}[1]{\markboth{#1}{#1}}
\renewcommand{\sectionmark}[1]{\markright{\thesection\ \ #1}}
\fancypagestyle{plain}{\fancyhf{}\fancyfoot[C]{\small\thepage}}
\setlength{\headheight}{14pt}
\setcounter{tocdepth}{1}
\setcounter{secnumdepth}{2}
\raggedbottom
\emergencystretch=2em
\tolerance=1200
\hbadness=2000

\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[chapter]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{principle}[theorem]{Principle}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\newtheorem{construction}[theorem]{Construction}
\newtheorem{problem}[theorem]{Open Problem}
\newtheorem{exercise}{Exercise}[chapter]
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\crefname{principle}{Principle}{Principles}
\crefname{construction}{Construction}{Constructions}
\crefname{conjecture}{Conjecture}{Conjectures}
\crefname{exercise}{Exercise}{Exercises}
\crefname{problem}{Open Problem}{Open Problems}

\newcommand{\stag}[2]{{\footnotesize\sffamily\upshape\color{#1}[#2]}\enspace\ignorespaces}
\newcommand{\Scl}[1][]{\stag{ink2}{classical\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Sli}[1][]{\stag{fwd}{literature\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Srf}[1][]{\stag{accent}{reformulation\if\relax\detokenize{#1}\relax\else; cf.\ #1\fi}}
\newcommand{\Sob}{\stag{accent}{observation}}
\newcommand{\Spg}{\stag{bwd}{programme}}

\newcommand{\cat}[1]{\mathsf{#1}}
\newcommand{\Set}{\cat{Set}}
\newcommand{\FinSet}{\cat{FinSet}}
\newcommand{\Meas}{\cat{Meas}}
\newcommand{\Stoch}{\cat{Stoch}}
\newcommand{\FinStoch}{\cat{FinStoch}}
\newcommand{\BorelStoch}{\cat{BorelStoch}}
\newcommand{\FinProb}{\cat{FinProb}}
\newcommand{\FinStat}{\cat{FinStat}}
\newcommand{\Krn}{\cat{Krn}}
\newcommand{\Hilb}{\cat{Hilb}}
\newcommand{\Ban}{\cat{Ban}}
\newcommand{\Src}{\cat{Src}}
\newcommand{\Chan}{\cat{Chan}}
\newcommand{\Dyn}{\cat{Dyn}}
\newcommand{\Cat}{\cat{Cat}}
\newcommand{\CC}{\mathcal{C}}
\newcommand{\FF}{\mathcal{F}}
\newcommand{\GG}{\mathcal{G}}
\newcommand{\II}{\mathcal{I}}
\newcommand{\Sinf}{\mathcal{S}}
\newcommand{\Qlaw}{\mathcal{Q}}
\newcommand{\Fun}{\mathcal{F}}
\newcommand{\Dist}{\Delta}
\newcommand{\Giry}{\mathcal{G}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Rbar}{\overline{\mathbb{R}}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\Fp}{\mathbb{F}_p}
\newcommand{\E}{\mathbb{E}}
\newcommand{\Prob}{\mathbb{P}}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\colim}{colim}
\DeclareMathOperator{\Hom}{Hom}
\DeclareMathOperator{\Ext}{Ext}
\DeclareMathOperator{\Fix}{Fix}
\DeclareMathOperator{\supp}{supp}
\DeclareMathOperator{\TV}{TV}
\DeclareMathOperator{\rk}{rk}
\newcommand{\id}{\mathrm{id}}
\newcommand{\op}{\mathrm{op}}
\newcommand{\adj}{\dashv}
\newcommand{\iso}{\cong}
\newcommand{\st}{\mid}
\newcommand{\cpy}{\mathrm{copy}}
\newcommand{\del}{\mathrm{del}}
\newcommand{\KL}[2]{D(#1\,\|\,#2)}
\newcommand{\hb}{h}
\newcommand{\AZ}{A^{\Z}}
\newcommand{\BZ}{B^{\Z}}
\newcommand{\term}[1]{\textbf{\itshape #1}}

\newcommand{\Mfd}{\cat{Mfd}}
\newcommand{\Cart}{\cat{Cart}}
\newcommand{\SmSet}{\cat{SmSet}}
\newcommand{\Dflg}{\cat{Dflg}}
\newcommand{\CinfRing}{C^\infty\cat{Ring}}
\newcommand{\DMfd}{\cat{DMfd}}
\newcommand{\Spc}{\mathcal{S}}
\newcommand{\Sp}{\cat{Sp}}
\newcommand{\Cob}{\cat{Cob}}
\newcommand{\Bord}{\cat{Bord}}
\newcommand{\Vect}{\cat{Vect}}
\newcommand{\Top}{\cat{Top}}
\newcommand{\Para}{\cat{Para}}
\DeclareMathOperator{\Diff}{Diff}
\DeclareMathOperator{\Emb}{Emb}
\DeclareMathOperator{\Imm}{Imm}
\DeclareMathOperator{\Map}{Map}
\DeclareMathOperator{\Sing}{Sing}
\DeclareMathOperator{\Fr}{Fr}
\DeclareMathOperator{\Th}{Th}
\DeclareMathOperator{\Gr}{Gr}
\DeclareMathOperator{\GL}{GL}
\DeclareMathOperator{\ind}{ind}
\DeclareMathOperator{\codim}{codim}
\DeclareMathOperator{\vdim}{vdim}
\DeclareMathOperator{\tr}{tr}
\DeclareMathOperator{\grad}{grad}
\DeclareMathOperator{\Tub}{Tub}
\DeclareMathOperator{\Conf}{Conf}
\DeclareMathOperator{\diag}{diag}
\DeclareMathOperator{\relu}{ReLU}
\newcommand{\Or}{\mathrm{O}}
\newcommand{\SO}{\mathrm{SO}}
\newcommand{\RP}{\mathbb{RP}}
\newcommand{\CP}{\mathbb{CP}}
\newcommand{\C}{\mathbb{C}}
\newcommand{\trans}{\pitchfork}
\newcommand{\pt}{\mathrm{pt}}
\newcommand{\loss}{\mathcal{L}}

\tikzset{
  ogbox/.style={draw, thick, rounded corners=2pt, minimum width=1.25cm, minimum height=1.05cm, fill=black!4},
  fw/.style={-{Stealth[length=5pt]}, thick, fwd},
  bw/.style={-{Stealth[length=5pt]}, thick, bwd, densely dashed},
  dec/.style={circle, fill, inner sep=1.7pt},
  nat/.style={circle, draw, thick, fill=white, inner sep=1.7pt},
  leaf/.style={inner sep=2pt, font=\small},
  lab/.style={font=\small, inner sep=1.5pt},
  infoset/.style={densely dashed, thick, accent},
  st/.style={circle, draw, thick, fill=white, inner sep=1.6pt},
  ty/.style={rectangle, draw, thick, fill=black!8, inner sep=2.6pt},
}
\newcommand{\sol}[1]{\par\medskip\noindent\textbf{#1.}\enspace\ignorespaces}
\newcommand{\bookauthor}{Anonymous}
\newcommand{\DT}[1]{\cite[#1]{DTHC}}
\newcommand{\ev}{\mathrm{ev}}
\begin{document}
\frontmatter
\begin{titlepage}
\thispagestyle{empty}
\vspace*{1.1in}
\noindent{\Large\scshape\color{accent} Higher Categorical Semantics}\par
\vspace{0.9in}
\noindent{\fontsize{28}{36}\selectfont\bfseries\raggedright Parametrized Neural\\ Networks and Their\\ Representations in\\ Differential Topology\par}
\vspace{0.35in}
\noindent{\Large\itshape Families of maps, transversality,\\[2pt] surgery along training, and isotopy}\par
\vfill
\noindent\begin{tikzpicture}[x=1cm,y=1cm]
  \node[ogbox] (th) at (0,0.7) {$\Theta$};
  \node[ogbox] (x) at (0,-0.7) {$X$};
  \node[ogbox,minimum width=1.6cm] (F) at (2.6,0) {$F$};
  \node[lab] (y) at (5.0,0) {$Y$};
  \draw[fw] (th) -- (F);
  \draw[fw] (x) -- (F);
  \draw[fw] (F) -- (y);
  \node[lab] at (6.1,0) {$=$};
  \node[lab] (t2) at (7.2,0) {$\Theta$};
  \node[lab] (m) at (9.6,0) {$[X,Y]$};
  \draw[fw,black] (t2) -- (m);
\end{tikzpicture}\par
\vspace{0.5in}
\noindent{\large \bookauthor}\par
\vspace{4pt}
\noindent{\small 2026}
\end{titlepage}

\thispagestyle{empty}
\vspace*{\fill}
{\small
\noindent\textbf{About this text.}
This monograph applies the results of \emph{Differential Topology in Higher Categories} \cite{DTHC}, the
higher-categorical treatment of M.~W.~Hirsch's \emph{Differential Topology} \cite{Hir76} in the same series, to smooth
families of maps defined by neural network architectures. It is written to be read after that volume and
cites it by theorem number, in the form \DT{4.5}. It is an independent work and is not affiliated with or
endorsed by the publisher of Hirsch's book.

\medskip
\noindent\textbf{Status of results.}
Every numbered statement carries a tag: \textsf{classical} (standard mathematics, attributed),
\textsf{literature} (established in the machine learning or mathematical literature, attributed),
\textsf{reformulation} (a known result restated or specialized, with the derivation given here),
\textsf{observation} (an elementary consequence we have not found stated in this form), or
\textsf{programme} (a direction, not a result). The statements tagged \textsf{reformulation} and
\textsf{observation} are corollaries of classical differential topology applied to parametrized families;
none is claimed as a new theorem of differential topology, and none says anything quantitative about
training or generalization.

\medskip
\noindent\textbf{Authorship and assistance.}
The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of
the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall,
mathematical consistency checks, computation, and typesetting. The numerical examples and finite instances
of the structural claims are recomputed by the accompanying script \texttt{verify-parametrized-networks.py}
(\cref{app:computations}), which uses small networks implemented in pure Python. Proofs and bibliographic
details have not been independently refereed; corrections are welcome.

\medskip
\noindent Open access. Rights remain with the author.
}
\clearpage

\tableofcontents

\chapter{Preface}

A neural network architecture is a smooth map $F\colon\Theta\times X\to Y$: a family of maps from inputs to
outputs, parametrized by weights. Differential topology, in the form Hirsch gave it, is the study of what is
true of a generic member of a family of maps and of what happens along a generic path in one. The fit is
close enough that most of the qualitative questions asked about networks have answers that are corollaries
of theorems proved before 1976, once the questions are phrased as questions about families.

Is the decision boundary of a classifier a manifold? For almost every value of the parameters, yes, by the
parametric transversality theorem, provided the last layer has a bias. How does the boundary change during
training? By ambient isotopy, except at isolated times, at each of which it undergoes one surgery whose type
is the index of a critical point; this is Morse theory on the trace of the boundary. What is the set of
parameters that fit the data exactly? For almost every labelling, a submanifold of dimension equal to the
number of parameters minus the number of constraints, by the regular value theorem. Can a neural ordinary
differential equation separate two linked circles? Not in dimension three, because its time-one map is
ambient isotopic to the identity; in dimension four always, because there any two embeddings of a
one-manifold are isotopic and isotopies extend. Each of these is a paragraph of argument on top of the
companion volume, and this book supplies the paragraphs.

The higher-categorical layer of the companion volume enters in three places. A parametrized network is a
plot of the mapping object $[X,Y]$ in smooth sets, and composition of layers is composition in a category of
parametrized maps; backpropagation is the functoriality of the reverse derivative. The symmetries of an
architecture make the parameter space an atlas for a stack whose points are networks up to relabelling,
and the loss is a function on the stack. And the level sets that fail to be manifolds, at the exceptional
parameters, are derived manifolds whose cobordism classes are still defined; the surgery description is
what one sees of them from nearby.

The results are qualitative and the last chapter is frank about what that leaves out. Nothing here bounds a
rate of convergence or a generalization error. Activation functions are assumed smooth except where stated,
and the piecewise-linear case, which needs a different transversality theory, is treated only by reference.
What the book offers is a dictionary, a set of theorems with honest hypotheses, and a script that checks
the examples.

\chapter{Introduction}

\section*{What is assumed}

The companion volume \cite{DTHC}, at least Chapters 1, 3, 4, 8 and 10, or the corresponding chapters of
Hirsch's book. From machine learning, only the definition of a feedforward network and of gradient descent.

\section*{How the book is organized}

Part~I sets up families, their derivatives and their symmetries. Part~II studies level sets: of the network
in input space (decision boundaries), of the network along a path of parameters (surgery), and of the loss
in parameter space (the interpolation manifold). Part~III studies the maps themselves: when they are
embeddings, when they are isotopic, when they are invertible. Part~IV is an assessment.
\Cref{app:dictionary} gives the dictionary between the two subjects in one table.

\section*{Conventions}

Manifolds and maps are smooth. A \emph{family} is a smooth map $F\colon\Theta\times X\to Y$; we write
$f_\theta=F(\theta,-)$. $P=\dim\Theta$ is the number of parameters, $d=\dim X$ the input dimension, $n$ the
number of data points. ``Almost every'' refers to Lebesgue measure on a Euclidean parameter space, and
``generic'' means belonging to a residual set whose complement has measure zero. An activation function
$\sigma\colon\R\to\R$ is smooth unless it is explicitly $\relu(s)=\max(s,0)$. Citations of the form
\DT{4.5} refer to numbered statements of the companion volume.
\mainmatter
\part{Families}
\chapter{Families of Maps}\label{ch:families}

\section{Parametrized maps}

\begin{definition}\label{def:family}
A \term{parametrized map} from $X$ to $Y$ is a pair $(\Theta,F)$ of a manifold $\Theta$ and a smooth map
$F\colon\Theta\times X\to Y$. Its \term{members} are the maps $f_\theta=F(\theta,-)$, and its
\term{realization map} is $\theta\mapsto f_\theta$, from $\Theta$ to $C^\infty(X,Y)$.
\end{definition}

By \DT{Thm.~1.6} a parametrized map is exactly a plot of the mapping object $[X,Y]$ in smooth sets, with
domain $\Theta$; the realization map is the function it induces on points. Everything in this book is a
statement about plots of mapping objects. The topologies on $C^\infty(X,Y)$ play no direct role, since
parameter spaces are finite-dimensional; what is used is the relation between properties of the total map
$F$ and properties of its members.

\begin{definition}\label{def:para}
$\Para$ is the bicategory whose objects are manifolds, whose morphisms $X\to Y$ are parametrized maps, and
whose $2$-cells $(\Theta',F')\Rightarrow(\Theta,F)$ are smooth maps $r\colon\Theta'\to\Theta$ with
$F'=F\circ(r\times\id_X)$. The composite of $(\Theta,F)\colon X\to Y$ and $(\Theta',G)\colon Y\to Z$ is
\[\bigl(\Theta\times\Theta',\ (\theta,\theta',x)\mapsto G(\theta',F(\theta,x))\bigr).\]
\end{definition}

Composition is associative up to the canonical reassociation of products, and the identity of $X$ is
parametrized by a point. The construction is due to Fong, Spivak and Tuy\'eras \cite{FST19} and is developed
in \cite{CGGWZ22}. A $2$-cell is a reparametrization: sharing weights between layers is the $2$-cell induced
by a diagonal map, and freezing a weight is the one induced by the inclusion of a slice.

\begin{proposition}\label{prop:plot}\Srf[\DT{Thm.~1.6}; \cite{FST19}]
Composition in $\Para$ is the internal composition
$[X,Y]\times[Y,Z]\to[X,Z]$ of smooth sets, evaluated on plots: the composite of two parametrized maps is the
plot $\Theta\times\Theta'\to[X,Y]\times[Y,Z]\to[X,Z]$. Hence $\Para$ is the bicategory of plots of the
category of manifolds enriched in smooth sets.
\end{proposition}

\begin{proof}
The internal composition is the map of sheaves which on plots with domain $U$ sends
$(F,G)$ to $(u,x)\mapsto G(u,F(u,x))$. Precomposing with the product plot, whose components depend on
separate factors of $U=\Theta\times\Theta'$, gives the formula of \cref{def:para}.
\end{proof}

\section{Architectures}

\begin{definition}\label{def:arch}
A \term{feedforward architecture} with widths $(n_0,\dots,n_L)$ and activation $\sigma$ is the composite in
$\Para$ of the layers
\begin{align*}
\ell_i&\colon\R^{n_{i-1}}\to\R^{n_i}, & \ell_i(W_i,b_i;x)&=\sigma(W_ix+b_i)\qquad(i<L),\\
\ell_L&\colon\R^{n_{L-1}}\to\R^{n_L}, & \ell_L(W_L,b_L;x)&=W_Lx+b_L,
\end{align*}
with $\sigma$ applied to each coordinate. Its parameter space is
$\Theta=\prod_i(\R^{n_i\times n_{i-1}}\times\R^{n_i})$, of dimension $P=\sum_in_i(n_{i-1}+1)$. The
number of hidden units is $H=n_1+\dots+n_{L-1}$.
\end{definition}

If $\sigma$ is smooth the family is smooth, and real-analytic if $\sigma$ is (as $\tanh$, the logistic
function and softplus are). For $\sigma=\relu$ the family is continuous and piecewise polynomial in
$(\theta,x)$, and it is not a morphism of $\Para$ as defined; statements about it are made separately.
Convolutional and attention layers are parametrized maps in the same sense, with $2$-cells expressing weight
sharing. Nothing below depends on the feedforward form except where an explicit hypothesis on the last or
first layer is made.

The image of the realization map, the set $\mathcal F=\{f_\theta\}$ of functions the architecture can
compute, is a subset of $C^\infty(X,Y)$. With the plots that factor locally through $\Theta$ it is a
diffeological space, the quotient of $\Theta$ by the relation $f_\theta=f_{\theta'}$. It is in general not a
manifold; \cref{ch:symmetry} describes its singularities.

\section{Data, loss, and training}

Fix inputs $x_1,\dots,x_n\in X$ and targets $y_1,\dots,y_n\in Y=\R^m$. The \term{evaluation map} is
\[\ev\colon\Theta\to Y^n=\R^{nm},\qquad\ev(\theta)=\bigl(f_\theta(x_1),\dots,f_\theta(x_n)\bigr),\]
and the squared loss is $\loss(\theta)=\tfrac12|\ev(\theta)-y|^2$. More generally a loss is
$\loss=\lambda\circ\ev$ for a smooth $\lambda\colon\R^{nm}\to\R$. A \term{training path} is a smooth curve
$\gamma\colon[0,T]\to\Theta$; gradient flow is the curve with $\dot\gamma=-\nabla\loss(\gamma)$, for the
Euclidean metric on $\Theta$. A path in $\Theta$ is a plot $[0,T]\to[X,Y]$: a one-parameter family of
maps. When every member is an embedding of a submanifold $K\subset X$ it is an isotopy of $K$ in the sense of
\DT{Def.~10.1}.

Three kinds of question organize the rest of the book.

\begin{enumerate}[label=(Q\arabic*)]
\item What does a generic member $f_\theta$ look like? (\cref{ch:decision,ch:isotopy,ch:degree})
\item What happens to the members along a generic path? (\cref{ch:paths})
\item What does the set of parameters satisfying a condition, such as $\loss(\theta)=0$, look like?
(\cref{ch:symmetry,ch:loss})
\end{enumerate}

In each case the answer comes from the same mechanism. A condition on a member, or on a parameter, is the
condition that a point lies in the preimage of a submanifold under a map built from $F$; the preimage is a
manifold when the map is transverse to the submanifold; and transversality of the map built from $F$ can be
checked on the architecture, usually by exhibiting one direction in parameter space, a bias, in which the
map moves freely.

\begin{principle}[Richness]\label{prin:rich}\Srf[\DT{Thm.~4.5}]
Let $F\colon\Theta\times X\to Y$ be a family and $A\subset Y$ a submanifold. If $F$ is transverse to $A$, in
particular if $F$ is a submersion, then $f_\theta$ is transverse to $A$ for almost every $\theta$. A
feedforward architecture is a submersion onto $Y=\R^{n_L}$ because of the last-layer bias:
$\partial F/\partial b_L=\id$.
\end{principle}

This is \DT{Thm.~4.5} with its hypothesis verified. It is worth stating separately because the hypothesis is
about the family and not about function space. A statement that a property is generic among all smooth maps
(for instance, that Morse functions are dense) says nothing about a given architecture, whose realization is
a thin subset of all maps. Genericity transfers to an architecture only through a transversality property of
$F$ itself, and each use of \cref{prin:rich} below names the parameters that provide it.

\section{The dictionary in brief}

\begin{center}\small
\begin{tabular}{@{}p{0.34\textwidth}p{0.42\textwidth}l@{}}
\toprule
network notion & object of differential topology & Ch.\\
\midrule
architecture & plot of a mapping object; morphism of $\Para$ & \ref{ch:families}\\
backpropagation & functoriality of the reverse derivative & \ref{ch:backprop}\\
weight-space symmetry & group action with invariant realization & \ref{ch:symmetry}\\
decision boundary & preimage of a regular value & \ref{ch:decision}\\
change of topology in training & handle attachment on the trace & \ref{ch:paths}\\
interpolating parameters & fibre of the evaluation map & \ref{ch:loss}\\
barrier between minima & death level in the sublevel barcode & \ref{ch:loss}\\
hidden representation of data & immersion or embedding of a manifold & \ref{ch:isotopy}\\
neural ODE, residual network & diffeotopy; isotopy of embeddings & \ref{ch:isotopy}\\
invertible network & diffeomorphism of degree one & \ref{ch:degree}\\
equilibrium of an implicit layer & transverse fixed point & \ref{ch:degree}\\
\bottomrule
\end{tabular}
\end{center}

\section*{Notes}

The categorical treatment of learners as parametrized maps with update and request functions begins with
\cite{FST19}; the bicategory $\Para$ and its combination with lenses and reverse derivatives is in
\cite{CGGWZ22}. The point of view that a network is a family in the sense of singularity theory, to which
transversality applies, is implicit in much work on loss landscapes, and explicit for piecewise-linear
networks in \cite{GL22}.

\section*{Exercises}

\begin{exercise}\label{exer:count}
Compute $P$ and $H$ for the architectures with widths $(2,3,1)$, $(2,4,4,1)$ and $(d,k,1)$.
\end{exercise}

\begin{exercise}\label{exer:paracomp}
Verify that composition in $\Para$ is associative up to the reassociation
$(\Theta\times\Theta')\times\Theta''\cong\Theta\times(\Theta'\times\Theta'')$, and that this isomorphism
is a $2$-cell in both directions.
\end{exercise}

\begin{exercise}\label{exer:sharing}
Let $(\Theta,F)\colon X\to X$ be a layer. Express the weight-tied network $f_\theta\circ f_\theta$ as the
source of a $2$-cell into the composite $(\Theta\times\Theta,F\circ F)$, and show that its derivative with
respect to $\theta$ is the sum of the two partial derivatives of the untied network on the diagonal.
\end{exercise}

\begin{exercise}\label{exer:notrich}
Give an example of a family $F\colon\R\times\R\to\R$ no member of which is a Morse function, although Morse
functions are dense in $C^\infty(\R,\R)$. Which hypothesis of \cref{prin:rich} fails for the relevant jet
condition?
\end{exercise}
\chapter{Derivatives}\label{ch:backprop}

Two derivatives of a family matter: with respect to the input, which governs the geometry of each member, and
with respect to the parameters, which governs training. Both are instances of the tangent functor of
\DT{Ch.~2}, and the algorithm that computes the second efficiently is the functoriality of its transpose.

\section{Backpropagation}

Recall from \DT{Def.~2.10} the category $\cat{Lens}$ and from \DT{Prop.~2.11} the functor
$f\mapsto(f,R[f])$, $R[f](x,w)=(Df_x)^{\mathsf T}w$, from Euclidean spaces to lenses. For a parametrized map
$F\colon\Theta\times X\to Y$ the reverse derivative has two components,
\[R[F](\theta,x,w)=\bigl(R_\Theta[F](\theta,x,w),\ R_X[F](\theta,x,w)\bigr)
=\bigl((D_\theta F)^{\mathsf T}w,\ (D_xF)^{\mathsf T}w\bigr),\]
the first being the update direction for the parameters and the second the covector passed back to the
previous layer.

\begin{theorem}[Backpropagation is a functor]\label{thm:backprop}
\Sli[Fong--Spivak--Tuy\'eras \cite{FST19}; Cruttwell et al.\ \cite{CGGWZ22}]
The assignment $(\Theta,F)\mapsto(\Theta,(F,R[F]))$ is a functor of bicategories from $\Para$ over Euclidean
spaces to parametrized lenses. In particular, for a composite of layers $F=\ell_L\circ\dots\circ\ell_1$ the
component $R_{\Theta_i}[F]$ is obtained by composing the lenses of the layers: a forward pass computing the
activations $x_i=\ell_i(\theta_i,x_{i-1})$, and a backward pass computing $w_{i-1}=R_X[\ell_i](\theta_i,
x_{i-1},w_i)$ and emitting $R_\Theta[\ell_i](\theta_i,x_{i-1},w_i)$.
\end{theorem}

\begin{proof}
Composition in $\Para$ is composition of maps after taking products with identities, and $R$ is a
product-preserving functor to lenses by \DT{Prop.~2.11}.
\end{proof}

The gradient of a loss $\loss=\lambda\circ\ev$ is $R[\ev](\theta,\nabla\lambda)$. The theorem says the
cost of the gradient is that of one forward and one backward pass, whatever the number of parameters, which
is the practical content of reverse-mode differentiation. The script implements both modes for a small
network and checks them against each other and against finite differences.

\section{The Jacobian of the evaluation map}

\begin{definition}\label{def:jac}
For data $x_1,\dots,x_n$ the \term{parameter Jacobian} is $J(\theta)=D\ev_\theta$, an $nm\times P$ matrix.
The \term{tangent kernel} is the $nm\times nm$ matrix $K(\theta)=J(\theta)J(\theta)^{\mathsf T}$.
\end{definition}

\begin{proposition}\label{prop:ntk}\Scl[Jacot--Gabriel--Hongler \cite{JGH18}]
$K(\theta)$ is symmetric positive semidefinite of rank $\rk J(\theta)$. Along gradient flow of
$\loss=\lambda\circ\ev$ the outputs $u(t)=\ev(\gamma(t))$ satisfy $\dot u=-K(\gamma)\nabla\lambda(u)$.
\end{proposition}

\begin{proof}
$\dot u=J\dot\gamma=-J\nabla\loss=-JJ^{\mathsf T}\nabla\lambda$.
\end{proof}

So the tangent kernel is the Riemannian metric that the Euclidean metric of $\Theta$ induces, by pushing
forward along $\ev$, on the cotangent directions of output space; where $J$ is onto, $K$ is invertible and
the outputs follow a gradient flow for the metric $K^{-1}$.

\section{Rank}

The rank of $J$ is the first invariant of a parameter. It is lower semicontinuous, and for analytic
architectures its generic value is attained almost everywhere.

\begin{proposition}\label{prop:rank}\Scl
Let $F$ be real-analytic in $\theta$ on a connected $\Theta$, and let $r$ be the maximum of $\rk J(\theta)$.
Then $\{\theta:\rk J(\theta)=r\}$ is open and dense and its complement has measure zero. On it the
fibres of $\ev$ are submanifolds of dimension $P-r$, and $\ev$ is locally a submersion onto an
$r$-dimensional submanifold of $\R^{nm}$.
\end{proposition}

\begin{proof}
The complement is the common zero set of the $r\times r$ minors of $J$, which are analytic and not all
identically zero. The zero set of a non-zero analytic function on a connected domain is closed with empty
interior and measure zero. The rest is the constant rank theorem.
\end{proof}

\begin{definition}\label{def:fdim}
The \term{functional dimension} of the architecture at $\theta$ is the supremum over all finite sets of
inputs of $\rk J(\theta)$: the rank of the differential of the realization map \cite{GLMW22}.
\end{definition}

The functional dimension is at most $P$, and \cref{ch:symmetry} shows when it is smaller. The strata
$\{\rk J=s\}$ for $s<r$ are where the fibres of $\ev$ jump in dimension. They contain the parameters at
which hidden units are redundant, and they are responsible for most of what is singular in the loss
landscape (\cref{ch:loss}).

\section{Second derivatives}

By the composition law for $2$-jets \DT{Prop.~2.9}, the Hessian of a composite $\lambda\circ\ev$ is
\[\nabla^2\loss(\theta)=J^{\mathsf T}\,\nabla^2\lambda\,J+\sum_k\frac{\partial\lambda}{\partial u_k}\,
\nabla^2\ev_k(\theta),\]
a Gauss--Newton term and a term weighted by the gradient of $\lambda$ in output space.

\begin{proposition}\label{prop:hess}\Scl
At a parameter where $\nabla\lambda(\ev(\theta))=0$, in particular at a zero of the squared loss,
$\nabla^2\loss=J^{\mathsf T}\nabla^2\lambda\,J$. For the squared loss this is $J^{\mathsf T}J$, positive
semidefinite of rank $\rk J$, with kernel $\ker J$.
\end{proposition}

The non-zero eigenvalues of $J^{\mathsf T}J$ and of the tangent kernel $JJ^{\mathsf T}$ coincide. So at an
interpolating parameter the curvature of the loss in the directions normal to the solution set is the
spectrum of the tangent kernel.

\section*{Notes}

Reverse-mode differentiation has been rediscovered several times; Griewank's history is the standard
reference. The compositional formulation is from \cite{FST19} and \cite{CGGWZ22}, and the axiomatics of
reverse derivatives from \cite{CCG20}. The tangent kernel was introduced in \cite{JGH18}. Functional
dimension for $\relu$ networks is studied in \cite{GLMW22}.

\section*{Exercises}

\begin{exercise}\label{exer:bp}
Write out the lens of a layer $\ell(W,b;x)=\sigma(Wx+b)$, and compose two of them to obtain the gradient of
$\tfrac12(a^{\mathsf T}\sigma(Wx+b)+c-y)^2$ with respect to $(W,b,a,c)$.
\end{exercise}

\begin{exercise}\label{exer:ntk}
Show that $\rk K(\theta)=\rk J(\theta)$ and that, for $n$ data points and scalar output, $K(\theta)$ is
invertible if and only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, restricted
to the data, span $\R^n$.
\end{exercise}

\begin{exercise}\label{exer:hess}
Derive the formula for $\nabla^2(\lambda\circ\ev)$ from the composition of $2$-jets.
\end{exercise}

\begin{exercise}\label{exer:linear}
For the linear family $F(\theta,x)=\langle\theta,\phi(x)\rangle$ with a fixed feature map
$\phi\colon X\to\R^P$, compute $J$, $K$, the fibres of $\ev$, and the functional dimension.
\end{exercise}
\chapter{Symmetries and the Moduli of Networks}\label{ch:symmetry}

The realization map $\theta\mapsto f_\theta$ is never injective. Its fibres are the first thing to understand
about an architecture, since every function on parameter space that is defined in terms of the network's
input--output behaviour, such as the loss, is constant on them. This chapter describes the fibres for
networks with one hidden layer, where the theory is complete, and draws the consequences that hold in
general.

\section{Symmetries}

\begin{definition}\label{def:sym}
A \term{symmetry} of a family $F\colon\Theta\times X\to Y$ is a diffeomorphism $g$ of $\Theta$ with
$F(g\theta,x)=F(\theta,x)$ for all $\theta,x$: an invertible $2$-cell of $(\Theta,F)$ in $\Para$.
\end{definition}

For a hidden unit $j$ in layer $i$ of a feedforward architecture, call its \term{incoming} parameters the
$j$-th row of $W_i$ and the $j$-th entry of $b_i$, and its \term{outgoing} parameters the $j$-th column of
$W_{i+1}$.

\begin{proposition}\label{prop:syms}\Scl
Every feedforward architecture has the following symmetries.
\begin{enumerate}[label=(\roman*)]
\item Permutations of the hidden units of a layer, acting on incoming and outgoing parameters together.
\item If $\sigma$ is odd: for each hidden unit, the sign change of its incoming and outgoing parameters.
\item If $\sigma$ is positively homogeneous of degree one, as $\relu$ is: for each hidden unit and
$\lambda>0$, multiplication of the incoming parameters by $\lambda$ and of the outgoing ones by
$\lambda^{-1}$.
\end{enumerate}
For $\sigma=\tanh$ these generate a finite group $G$ of order $\prod_{i=1}^{L-1}2^{n_i}n_i!$.
\end{proposition}

\section{One hidden layer}

Consider $F(\theta,x)=\sum_{j=1}^Ha_j\,\sigma(\langle w_j,x\rangle+b_j)+c$ on $X=\R^d$, with
$\theta=(w_j,b_j,a_j)_{j\le H}$ and $c$, so $P=H(d+2)+1$, and $\sigma=\tanh$.

\begin{definition}\label{def:irr}
The parameter $\theta$ is \term{irreducible} if every $a_j\ne0$, every $w_j\ne0$, and
$(w_j,b_j)\ne\pm(w_k,b_k)$ for $j\ne k$. The set of irreducible parameters is $\Theta^{\mathrm{irr}}$.
\end{definition}

\begin{theorem}[Identifiability]\label{thm:ident}\Sli[Sussmann \cite{Sus92}; Fukumizu \cite{Fuk96}]
Let $\theta$ be irreducible. Then $f_{\theta'}=f_\theta$ for a parameter $\theta'$ with the same number of
hidden units if and only if $\theta'=g\theta$ for some $g\in G$. Moreover $\theta$ is irreducible if and
only if the functions $\partial F/\partial\theta_k(\theta,\cdot)$, $k=1,\dots,P$, are linearly independent;
so the functional dimension is $P$ exactly on $\Theta^{\mathrm{irr}}$.
\end{theorem}

For deep networks with $\tanh$ the corresponding statement holds for generic parameters, by a theorem of
Fefferman \cite{Fef94}; for $\relu$ networks see \cite{PL20}. The complement of $\Theta^{\mathrm{irr}}$ is a
finite union of proper linear subspaces, so $\Theta^{\mathrm{irr}}$ is open and dense, and $G$ acts freely on
it.

\begin{proposition}[The moduli of irreducible networks]\label{prop:moduli}\Sob
The quotient $\Theta^{\mathrm{irr}}/G$ is a manifold of dimension $P$ on which the realization map is
injective. Let $Z=\{(w,b,a):w\ne0,\ a>0\}$ and $Z'=\{(w,b):w\ne0\}/\pm$. Then
\[\Theta^{\mathrm{irr}}/G\ \cong\ \bigl\{\text{sets of $H$ points of $Z$ with pairwise distinct images in }
Z'\bigr\}\times\R .\]
If $d\ge2$ it is connected and its fundamental group maps onto the symmetric group $S_H$.
\end{proposition}

\begin{proof}
A free action of a finite group has a manifold quotient. Using the sign change to make each $a_j$ positive
identifies $\Theta^{\mathrm{irr}}/(\Z/2)^H$ with the ordered $H$-tuples in $Z$ having distinct images in $Z'$,
times the line of the output bias; dividing by $S_H$ forgets the order. For $d\ge2$, $Z$ is connected, and
the tuples with a coincidence in $Z'$ form a closed subset of codimension $d+1\ge2$ in $Z^H$, so the space
of ordered tuples is connected. A connected covering with group $S_H$ gives a surjection from the
fundamental group of the base.
\end{proof}

So there are loops of networks, along which the function computed changes continuously and returns to
itself, that cannot be lifted to loops of parameters: at the end of the loop two hidden units have exchanged
roles. An explicit one moves $(w_1,b_1,a_1)$ to $(w_2,b_2,a_2)$ and back along two different arcs. The space
of functions an architecture computes has topology that its parameter space, a Euclidean space, does not.

\begin{proposition}[Fibres at reducible parameters]\label{prop:fibres}
\Scl[cf.\ \cite{Sus92}, \cite{Fuk96}, \cite{SGJ21}]
At a reducible parameter the fibre of the realization map has positive dimension:
\begin{enumerate}[label=(\roman*)]
\item if $a_j=0$, the fibre contains the affine space of all values of $(w_j,b_j)$, of dimension $d+1$;
\item if $(w_j,b_j)=(w_k,b_k)$, it contains the line $a_j+a_k=\mathrm{const}$;
\item if $w_j=0$, it contains the surface $a_j\sigma(b_j)+c=\mathrm{const}$ in the coordinates
$(a_j,b_j,c)$.
\end{enumerate}
\end{proposition}

Each case is a way for a network with $H$ hidden units to compute a function that needs fewer. The set of
parameters in $\Theta_H$ computing a given function of an irreducible network with $H'<H$ units is a union
of such families, permuted by $G$; it is the subject of a detailed combinatorial study in \cite{SGJ21}. At
these parameters the rank of $J$ drops, by \cref{thm:ident}. The image $\mathcal F$ of the realization map is
therefore a $P$-dimensional manifold $\Theta^{\mathrm{irr}}/G$ together with a singular set, of lower
dimension, onto which the reducible strata of $\Theta$ are collapsed.

\section{Homogeneous activations}

\begin{proposition}\label{prop:relu}\Sli[cf.\ \cite{GLMW22}]
Let $\sigma$ be positively homogeneous of degree one. For each hidden unit $j$ let $\xi_j$ be the vector
field on $\Theta$ equal to the incoming parameters of $j$ in the incoming coordinates, minus the outgoing
parameters in the outgoing coordinates, and zero elsewhere. Then $D_\theta F(\theta,x)\,\xi_j(\theta)=0$
wherever $F$ is differentiable. Consequently the functional dimension of the architecture is at most $P-H$
at every parameter.
\end{proposition}

\begin{proof}
$\xi_j$ is the generator of the one-parameter group of symmetries of \cref{prop:syms}(iii); differentiate
$F(g_\lambda\theta,x)=F(\theta,x)$ at $\lambda=1$. The $\xi_j$ for different units have disjoint supports in
the incoming coordinates, so they are independent wherever each unit has a non-zero incoming parameter, and
the bound holds on the closure of that set by semicontinuity of rank.
\end{proof}

The script confirms both statements on small networks: for $\tanh$ the Jacobian has full rank $P$ at a
random parameter and loses rank at reducible ones; for $\relu$ its rank never exceeds $P-H$ and the fields
$\xi_j$ lie in its kernel.

For $\relu$ the symmetry group is not compact, and its quotient is not a manifold or even a Hausdorff space:
the orbit of a unit with zero outgoing weight is not closed, since
$(\lambda w,\lambda b,0)\to(0,0,0)$ as $\lambda\to0$. The correct object is the quotient stack, in which
such points are remembered with their stabilizers. This is the same reason that vector bundles in
\DT{Ch.~5} formed a stack and not a space, and the loss is a function on it.

\section{Invariant functions}

\begin{proposition}\label{prop:invariant}\Scl
Let $\loss\colon\Theta\to\R$ be invariant under a group $G$ of symmetries. Then the critical set of
$\loss$ is a union of $G$-orbits. If $G$ contains a one-parameter group with generator $\xi$, then
$d\loss(\xi)=0$ identically and, at every critical point, $\xi$ lies in the kernel of the Hessian. If
moreover $\nabla\lambda$ vanishes at $\ev(\theta)$ for a loss $\loss=\lambda\circ\ev$, then the whole fibre
of $\ev$ through $\theta$ is critical.
\end{proposition}

\begin{proof}
Differentiating $\loss(g_s\theta)=\loss(\theta)$ in $s$ gives $d\loss_\theta(\xi)=0$ for all $\theta$;
differentiating that identity in a direction $v$ at a critical point gives
$\nabla^2\loss(\xi,v)+d\loss(D_v\xi)=\nabla^2\loss(\xi,v)=0$. The last statement is
$\nabla\loss=J^{\mathsf T}\nabla\lambda$.
\end{proof}

\begin{corollary}\label{cor:notmorse}\Sob
For an architecture with a positively homogeneous activation and at least one hidden unit, no loss of the
form $\lambda\circ\ev$ is a Morse function on any open set containing a critical point at which some hidden
unit has a non-zero incoming parameter: the Hessian at every such critical point has nullity at least
the number of such units.
\end{corollary}

So Morse theory in the form of \DT{Ch.~8} never applies directly to the loss of a $\relu$ network. It can
apply to the function induced on a slice transverse to the orbits, or in Morse--Bott form, with critical
manifolds in place of critical points. For $\tanh$ networks there is no continuous symmetry and no such
obstruction on $\Theta^{\mathrm{irr}}$; the degeneracies there come from the reducible strata and from
overparametrization (\cref{ch:loss}).

\section*{Notes}

Identifiability of networks up to the obvious symmetries was established for one hidden layer by Sussmann
and by Albertini and Sontag, and for deep $\tanh$ networks by Fefferman. The relation between reducibility
and singularity of the Fisher information matrix is due to Fukumizu; it is the starting point of
Watanabe's singular learning theory \cite{Wat09}. The use of permutation symmetry to compare trained
networks is discussed in \cref{ch:loss}.

\section*{Exercises}

\begin{exercise}\label{exer:verifysym}
Verify the three symmetries of \cref{prop:syms} for a network with one hidden layer.
\end{exercise}

\begin{exercise}\label{exer:rankdrop}
For the one-hidden-layer $\tanh$ network, show that if $a_j=0$ then $\rk J(\theta)\le P-(d+1)$ for every set of
inputs, and if $(w_j,b_j)=(w_k,b_k)$ then $\rk J(\theta)\le P-1$.
\end{exercise}

\begin{exercise}\label{exer:reluscale}
For $F(\theta,x)=a\,\relu(wx+b)+c$ with $d=1$, write down $\xi$ and verify $D_\theta F\cdot\xi=0$ by hand.
What is the functional dimension?
\end{exercise}

\begin{exercise}\label{exer:order}
Compute the order of $G$ for the $\tanh$ architectures with widths $(2,3,1)$ and $(2,4,4,1)$.
\end{exercise}
\part{Level Sets}
\chapter{Transversality and Decision Boundaries}\label{ch:decision}

A binary classifier is a function $f_\theta\colon X\to\R$ whose sign is the predicted class, and its
\term{decision boundary} is $B_\theta=f_\theta^{-1}(0)$. This chapter answers question (Q1) of
\cref{ch:families} for decision boundaries: for almost every parameter the boundary is a smooth hypersurface;
its isotopy type is constant on the components of the complement of a closed null set of parameters; and
every isotopy type of compact hypersurface occurs for wide enough networks.

\section{The boundary is generically a hypersurface}

\begin{theorem}[Generic decision boundaries]\label{thm:boundary}\Srf[\DT{Thm.~4.1, Thm.~1.12}]
Let $F\colon\Theta\times X\to\R$ be a family of the form $F((\theta',c),x)=G(\theta',x)+c$.
\begin{enumerate}[label=(\alph*)]
\item For every $\theta'$, the set of $c$ for which $0$ fails to be a regular value of $f_{(\theta',c)}$ has
measure zero.
\item For almost every $\theta$, the decision boundary $B_\theta$ is a smooth hypersurface in $X$, closed as
a subset and possibly empty, with trivial normal bundle oriented by $\nabla f_\theta$; and
$X\setminus B_\theta$ is the disjoint union of the open sets $\{f_\theta>0\}$ and $\{f_\theta<0\}$.
\item If $X$ is a compact manifold with boundary, then for almost every $\theta$ the boundary $B_\theta$ is
moreover a neat submanifold: it meets $\partial X$ transversally, in $\partial B_\theta$.
\end{enumerate}
\end{theorem}

\begin{proof}
(a) $0$ is a regular value of $G(\theta',\cdot)+c$ exactly when $-c$ is a regular value of
$G(\theta',\cdot)$, and the critical values of a smooth function have measure zero by \DT{Thm.~4.1}. (b)
follows from (a) by Fubini's theorem and \DT{Thm.~1.12}. (c) Apply (a) also to the restriction to
$\partial X$ and use \DT{Thm.~1.17}.
\end{proof}

Every feedforward architecture with a bias in its last layer has this form, whatever its depth and whatever
smooth activation it uses. Part (a) is stronger than the conclusion of \cref{prin:rich}, since it holds for
each value of the other parameters separately; it says that a non-smooth decision boundary can always be
repaired by an arbitrarily small change of the output bias alone. The level sets $f_\theta^{-1}(s)$ for
other thresholds $s$ are covered by the same statement.

\begin{example}\label{ex:cross}
Let $f(x)=x_1^2-x_2^2+c$. For $c\ne0$ the boundary is a hyperbola, a $1$-manifold with two components. For
$c=0$ it is a pair of crossing lines, not a manifold. In the language of \DT{Ch.~4}, the boundary at $c=0$ is
a derived fibre product that is not a manifold; its cobordism class is that of either neighbour.
\end{example}

\section{Several classes}

A classifier with $k$ classes is a map $f_\theta\colon X\to\R^k$, and the predicted class is the index of the
largest coordinate. For a non-empty $I\subset\{1,\dots,k\}$ let $A_I\subset\R^k$ be the set of points whose
largest coordinate is attained exactly on $I$. Then $A_I$ is an open subset of the linear subspace
$\{y_i=y_j:i,j\in I\}$, a submanifold of codimension $|I|-1$, and the $A_I$ partition $\R^k$.

\begin{proposition}\label{prop:multiclass}\Srf[\DT{Thm.~4.5}]
Let $F\colon\Theta\times X\to\R^k$ be a family whose last layer is affine with a bias. For almost every
$\theta$, $f_\theta$ is transverse to every $A_I$. Then the decision region of class $i$ is the open set
$f_\theta^{-1}(A_{\{i\}})$; the set where exactly the classes in $I$ tie is a submanifold of codimension
$|I|-1$; and no more than $d+1$ classes tie anywhere.
\end{proposition}

\begin{proof}
$F$ is a submersion, so \DT{Thm.~4.5} applies to each of the finitely many $A_I$, and a finite intersection
of sets of full measure has full measure. A submanifold of codimension greater than $d$ is empty.
\end{proof}

In the plane, then, a generic classifier has decision regions bounded by smooth arcs meeting three at a time
at isolated points, like the cells of a foam, and four regions never meet at a point.

\section{Chambers}

Let $X$ be compact, and for simplicity without boundary. Define
\[\Sigma=\{(\theta,x):F(\theta,x)=0,\ d_xF(\theta,x)=0\}\subset\Theta\times X,
\qquad\Delta=\pi(\Sigma)\subset\Theta ,\]
the set of parameters for which $0$ is not a regular value: the \term{discriminant} of the family.

\begin{theorem}[Chambers]\label{thm:chambers}\Srf[\DT{Lem.~4.4, Thm.~8.5, Thm.~10.3}]
Let $F$ be as in \cref{thm:boundary} with $X$ compact. Then $\Delta$ is closed and of measure zero. If
$\gamma$ is a path in $\Theta\setminus\Delta$, there is a diffeotopy $\varphi_t$ of $X$ with
$\varphi_t(B_{\gamma(0)})=B_{\gamma(t)}$. In particular the diffeomorphism type of the pair $(X,B_\theta)$ is
constant on each connected component of $\Theta\setminus\Delta$.
\end{theorem}

\begin{proof}
$\Sigma$ is closed and $\pi|_\Sigma$ is proper because $X$ is compact, so $\Delta$ is closed; its complement
has full measure by \cref{thm:boundary}. Over $\Theta\setminus\Delta$ the set $W=F^{-1}(0)$ is a
submanifold of $\Theta\times X$, since $d_xF\ne0$ there, and by \DT{Lem.~4.4} the projection
$W\to\Theta\setminus\Delta$ is a submersion; it is proper. By Ehresmann's theorem \DT{Thm.~8.5} it is a fibre
bundle. Pulled back along $\gamma$ it is trivial, which gives an isotopy of embeddings
$B_{\gamma(0)}\to X$ with image $B_{\gamma(t)}$ at time $t$, and the isotopy extends to a diffeotopy of $X$ by
\DT{Thm.~10.3}.
\end{proof}

The components of $\Theta\setminus\Delta$ are the \term{chambers} of the architecture. The theorem says that
a classifier has a well-defined topology (the number of components of its decision boundary and of its
decision regions, their genera, how they are nested) which does not change under small changes of the
parameters, and can change only when the parameter crosses $\Delta$. What happens at a crossing is the
subject of the next chapter. The symmetry group of \cref{ch:symmetry} permutes the chambers.

For manifolds with boundary the same holds with $\Delta$ enlarged by the parameters for which $B_\theta$ is
tangent to $\partial X$.

\begin{proposition}[Walls]\label{prop:walls}\Sob
Consider the one-hidden-layer architecture
$F(\theta,x)=\sum_{j\le H}a_j\sigma(\langle w_j,x\rangle+b_j)+c$ on $\R^d$ with $\sigma'>0$. At every point
where the matrix $W$ with rows $w_j$ has rank $d$, the map
$\Phi(\theta,x)=(F,d_xF)\in\R\times\R^d$ is a submersion. Hence, over the open set of such parameters,
$\Sigma$ is a submanifold of dimension $P-1$, and $\Delta$ is the image of a manifold of dimension $P-1$.
\end{proposition}

\begin{proof}
$d_xF=W^{\mathsf T}(a\odot\sigma'(z))$ with $z=Wx+b$. A variation $\delta a$ changes $d_xF$ by
$W^{\mathsf T}(\delta a\odot\sigma'(z))$, and $\delta a\mapsto\delta a\odot\sigma'(z)$ is a linear
isomorphism since $\sigma'>0$; so if $W^{\mathsf T}$ is onto, every variation of $d_xF$ is produced. The
resulting change in $F$ is compensated by $\delta c$.
\end{proof}

So the discriminant is no larger than a hypersurface, and the picture of parameter space is that of open
chambers separated by walls of codimension one. The proposition is stated for the simplest architecture;
the argument needs only that the parameters can move the value and the input-gradient of the network
independently at each point, which holds for deeper networks at parameters where the layer Jacobians have
full rank.

\section{Every hypersurface is a decision boundary}

\begin{theorem}[Realization]\label{thm:realize}\Srf[\DT{Thm.~6.2}; Hornik \cite{Hor91}]
Let $K\subset\R^d$ be a compact domain with smooth boundary and $B\subset\operatorname{int}K$ a closed
hypersurface. Let $\sigma$ be smooth, bounded and non-constant. Then for all sufficiently large $H$ there is
a non-empty open set of parameters $\theta$ of the one-hidden-layer architecture with $H$ units for which
$B_\theta\cap K$ is carried to $B$ by a diffeomorphism of $K$ isotopic to the identity.
\end{theorem}

\begin{proof}
A closed hypersurface in $\R^d$ separates and so has a trivial normal bundle; using a tubular neighbourhood
\DT{Thm.~6.2} one constructs a smooth $g\colon K\to\R$ with $g^{-1}(0)=B$, $0$ a regular value, and $g\ne0$ on
$\partial K$. Since $B$ is compact, there is $\varepsilon>0$ such that every $h$ with
$\|h-g\|_{C^1(K)}<\varepsilon$ has $0$ as a regular value and does not vanish on $\partial K$. By Hornik's
theorem, networks with one hidden layer are dense in $C^1(K)$, so some $f_{\theta_0}$ is within
$\varepsilon$ of $g$, and so is $f_\theta$ for $\theta$ near $\theta_0$, the realization map being continuous
into $C^1(K)$. The family $(1-s)g+sf_\theta$, $s\in[0,1]$, stays within $\varepsilon$ of $g$, and the
argument of \cref{thm:chambers} applied to it gives the diffeotopy.
\end{proof}

So there is no topological restriction on decision boundaries beyond those satisfied by every compact
hypersurface in $\R^d$: it is orientable, it separates, and it bounds (\cref{prop:constraints}). The theorem
is silent on how large $H$ must be. Upper bounds for the Betti numbers of decision regions in terms of the
architecture, which give lower bounds for $H$, are due to Bianchini and Scarselli \cite{BS14}.

\section{Piecewise-linear networks}

For $\sigma=\relu$ a network is piecewise linear, its decision boundary is a polyhedral complex, and the
smooth theory does not apply as it stands. The counterpart of \cref{thm:boundary} is due to Grigsby and
Lindsey \cite{GL22}. They define what it means for a $\relu$ network to be generic and transversal, in
terms of the bent hyperplane arrangement formed by the zero sets of its units, and prove that almost all
$\relu$ networks have both properties. As an application they show that for the architecture with one
hidden layer of $d+1$ units, a decision region has at most one bounded connected component: a restriction on
topology coming from the architecture, of the kind \cref{thm:realize} rules out for unbounded width.

\section*{Notes}

That decision boundaries are generically manifolds is folklore, and is usually justified by Sard's theorem
in the form (a). The statement about chambers is the standard structure theory of discriminants in
singularity theory, applied to this family. For topological complexity of decision regions see
\cite{BS14}, and for an empirical study of how networks change the topology of data, \cite{NZL20}.

\section*{Exercises}

\begin{exercise}\label{exer:d1}
Let $d=1$ and $X=[-R,R]$. Show that for almost every $\theta$ the decision boundary is a finite set at which
$f_\theta$ changes sign, and that the parity of its cardinality is determined by the signs of
$f_\theta(\pm R)$.
\end{exercise}

\begin{exercise}\label{exer:plane}
Let $X$ be a closed disc in the plane. Describe the generic decision boundary up to diffeomorphism.
\end{exercise}

\begin{exercise}\label{exer:crossex}
For the family $F(c,x)=x_1^2-x_2^2+c$ on the plane, determine $\Sigma$ and $\Delta$, and describe the
chambers.
\end{exercise}

\begin{exercise}\label{exer:triple}
For a three-class classifier in the plane satisfying the conclusion of \cref{prop:multiclass}, show that
near a point where all three classes tie the decision regions look like three sectors meeting at a point.
\end{exercise}
\chapter{Topology Change along Paths: Surgery and Cobordism}\label{ch:paths}

By \cref{thm:chambers} the topology of a decision boundary can change during training only when the
parameter crosses the discriminant. This chapter answers question (Q2): for a generic path the crossings
are isolated, and at each of them the boundary changes by one surgery, determined by the index of a
critical point. The union of the boundaries along the path is a cobordism, and the surgeries are the
critical points of its height function. This is the Morse theory of \DT{Ch.~8} applied to a manifold that the
path itself creates.

\section{The trace of a path}

Let $F\colon\Theta\times X\to\R$ be a family as in \cref{thm:boundary}, with $X$ compact of dimension $d$, and
let $\gamma\colon[0,1]\to\Theta$ be a path. Write $f_t=f_{\gamma(t)}$ and $B_t=f_t^{-1}(0)$, and define
\[\Phi_\gamma\colon[0,1]\times X\to\R,\quad\Phi_\gamma(t,x)=F(\gamma(t),x);\qquad
W_\gamma=\Phi_\gamma^{-1}(0)\subset[0,1]\times X .\]

\begin{theorem}[The trace is a cobordism]\label{thm:trace}\Srf[\DT{Thm.~1.17, Thm.~4.5}]
If $0$ is a regular value of $\Phi_\gamma$ and of $f_0$ and $f_1$, then $W_\gamma$ is a compact neat
hypersurface in $[0,1]\times X$ with $\partial W_\gamma=B_0\sqcup B_1$: a cobordism from $B_0$ to $B_1$
embedded in $[0,1]\times X$. For a path with endpoints outside $\Delta$, this holds after an arbitrarily
small perturbation of the bias component of $\gamma$ fixed at the endpoints.
\end{theorem}

\begin{proof}
The first statement is \DT{Thm.~1.17}. For the second, replace the bias component $c(t)$ by
$c(t)+s\,\beta(t)$ with $\beta>0$ on $(0,1)$ and vanishing with all derivatives at the ends. The map
$(s,t,x)\mapsto\Phi_\gamma(t,x)+s\beta(t)$ is a submersion on $\{0<t<1\}$, so by \DT{Thm.~4.5} zero is a
regular value for almost every $s$; near $t=0,1$ it is one already because $f_0,f_1$ have $0$ as a
regular value.
\end{proof}

The times at which $\gamma$ meets $\Delta$ are, by \DT{Lem.~4.4}, exactly the critical values of the height
function $\tau\colon W_\gamma\to[0,1]$, $\tau(t,x)=t$. At such a point $(t_0,x_0)$ we have $d_xf_{t_0}(x_0)=0$:
$x_0$ is a critical point of $f_{t_0}$ with critical value $0$.

\section{Surgery}

\begin{theorem}[One surgery at a time]\label{thm:surgery}\Srf[\DT{Thm.~8.6, Cor.~8.7}]
Let $\gamma$ be a path for which $W_\gamma$ is a manifold and $\tau$ is a Morse function with distinct
critical values $t_1<\dots<t_r$ in $(0,1)$. Then:
\begin{enumerate}[label=(\alph*)]
\item On each interval of $[0,1]\setminus\{t_1,\dots,t_r\}$ the boundary $B_t$ moves by an ambient isotopy of
$X$.
\item At each $t_i$ there is exactly one point $x_i\in B_{t_i}$ at which $B_{t_i}$ fails to be a manifold. It
is a non-degenerate critical point of $f_{t_i}$ with $\partial_t\Phi_\gamma(t_i,x_i)\ne0$, and near
$(t_i,x_i)$ there are coordinates on $X$ in which
\[\Phi_\gamma(t,x)=\pm(t-t_i)-x_1^2-\dots-x_k^2+x_{k+1}^2+\dots+x_d^2 ,\]
with $k$ the index of $x_i$ as a critical point of $f_{t_i}$.
\item $B_{t_i+\varepsilon}$ is obtained from $B_{t_i-\varepsilon}$ by a surgery: a copy of
$S^{\lambda-1}\times D^{d-\lambda}$ is removed and $D^\lambda\times S^{d-\lambda-1}$ is glued in, where
$\lambda$ is the index of $\tau$ at $(t_i,x_i)$; $\lambda=k$ or $d-k$ according to the sign in (b).
Correspondingly the region $\{f_t\le0\}$ gains or loses a handle of index $k$.
\end{enumerate}
\end{theorem}

\begin{proof}
(a) is \cref{thm:chambers}. For (b): at a critical point of $\tau$ on $W_\gamma$ the tangent space of
$W_\gamma$ is $\{dt=0\}$, so $d_x\Phi_\gamma=0$ and, $W_\gamma$ being a manifold, $\partial_t\Phi_\gamma\ne0$.
Solving $\Phi_\gamma=0$ for $t$ as a function of $x$ near the point exhibits $W_\gamma$ as a graph over $X$
on which $\tau$ is the function $t(x)$, whose Hessian at $x_i$ is
$-\nabla^2f_{t_i}(x_i)/\partial_t\Phi_\gamma$. So $\tau$ is non-degenerate there exactly when $x_i$ is a
non-degenerate critical point of $f_{t_i}$, and the normal form follows from the Morse lemma with parameters.
(c) is \DT{Thm.~8.6} for $\tau$ on $W_\gamma$: the level sets of $\tau$ are the $B_t$, and passing a critical
point of index $\lambda$ changes the level set by the stated surgery.
\end{proof}

In words: the topology of a decision boundary changes exactly when a critical point of the network function
crosses the threshold, and the change is read off from the index of the critical point and the direction of
crossing. \Cref{tab:surgery} lists the cases for inputs of dimension two and three.

\begin{table}[ht]\centering\small
\begin{tabular}{@{}llp{0.58\textwidth}@{}}
\toprule
$d$ & index $k$ of $x_i$ & effect on the region $\{f_t\le0\}$ when the critical value becomes negative\\
\midrule
$2$ & $0$ (minimum) & a new disc appears; the boundary gains a circle\\
$2$ & $1$ (saddle) & two parts of the region join, or one part meets itself; two boundary arcs
reconnect, so the number of boundary circles changes by $\pm1$\\
$2$ & $2$ (maximum) & a hole in the region is filled; a boundary circle disappears\\
$3$ & $0$ & a new ball appears; the boundary gains a sphere\\
$3$ & $1$ & a tube joins two parts of the region, or adds a handle to one\\
$3$ & $2$ & a tunnel through the region is closed by a disc, or a cavity is split in two\\
$3$ & $3$ & a cavity is filled; a boundary sphere disappears\\
\bottomrule
\end{tabular}
\caption{Elementary changes of topology of a decision region, by the index of the critical point crossing
zero from above. Crossing in the other direction reverses each.}\label{tab:surgery}
\end{table}

\begin{proposition}\label{prop:chi}\Sob
In the situation of \cref{thm:surgery}(c), $\chi(B_{t_i+\varepsilon})-\chi(B_{t_i-\varepsilon})$ equals
$2(-1)^\lambda$ if $d$ is odd and $0$ if $d$ is even, and the Euler characteristic of the region
$\{f_t\le0\}$ changes by $\pm(-1)^k$.
\end{proposition}

\begin{proof}
The change is $\chi(D^\lambda\times S^{d-\lambda-1})-\chi(S^{\lambda-1}\times D^{d-\lambda})
=\chi(S^{d-\lambda-1})-\chi(S^{\lambda-1})=(-1)^{d-\lambda-1}-(-1)^{\lambda-1}$. The second statement is
additivity of $\chi$ under attaching a $k$-cell.
\end{proof}

\begin{example}\label{ex:twoblobs}
Let $F(t,x)=(x_1^2-1)^2+x_2^2-t$ on the plane. The function $(x_1^2-1)^2+x_2^2$ has minima at $(\pm1,0)$ with
value $0$ and a saddle at the origin with value $1$. For $t<0$ the region $\{F\le0\}$ is empty. At $t=0$ two
discs are born (two critical points cross at the same time, which is not generic; a small asymmetry
separates them). At $t=1$ the saddle crosses and the two discs merge. The numbers of components are $0$,
$2$, $1$, and the Euler characteristics of the region $0$, $2$, $1$. The script counts components on a
grid.
\end{example}

\section{Genericity of paths}

Are the hypotheses of \cref{thm:surgery} generic? For paths in the space of all functions, yes: this is the
content of \DT{Thm.~8.12}. For paths in a given architecture one needs enough parameters to move critical
points and critical values independently.

\begin{proposition}\label{prop:genericpath}\Srf[\DT{Thm.~4.6, Thm.~8.12}]
For the architecture of \cref{prop:walls}, with $H\ge d$, let $\gamma$ be a path with endpoints in
$\Theta\setminus\Delta$ along which $W$ has rank $d$. Then arbitrarily close to $\gamma$, with the same
endpoints, there is a path satisfying the hypotheses of \cref{thm:surgery}.
\end{proposition}

\begin{proof}[Sketch]
By \cref{prop:walls} the map $(\theta,x)\mapsto(F,d_xF)$ is a submersion along the path, so the family of
perturbations of $\gamma$ by compactly supported variations of $(a,c)$ induces a family of maps
$(t,x)\mapsto(\Phi,d_x\Phi)$ transverse to the zero section; by \DT{Thm.~4.5} a generic member has $\Sigma$
pulled back to a finite set of points at which the crossing is transverse. Non-degeneracy of the Hessian at
those points, and distinctness of the times, are further conditions of positive codimension on the
$2$-jet, which a variation of $W$ and $b$ moves freely; \DT{Thm.~4.6} applies.
\end{proof}

A training path produced by gradient descent is one particular path, and nothing guarantees that it is
generic in this sense. The proposition says that if it is not, an arbitrarily small perturbation is, and
that the perturbed path sees a finite sequence of elementary surgeries. Paths with symmetry are the
typical exception (\cref{ex:twoblobs}): a symmetric network crossing a wall has several critical points
crossing together.

\section{What cobordism does and does not see}

\begin{proposition}\label{prop:constraints}\Scl
Let $d\ge2$ and let $B=f^{-1}(0)\subset\R^d$ be compact with $0$ a regular value. Then $B$ is the boundary of a
compact region of $\R^d$; it is orientable, its normal bundle is trivial, and $TB\oplus\R$ is trivial. In
particular $B$ is null-cobordant, all its Stiefel--Whitney numbers vanish, and no closed non-orientable
surface is a decision boundary in $\R^3$.
\end{proposition}

\begin{proof}
Choose a closed ball $D$ containing $B$ in its interior. Its boundary sphere is connected and $f$ does not
vanish on it, so $f$ has a constant sign there, say positive. Then $\{f\le0\}\cap D$ is a compact manifold
with boundary $B$. The normal bundle is trivialized by $\nabla f$, and $TB\oplus\nu=T\R^d|_B$ is trivial.
\end{proof}

So the cobordism class of a decision boundary carries no information: it is always zero, as
\DT{Thm.~4.16} predicts for a fibre of a map to a contractible space. What carries information is
finer: the diffeomorphism type of the pair $(X,B_t)$ in each chamber, and the sequence of surgeries along a
path.

That sequence is not an invariant of the endpoints, but its ambiguity is controlled. Parameter space is
contractible, so any two paths with the same endpoints are homotopic, and a generic homotopy between
generic paths meets a short list of codimension-two events.

\begin{proposition}[Moves]\label{prop:moves}\Srf[Cerf; \DT{Thm.~8.12}]
Let $\gamma_0,\gamma_1$ be generic paths with the same endpoints in an architecture rich enough for jet
transversality in two parameters. Their surgery sequences differ by a finite sequence of moves of three
kinds: exchanging the order of two consecutive surgeries at different critical points; inserting or
deleting a pair of consecutive surgeries of indices $\lambda,\lambda+1$ that cancel (a birth--death point of
$f_t$ on the zero level); and inserting or deleting a surgery followed by its inverse (a critical value that
touches zero and returns).
\end{proposition}

The proposition is the two-parameter analogue of \cref{thm:surgery}, and is the form Cerf's theorem takes
for level sets. It says that the surgery sequence of a training run is well defined modulo these moves, and
that anything invariant under them, for instance the parity of the number of surgeries of each index
modulo the cancellation of adjacent pairs, depends only on the endpoints.

\section*{Notes}

The description of how level sets change in one-parameter families is classical Morse and Cerf theory;
see Milnor's lectures on the h-cobordism theorem. Its use for decision boundaries during training appears
in empirical form in studies that track Betti numbers of decision regions and of data representations,
such as \cite{NZL20}. Persistent homology of the family $\{f_t\le s\}$ in the two parameters $(t,s)$ is the
natural computational tool, and the events of \cref{prop:moves} are the generic singularities of its
bifiltration.

\section*{Exercises}

\begin{exercise}\label{exer:blobs}
Verify \cref{ex:twoblobs}: find the critical points and indices of $(x_1^2-1)^2+x_2^2$ and the topology of
the sublevel sets.
\end{exercise}

\begin{exercise}\label{exer:d1path}
For $d=1$, show that along a generic path the points of the decision boundary are born and die in pairs, and
that the parity of their number is constant if $X$ is a circle.
\end{exercise}

\begin{exercise}\label{exer:chi3}
In $d=3$, a decision region evolves from empty to a solid torus. What is the smallest number of surgeries,
and what are their indices?
\end{exercise}

\begin{exercise}\label{exer:hessgraph}
Prove the formula for the Hessian of $\tau$ used in the proof of \cref{thm:surgery}(b).
\end{exercise}
\chapter{The Loss Landscape}\label{ch:loss}

Question (Q3) asks for the shape of a set of parameters. The most important such set is the zero set of the
loss, the parameters that fit the data exactly, and the regular value theorem describes it. The rest of the
landscape is organized by the sublevel sets of the loss, to which the Morse theory of \DT{Ch.~8} applies
with two caveats established in \cref{ch:symmetry} and below: the loss is rarely a Morse function, and
parameter space is not compact.

\section{The interpolation manifold}

Let $\ev\colon\Theta\to\R^N$ be the evaluation map on $n$ data points, $N=nm$, and
$\loss=\tfrac12|\ev-y|^2$. Write $M_y=\ev^{-1}(y)=\loss^{-1}(0)$.

\begin{theorem}[Interpolating parameters]\label{thm:interp}
\Srf[\DT{Thm.~1.12, Thm.~4.1}; Cooper \cite{Coo21}]
\begin{enumerate}[label=(\alph*)]
\item For almost every $y\in\R^N$, $M_y$ is a submanifold of $\Theta$ of dimension $P-N$, possibly empty. If
$P<N$ it is empty for almost every $y$.
\item If $y$ is a regular value and $\theta\in M_y$, then $T_\theta M_y=\ker J(\theta)$ and
$\nabla^2\loss(\theta)=J^{\mathsf T}J$ has rank $N$: it vanishes on $T_\theta M_y$ and is positive definite on
its orthogonal complement.
\item $M_y$ is non-empty for $y$ in the image of $\ev$, and that image has non-empty interior if and only if
$J$ has rank $N$ somewhere.
\end{enumerate}
\end{theorem}

\begin{proof}
(a) By \DT{Thm.~4.1} almost every $y$ is a regular value of $\ev$, and then \DT{Thm.~1.12} applies; if $P<N$
every point of the image is a critical value. (b) $T_\theta M_y=\ker D\ev_\theta$ by \DT{Thm.~1.12}, and the
Hessian is given by \cref{prop:hess}. (c) If $J$ has rank $N$ at $\theta$ then $\ev$ is open near $\theta$;
conversely if the rank is everywhere less than $N$ the image has measure zero.
\end{proof}

Cooper proved the statement in this form for feedforward networks with smooth activations, and showed in
addition that for sufficiently wide networks $M_y$ is non-empty for generic data. In the overparametrized
regime $P>N$, then, the global minima of the loss are not isolated: they form a manifold whose dimension is
the excess of parameters over constraints, and the loss is non-degenerate in the directions normal to it.
In the terminology of Morse theory the loss is a Morse--Bott function near $M_y$. By \cref{prop:hess} its
normal curvature is the spectrum of the tangent kernel.

\begin{proposition}[Convergence near the interpolation manifold]\label{prop:PL}
\Sli[Liu--Zhu--Belkin \cite{LZB22}]
Let $U\subset\Theta$ be an open set on which the smallest eigenvalue of the tangent kernel $K=JJ^{\mathsf T}$
is at least $\mu>0$. Then $|\nabla\loss|^2\ge2\mu\,\loss$ on $U$. A gradient flow line starting at
$\theta_0$ satisfies $\loss(\gamma(t))\le\loss(\theta_0)e^{-2\mu t}$ and has length at most
$\sqrt{2\loss(\theta_0)/\mu}$ for as long as it stays in $U$; if the ball of that radius about $\theta_0$
lies in $U$, the flow line converges to a point of $M_y$.
\end{proposition}

\begin{proof}
$|\nabla\loss|^2=(\ev-y)^{\mathsf T}JJ^{\mathsf T}(\ev-y)\ge\mu|\ev-y|^2=2\mu\loss$. Then
$\frac{d}{dt}\loss=-|\nabla\loss|^2\le-2\mu\loss$, and
$\frac{d}{dt}\sqrt{\loss}=-|\nabla\loss|^2/(2\sqrt\loss)\le-\sqrt{\mu/2}\,|\nabla\loss|$, which integrates to
the bound on the length $\int|\dot\gamma|=\int|\nabla\loss|$.
\end{proof}

The inequality is the quantitative form of transversality: $\mu>0$ says that $\ev$ is a submersion on $U$
with a uniform bound. It is the only quantitative statement in this book, and it is quoted to show where the
qualitative theory hands over to analysis.

\section{Sublevel sets and barriers}

For $c\in\R$ let $\Theta^c=\{\loss\le c\}$. Two parameters are \term{connected at level $c$} if they lie in
the same component of $\Theta^c$, and the \term{barrier} between them is the infimum of such $c$. This is the
notion studied empirically under the name of mode connectivity \cite{GIP18,DVSH18}.

\begin{proposition}[Barriers]\label{prop:barrier}\Srf[\DT{Cor.~8.7, Prop.~8.10}]
Let $\loss\colon\Theta\to\R$ be a Morse function with compact sublevel sets. Then the components of
$\Theta^c$ change only at critical values: a component is born at each local minimum, and two components
merge at a critical point of index $1$ whose two descending directions lead to different components. The
barrier between two local minima is the level at which their components merge; it is attained at a
critical point of index $1$, and equals the minimum over paths joining them of the maximum of $\loss$ along
the path.
\end{proposition}

So barriers are the death times in the degree-zero barcode of the sublevel functor \DT{Prop.~8.10}, and the
barcode is the complete record of connectivity at all levels. The hypotheses hold, for instance, for a
generic analytic loss with weight decay, $\loss+\tfrac\varepsilon2|\theta|^2$, on an architecture without
continuous symmetries. Without weight decay the sublevel sets are not compact and the statement can fail
(\cref{prop:notproper}).

\begin{example}[Two parameters]\label{ex:hyperbola}
Let $F(a,b;x)=abx$ with one data point $(x,y)=(1,1)$, so $\loss=\tfrac12(ab-1)^2$. The interpolation manifold
is the hyperbola $ab=1$, of dimension $P-N=1$, with two components exchanged by the symmetry
$(a,b)\mapsto(-a,-b)$. The only other critical point is the origin, a saddle with $\loss=\tfrac12$ and
Hessian eigenvalues $\pm1$. For $c<\tfrac12$ the sublevel set has two components; at $c=\tfrac12$ they merge.
So the barrier between $(1,1)$ and $(-1,-1)$ is $\tfrac12$, attained on the straight segment between them,
and no path does better. After applying the symmetry to one of the two points they coincide. The script
checks the component counts on a grid.
\end{example}

The example is the smallest instance of a general phenomenon. By \cref{prop:invariant} minima come in orbits
of the symmetry group, and two minima in the same orbit compute the same function but may be separated by a
barrier. The barrier between two trained networks should therefore be measured after choosing
representatives of their orbits, which is the content of the conjecture of Entezari et al.\ that barriers
largely disappear once permutations are accounted for \cite{EKSN22}. In the language of \cref{ch:symmetry},
the relevant sublevel sets are those of the function induced on the quotient $\Theta/G$.

Two results from the literature describe the overparametrized case. For a class of deep networks with
piecewise-linear activations in which one hidden layer is wider than the number of training points, Nguyen
proved that every sublevel set of the loss is connected \cite{Ngu19}. And \c{S}im\c{s}ek et al.\ describe the
critical manifolds that the reducible strata of \cref{prop:fibres} contribute, and show how adding hidden
units connects the $|G|$ copies of a minimum \cite{SGJ21}.

\section{Why the loss is rarely a Morse function}

Three mechanisms make the loss degenerate, and they are independent.

\begin{enumerate}[label=(\roman*)]
\item \emph{Overparametrization.} If $P>N$ the zero-loss minima form the manifold $M_y$ of positive
dimension (\cref{thm:interp}).
\item \emph{Continuous symmetry.} For homogeneous activations every critical point has a Hessian of nullity
at least $H$ (\cref{cor:notmorse}).
\item \emph{Reducible strata.} A critical point of a smaller network gives a positive-dimensional family of
critical points of a larger one, along the fibres of \cref{prop:fibres}; these are typically saddles
\cite{FA00}.
\end{enumerate}

In each case the degeneracy is of Morse--Bott type on an open dense part of the critical set, and the handle
theorem \DT{Thm.~8.6} has a Morse--Bott version in which a critical manifold contributes a bundle of cells
over itself. But the density of Morse functions \DT{Cor.~4.8} is of no direct use: a generic perturbation of
the loss as a function on $\Theta$ is not the loss of any data set, and perturbing the data preserves all
three mechanisms. This is the point of \cref{prin:rich} again. Genericity must be established within the
family one actually has.

\section{Critical points at infinity}

\begin{proposition}\label{prop:notproper}\Sob
For architectures with a bounded activation the loss is not proper, and its infimum need not be attained.
For $F(w;x)=\tanh(wx)$ with the single data point $(1,1)$, $\loss(w)=\tfrac12(\tanh w-1)^2$ has no critical
point, its infimum $0$ is not attained, and every sublevel set $\Theta^c$ with $0<c<2$ is a half-line.
\end{proposition}

\begin{proof}
$\loss'(w)=(\tanh w-1)\operatorname{sech}^2w<0$ for all $w$.
\end{proof}

So the topology of sublevel sets can change with no critical point to account for it, the change being
carried off to infinity, and the regular interval theorem \DT{Thm.~8.4}, which needs compactness, does not
apply. Any use of Morse theory for losses must either restore compactness (weight decay, a bounded
parameter domain) or account for these ends explicitly, as one does for the Palais--Smale condition in
infinite dimensions.

\section*{Notes}

The manifold of global minima was described by Cooper \cite{Coo21}; the Polyak--\L{}ojasiewicz condition in
this setting is due to Liu, Zhu and Belkin. Mode connectivity was observed by Garipov et al.\ and Draxler et
al., and analysed for wide networks by Freeman and Bruna and by Nguyen; permutation alignment is studied in
\cite{EKSN22}. For the singular structure of the landscape at reducible parameters see \cite{FA00,SGJ21} and
Watanabe's theory \cite{Wat09}.

\section*{Exercises}

\begin{exercise}\label{exer:linreg}
For the linear family of \cref{exer:linear} with $P>n$ and features in general position, describe $M_y$ and
the Hessian of the loss.
\end{exercise}

\begin{exercise}\label{exer:hyperbola}
Verify \cref{ex:hyperbola}: find all critical points of $\tfrac12(ab-1)^2$, their Hessians, and the
components of the sublevel sets.
\end{exercise}

\begin{exercise}\label{exer:PL}
In the setting of \cref{prop:PL}, show that the limit point of the flow line lies within
$\sqrt{2\loss(\theta_0)/\mu}$ of $\theta_0$.
\end{exercise}

\begin{exercise}\label{exer:deeplin}
For $F(a_1,\dots,a_k;x)=a_1\cdots a_kx$ with data $(1,1)$, show that $M_y$ has $2^{k-1}$ components and
dimension $k-1$, and that the origin is a degenerate critical point for $k\ge3$.
\end{exercise}
\part{Maps}
\chapter{Embeddings, Isotopy, and Depth}\label{ch:isotopy}

The previous three chapters studied level sets. This one studies the maps themselves, and in particular
what a network does to a data set regarded as a subset of input space. The standing assumption, usually
called the manifold hypothesis \cite{FMN16}, is that the data lie on or near a compact submanifold
$K\subset\R^D$ of dimension $m$ much smaller than $D$, with the classes occupying disjoint closed subsets
$K_+,K_-$. A hidden layer of width $k$ is a map $h_\theta\colon\R^D\to\R^k$, and the question is what
$h_\theta(K)$ can be.

\section{The first layer generically embeds}

\begin{proposition}\label{prop:firstlayer}\Srf[Whitney; \DT{Thm.~4.5, Cor.~4.8, Prop.~3.12}]
Let $K\subset\R^D$ be a compact submanifold of dimension $m$, let $k\ge2m+1$, and let $\sigma$ be a
diffeomorphism of $\R$ onto an open interval. Then for almost every $W\in\R^{k\times D}$ and every $b$, the
map $x\mapsto\sigma(Wx+b)$ restricted to $K$ is an embedding. The set of such $(W,b)$ is open.
\end{proposition}

\begin{proof}
Applying $\sigma$ coordinatewise after a translation is a diffeomorphism onto its image, so the question is
whether $W|_K$ is an embedding. The map $(W,(x,y))\mapsto W(x-y)$ on
$\R^{k\times D}\times(K\times K\setminus\Delta)$ is a submersion to $\R^k$, so by \DT{Thm.~4.5} for almost
every $W$ its restriction to $K\times K\setminus\Delta$, a manifold of dimension $2m<k$, is transverse to
$0$ and hence misses it: $W|_K$ is injective. Likewise $(W,(x,v))\mapsto Wv$ on the unit tangent bundle of
$K$, of dimension $2m-1$, is a submersion, so for almost every $W$ no unit tangent vector is killed: $W|_K$ is
an immersion. An injective immersion of a compact manifold is an embedding. Openness is \DT{Prop.~3.12}.
\end{proof}

So a hidden layer wider than twice the intrinsic dimension of the data loses no topological information for
almost every choice of weights, however small it is compared with the ambient dimension $D$. This is
Whitney's embedding theorem read as a statement about random projections. Narrower layers may fail to be
injective on $K$, and that is where the topology of a representation can change.

\section{What injective networks cannot do}

A classifier ends with a linear map, so the last hidden representation must make the classes linearly
separable. If the network up to that point is injective, its restriction to $K$ is a homeomorphism onto
the image, and separability becomes a constraint on how $K_+$ and $K_-$ sit in the ambient space.

\begin{theorem}[Nested classes]\label{thm:obstruction}\Scl[invariance of domain; cf.\ \cite{Joh19}]
Let $h\colon\R^d\to\R^d$ be continuous and injective, let $\beta\subset\R^d$ be a closed topological ball with
boundary sphere $S$, and let $p$ be an interior point of $\beta$. Then $h(p)$ lies in the convex hull of
$h(S)$. In particular no hyperplane separates $h(p)$ from $h(S)$.
\end{theorem}

\begin{proof}
Suppose a linear functional $\lambda$ satisfies $\lambda>0$ on $h(S)$ and $\lambda(h(p))\le0$. Let $q$ be a
point of the compact set $h(\beta)$ at which $\lambda$ is smallest. Then $\lambda(q)\le0$, so $q\notin h(S)$
and $q\in h(\operatorname{int}\beta)$, which is open in $\R^d$ by invariance of domain. But a linear
functional has no local minimum on an open set.
\end{proof}

\begin{corollary}\label{cor:width}\Scl[\cite{Joh19,HS17}]
A feedforward network on $\R^d$ in which every hidden layer has width $d$, every weight matrix is
invertible, and the activation is injective, cannot separate a point from a sphere surrounding it. To do so
a network must either have a layer of width at least $d+1$ or use a layer that is not injective.
\end{corollary}

Both escapes are used in practice. A wider layer provides the extra dimension into which the inner class can
be lifted. A $\relu$ layer is not injective: it folds space along hyperplanes, and a network of $\relu$
layers can map a nested data set to a separable one by collapsing. Naitzat, Zhitnikov and Lim \cite{NZL20}
measured the Betti numbers of the two classes layer by layer and found that well-trained networks reduce
them steadily, faster with $\relu$ than with smooth invertible activations. In the terms of this chapter:
smooth invertible layers of constant width act by diffeomorphisms and cannot change topology, and the
network must then rely on the few layers where injectivity fails.

\section{Neural ordinary differential equations}

A \term{neural ODE} \cite{CRBD18} is the time-$T$ map $\varphi_\theta$ of the equation
$\dot x=v_\theta(x,t)$, where $v_\theta$ is a network with input $(x,t)$ and output in $\R^d$.

\begin{proposition}\label{prop:node}\Srf[\DT{Thm.~10.2}]
If $v_\theta$ is smooth with bounded velocity, as it is when its last hidden layer has a bounded activation,
then $t\mapsto\varphi^t_\theta$ is a diffeotopy of $\R^d$. Hence $\varphi_\theta$ is a diffeomorphism isotopic
to the identity, it preserves orientation, and for every compact $K\subset\R^d$ the image $\varphi_\theta(K)$
is ambient isotopic to $K$.
\end{proposition}

\begin{corollary}[Obstructions]\label{cor:nodeobst}\Scl[cf.\ Dupont--Doucet--Teh \cite{DDT19}]
A neural ODE on $\R^d$ cannot:
\begin{enumerate}[label=(\alph*)]
\item for $d=1$, reverse the order of two points;
\item map a sphere and a point inside it to linearly separable sets (\cref{thm:obstruction});
\item for $d=3$, map two linked circles to unlinked ones, or a knotted circle to a round one.
\end{enumerate}
\end{corollary}

\begin{proof}
(a) A diffeomorphism of $\R$ isotopic to the identity is increasing. (c) The linking number is the degree of
the Gauss map \DT{Ex.~7.12} and is invariant under isotopy by \DT{Thm.~7.2}; the knot type is an isotopy
invariant by definition.
\end{proof}

These are statements about exact representation. A neural ODE can still approximate a map that violates
them, at the cost of a vector field with large derivatives, and \cite{DDT19} observed that training then
becomes slow and unstable. Their remedy was to append extra coordinates to the state.

\section{Augmentation}

\begin{theorem}[Flows in the stable range]\label{thm:augment}
\Srf[\DT{Thm.~10.9, Thm.~10.3, Thm.~10.2}]
Let $K$ be a compact manifold of dimension $m$ and let $e_0,e_1\colon K\to\R^N$ be embeddings, with
$N\ge2m+2$. Then there is a smooth, compactly supported, time-dependent vector field on $\R^N$ whose
time-one map $\varphi$ satisfies $\varphi\circ e_0=e_1$.
\end{theorem}

\begin{proof}
By \DT{Thm.~10.9} the two embeddings are isotopic. By the isotopy extension theorem \DT{Thm.~10.3} the
isotopy is induced by a diffeotopy of $\R^N$ with compact support, and by \DT{Thm.~10.2} the diffeotopy is
the flow of its velocity field.
\end{proof}

\begin{corollary}[Separation after augmentation]\label{cor:separate}\Sob
Let $K=K_+\sqcup K_-\subset\R^d$ be a compact submanifold of dimension $m$, the union of two disjoint compact
submanifolds, and let $N\ge\max(d+1,2m+2)$. Then there is a compactly supported flow on $\R^N$ whose
time-one map carries $K_+\times\{0\}$ and $K_-\times\{0\}$ into the open half-spaces $\{x_N>0\}$ and
$\{x_N<0\}$. The vector field may be taken to be a neural network with one hidden layer and input
$(x,t)$.
\end{corollary}

\begin{proof}
Take $e_0$ the inclusion of $K$ in $\R^d\times\{0\}\subset\R^N$ and $e_1=e_0\pm(0,\dots,0,1)$ on $K_\pm$, and
apply \cref{thm:augment} to get a field $v$ with flow $\varphi$. For the last statement, approximate $v$ in
$C^1$ on a compact set containing the support, by Hornik's theorem \cite{Hor91}; by Gr\"onwall's inequality
the time-one maps are then close in $C^1$ on compact sets, and the conclusion, being an open condition on
the images of the compact sets $K_\pm$, persists.
\end{proof}

So the obstructions of \cref{cor:nodeobst} disappear once the state space has dimension at least $2m+2$,
where $m$ is the dimension of the data and not of the ambient space. For finite data sets $m=0$ and two
dimensions suffice, in agreement with (a): points on a line cannot be reordered by a flow, points in the
plane can. For curves the bound is $4$, and it is sharp, since circles link in $\R^3$. For connected data
of dimension $m\ge2$ the bound can be lowered to $2m+1$ by the second part of \DT{Thm.~10.9}. Zhang et al.\
\cite{ZGUA20} prove a related statement about maps: every homeomorphism of $\R^d$ can be embedded in a flow on
$\R^{2d}$, and hence approximated by a neural ODE or an invertible residual network acting there. The dimension count of \cref{thm:augment}
depends on the data alone.

Residual networks fit the same description. A block $x\mapsto x+g(x)$ with $g$ of Lipschitz constant less
than one is a diffeomorphism isotopic to the identity (\cref{prop:iresnet}), and a composite of such blocks
is a discretized isotopy. Depth, for such architectures, is time.

\section{The space of representations}

\begin{problem}\label{prob:calculus}\Spg
For a compact data manifold $K$ of dimension $m$ and width $k\ge m+3$, the space $\Emb(K,\R^k)$ of
representations that preserve the topology of the data is described by manifold calculus
\DT{Thm.~10.10}: it is the limit of a tower whose first stage is the space of immersions and whose higher
stages are built from configuration spaces of $K$. Determine which components and which homotopy classes of
this space are reached by the realization map of a given architecture, as a function of width and depth.
\end{problem}

For $k\ge2m+2$ the space is connected and the question is about higher homotopy; in the range
$m+3\le k\le2m+1$ already the set of components is interesting, and \cref{cor:nodeobst}(c) is the first
example outside that range.

\section*{Notes}

The picture of a network as progressively untangling data manifolds goes back at least to expository
writing of Olah in 2014, where \cref{thm:obstruction} for $d=2$ and the role of an extra dimension are
described. Minimal width for universal approximation is treated in \cite{HS17,Joh19}. Neural ODEs are from
\cite{CRBD18}, their limitations and augmentation from \cite{DDT19}, and approximation properties from
\cite{ZGUA20}.

\section*{Exercises}

\begin{exercise}\label{exer:monotone}
Show that the time-one map of a bounded smooth vector field on $\R$ is increasing.
\end{exercise}

\begin{exercise}\label{exer:swap}
Write down a compactly supported vector field on $\R^2$ whose time-one map exchanges the points $(\pm1,0)$.
\end{exercise}

\begin{exercise}\label{exer:link}
Show that the linking number of two disjoint closed curves in $\R^3$ is unchanged when both are moved by a
diffeomorphism isotopic to the identity.
\end{exercise}

\begin{exercise}\label{exer:proj}
Let $K$ be the curve $t\mapsto(\cos t,\sin t,\cos2t,\sin2t,\cos3t)$ in $\R^5$. Show that the projection to
the first two coordinates is an embedding of $K$, and that the projection to the third and fourth is not.
\end{exercise}
\chapter{Degree and Invertibility}\label{ch:degree}

Some architectures are designed so that every member is invertible: normalizing flows, invertible residual
networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both
cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from
\DT{Ch.~7}.

\section{Global invertibility}

\begin{theorem}[Hadamard]\label{thm:hadamard}\Scl
A local diffeomorphism $f\colon\R^d\to\R^d$ is a diffeomorphism if and only if it is proper.
\end{theorem}

\begin{proof}
A proper local diffeomorphism is a covering map, and $\R^d$ is simply connected. In terms of degree: every
point is a regular value, the sign of $\det Df$ is constant, and so the number of preimages of a point is
the absolute value of the degree, which is constant; it is $1$ because a covering of a simply connected
space is trivial.
\end{proof}

A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the
only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded
one.

\begin{proposition}[Invertible residual blocks]\label{prop:iresnet}
\Scl[Banach; Behrmann et al.\ \cite{BGC19}]
Let $g\colon\R^d\to\R^d$ have Lipschitz constant $L<1$ and let $f=\id+g$.
\begin{enumerate}[label=(\alph*)]
\item $f$ is a bijection, its inverse is Lipschitz with constant $(1-L)^{-1}$, and
$x_{j+1}=y-g(x_j)$ converges to $f^{-1}(y)$ with $|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|$.
\item If $g$ is smooth then $f$ is a diffeomorphism with $\det Df>0$.
\item $f_s=\id+sg$, $0\le s\le1$, is a diffeotopy from the identity to $f$. In particular $f$ is proper of
degree $1$.
\end{enumerate}
\end{proposition}

\begin{proof}
(a) $x\mapsto y-g(x)$ is a contraction; and $|f(x)-f(x')|\ge(1-L)|x-x'|$. (b), (c): the eigenvalues of $Dg$
have modulus at most $L$, so $1+sDg$ is invertible for $s\in[0,1]$, and its determinant is continuous,
non-zero, and equal to $1$ at $s=0$.
\end{proof}

Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning
networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common
use all realize diffeomorphisms in the identity component of $\Diff(\R^d)$, and by \DT{Thm.~10.2} every
such diffeomorphism with compact support is conversely the time-one map of a flow.

\section{Topological mismatch}

A normalizing flow models a probability measure $\nu$ on $\R^d$ as the image $h_*\mu$ of a simple measure
$\mu$, usually a Gaussian, under a diffeomorphism $h$.

\begin{proposition}\label{prop:mismatch}\Scl[cf.\ Cornish et al.\ \cite{CCDD20}]
For a homeomorphism $h$ of $\R^d$ and a Borel probability measure $\mu$,
$\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu)$. Hence if the supports of $\mu$ and $\nu$ are not
homeomorphic, no homeomorphism satisfies $h_*\mu=\nu$.
\end{proposition}

A Gaussian has support $\R^d$. A target supported on two disjoint balls, on a sphere, or on any proper
closed subset cannot be represented exactly, and Cornish et al.\ show that a sequence of flows approximating
such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature
are the ones differential topology suggests: a target supported on a submanifold $M^m\subset\R^d$ is modelled
by a flow on $\R^m$ followed by an embedding $\R^m\to\R^d$, and when $M$ is not diffeomorphic to an open
subset of $\R^m$, by several such charts with a discrete index, which is an atlas in the sense of
\DT{Ch.~1}. The minimal number of charts is an invariant of $M$; for a sphere it is two.

\section{Equilibrium layers}

A \term{deep equilibrium layer} outputs a solution $z$ of $z=g_\theta(z,x)$ for a network $g_\theta$. The
existence and number of solutions is a fixed-point problem, and \DT{Prop.~7.11} describes its three aspects:
a generic signed count, a derived fibre product, and a trace.

\begin{theorem}[Equilibria of a bounded layer]\label{thm:deq}\Srf[\DT{Thm.~4.5, Thm.~7.2, Thm.~7.9}]
Let $g(z)=\tanh(Az+u)$ on $\R^k$, with $A\in\R^{k\times k}$ and $u\in\R^k$.
\begin{enumerate}[label=(\alph*)]
\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an
eigenvalue of $Dg$ there. The fixed points are then finite in number.
\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed
points is odd, and at least one exists.
\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with
$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.
\end{enumerate}
\end{theorem}

\begin{proof}
(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,
which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of
$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed
points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For
$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so
$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to
(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.
At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.
\end{proof}

In one dimension, $z=\tanh(az+u)$ has one solution for $a\le1$, and for $a>1$ and small $|u|$ three, with
signs $+,-,+$: two stable equilibria and an unstable one between them. The script finds the fixed points for
several $(a,u)$ and for a two-dimensional example and checks that the signed count is $1$.

Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the
transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The
exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set $\Delta$ of
\cref{ch:decision}; crossing it creates or destroys a pair of equilibria of opposite sign, as in the
birth--death family of \DT{Thm.~8.12}, and it is there that the output of an equilibrium layer can jump.

\section*{Notes}

The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and
Mazur, Palais). Invertible residual networks are from \cite{BGC19}. The observation that the topology of
the support obstructs normalizing flows, with a quantitative form, is in \cite{CCDD20}. Deep equilibrium
models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone
operator parametrizations) are a separate line of work, and \cref{thm:deq} describes what happens without
them.

\section*{Exercises}

\begin{exercise}\label{exer:banach}
Prove \cref{prop:iresnet}(a).
\end{exercise}

\begin{exercise}\label{exer:det}
Let $g$ be smooth with $\|Dg\|\le L<1$ everywhere. Show $\det(1+Dg)\ge(1-L)^d$.
\end{exercise}

\begin{exercise}\label{exer:deq1}
For $k=1$ and $u=0$, find the fixed points of $z=\tanh(az)$ as a function of $a$ and verify
\cref{thm:deq}(b). What happens at $a=1$?
\end{exercise}

\begin{exercise}\label{exer:supp}
Prove \cref{prop:mismatch}.
\end{exercise}
\part{Assessment}
\chapter{What Was Used, and What Is Missing}\label{ch:coda}

\section{The inputs}

\Cref{tab:used} lists the main statements of this book with the theorems of the companion volume from which
they follow. Every entry in the right-hand column was known by the mid-1970s.

\begin{table}[ht]\centering\small
\begin{tabular}{@{}p{0.47\textwidth}p{0.43\textwidth}@{}}
\toprule
statement here & from \cite{DTHC}\\
\midrule
backpropagation composes (\cref{thm:backprop}) & reverse chain rule, Prop.~2.11\\
generic decision boundaries are hypersurfaces (\cref{thm:boundary}) & Sard, Thm.~4.1; regular values, Thm.~1.12\\
chambers and walls (\cref{thm:chambers,prop:walls}) & fibrewise lemma 4.4; Ehresmann, Thm.~8.5; isotopy extension, Thm.~10.3\\
every hypersurface is realized (\cref{thm:realize}) & tubular neighbourhoods, Thm.~6.2; openness, Thm.~3.2\\
surgery along paths (\cref{thm:trace,thm:surgery}) & neat preimages, Thm.~1.17; handles, Thm.~8.6\\
moves between paths (\cref{prop:moves}) & Cerf, Thm.~8.12\\
interpolation manifold (\cref{thm:interp}) & regular values, Thm.~1.12; Sard, Thm.~4.1\\
barriers (\cref{prop:barrier}) & sublevel functor, Cor.~8.7, Prop.~8.10\\
first layer embeds (\cref{prop:firstlayer}) & parametric transversality, Thm.~4.5\\
neural ODEs are diffeotopies (\cref{prop:node}) & flows, Thm.~10.2\\
augmentation (\cref{thm:augment}) & isotopy in the stable range, Thm.~10.9; Thm.~10.3\\
odd number of equilibria (\cref{thm:deq}) & degree, Thm.~7.2; Thm.~4.5\\
\bottomrule
\end{tabular}
\caption{What is used.}\label{tab:used}
\end{table}

\section{What is and is not new}

Nothing in this book is a new theorem of differential topology, and most of its statements about networks
have appeared in the machine learning literature in some special case, with the references given in the
text: generic smoothness of decision boundaries, the manifold of global minima, the obstruction to neural
ODEs and its removal by augmentation, the mismatch of supports for flows. What the book adds is the uniform
derivation, with the hypothesis that makes each statement true stated as a transversality property of the
architecture (\cref{prin:rich}), and a small number of statements tagged as observations that we have not
found in this form: the description of the moduli of irreducible one-hidden-layer networks as a
configuration space, with its monodromy (\cref{prop:moduli}); the wall structure of the discriminant
(\cref{prop:walls}); the surgery description of training with its Euler characteristic count
(\cref{thm:surgery,prop:chi}) and its moves (\cref{prop:moves}); the dimension count $2m+2$ for
augmentation in terms of the data (\cref{thm:augment,cor:separate}); and the parity of the number of
equilibria (\cref{thm:deq}). Each is a short corollary and is offered as such.

The higher-categorical layer was used as language more than as a tool. The bicategory $\Para$ and the lens
functor organize \cref{ch:families,ch:backprop}; the quotient by symmetries is a stack and not a space for
homogeneous activations (\cref{ch:symmetry}); and a degenerate level set is a derived manifold with a
cobordism class (\cref{ex:cross}). The theorems that do the work are those of \cref{tab:used}.

\section{Limits}

\emph{Smoothness.} The standing assumption excludes $\relu$. For piecewise-linear networks the analogue of
\cref{thm:boundary} is the theorem of Grigsby and Lindsey \cite{GL22}; analogues of
\cref{thm:chambers,thm:surgery} would require stratified Morse theory, and are not proved here.

\emph{Genericity within the family.} ``Almost every parameter'' is with respect to Lebesgue measure on
$\Theta$, so the statements apply to parameters drawn from a distribution with a density, as at a random
initialization. They do not apply to distinguished parameters. In particular, if $g$ is an orthogonal
symmetry of the architecture, the gradient of an invariant loss is equivariant, and so the fixed-point set
of $g$ is invariant under gradient flow: a network initialized with two identical hidden units keeps them
identical for ever, and stays in a reducible stratum of measure zero where none of the generic statements
need hold.

\emph{Paths.} \Cref{thm:surgery} describes generic paths. A training trajectory is a particular path, the
solution of a differential equation, and we have proved nothing about whether it is generic. The honest
statement is \cref{prop:genericpath}: arbitrarily close to it there is a path with the stated behaviour.

\emph{Compactness.} Input domains are assumed compact for the isotopy statements, and parameter space is
never compact, which restricts the use of Morse theory for losses (\cref{prop:notproper}).

\emph{No rates.} With the single exception of \cref{prop:PL}, the statements are qualitative. They say that
a set has measure zero and not how small its neighbourhoods are; that an isotopy exists and not how large
its velocity must be; that a boundary of a given topology is realizable and not at what width. Each of these
has a quantitative counterpart in the literature that the present methods do not reach.

\emph{No learning theory.} Nothing here concerns generalization.

\section{Directions}

\begin{problem}[Piecewise-linear version]\label{prob:PL}\Spg
Prove the analogues of \cref{thm:chambers,thm:surgery} for $\relu$ networks that are generic and transversal
in the sense of \cite{GL22}, with handle attachments replaced by the elementary moves of stratified Morse
theory on the canonical polyhedral complex.
\end{problem}

\begin{problem}[Quantitative discriminant]\label{prob:quant}\Spg
For a given architecture and initialization distribution, estimate the measure of the
$\varepsilon$-neighbourhood of the discriminant $\Delta$ of \cref{ch:decision}, and hence the expected number
of surgeries along a path of given length. \Cref{prop:walls} suggests that $\Delta$ is a hypersurface and
that the answer is governed by its volume.
\end{problem}

\begin{problem}[The stack of networks]\label{prob:stack}\Spg
For homogeneous activations, describe the quotient stack $[\Theta/G]$ of \cref{ch:symmetry} and the critical
locus of a loss on it as a derived object, so that \cref{thm:interp} and the degeneracies listed in
\cref{ch:loss} become statements about a single space with a virtual dimension.
\end{problem}

Together with \cref{prob:calculus}, these are the places where the methods of the companion volume, and in
particular its higher-categorical parts (derived manifolds, stacks, the calculus of embeddings), would be
doing more than providing a vocabulary.
\appendix
\chapter{Dictionary}\label{app:dictionary}

{\small
\begin{longtable}{@{}p{0.25\textwidth}p{0.36\textwidth}p{0.29\textwidth}@{}}
\toprule
Network notion & Differential topology & Where\\
\midrule
\endhead
architecture $F\colon\Theta\times X\to Y$ & plot of the mapping object $[X,Y]$; morphism of $\Para$ & \cref{def:family,prop:plot}\\
composition of layers & composition of plots & \cref{def:para}\\
weight sharing, freezing & $2$-cells of $\Para$ & \cref{exer:sharing}\\
forward-mode derivative & tangent functor & \DT{Prop.~2.1}\\
backpropagation & reverse derivative as a functor to lenses & \cref{thm:backprop}\\
tangent kernel & Gram matrix $JJ^{\mathsf T}$ of the evaluation map & \cref{prop:ntk}\\
functional dimension & rank of the realization map & \cref{def:fdim,prop:rank}\\
permutation, sign, scaling symmetry & group of invertible $2$-cells & \cref{prop:syms}\\
redundant hidden unit & reducible parameter; positive-dimensional fibre & \cref{prop:fibres}\\
space of networks & quotient manifold, or stack & \cref{prop:moduli}\\
decision boundary & preimage of a regular value & \cref{thm:boundary}\\
multi-class boundaries & preimage of a stratification & \cref{prop:multiclass}\\
robust topology of a classifier & chamber of the discriminant complement & \cref{thm:chambers}\\
change of boundary topology & surgery; critical point of the trace & \cref{thm:surgery}\\
training run, seen in input space & embedded cobordism with a Morse function & \cref{thm:trace}\\
two runs with the same endpoints & Cerf moves & \cref{prop:moves}\\
interpolating parameters & fibre of the evaluation map over a regular value & \cref{thm:interp}\\
sharpness at a minimum & normal Hessian $J^{\mathsf T}J$ & \cref{prop:hess}\\
loss barrier, mode connectivity & death level in the degree-zero barcode & \cref{prop:barrier}\\
saturation, diverging weights & failure of properness & \cref{prop:notproper}\\
hidden representation & immersion or embedding of the data manifold & \cref{prop:firstlayer}\\
minimal width & invariance of domain & \cref{thm:obstruction}\\
neural ODE & diffeotopy & \cref{prop:node}\\
augmented neural ODE & isotopy of embeddings in the stable range & \cref{thm:augment}\\
residual block with small branch & diffeomorphism isotopic to the identity & \cref{prop:iresnet}\\
normalizing flow & diffeomorphism; pushforward of measures & \cref{prop:mismatch}\\
equilibrium layer & transverse fixed point; Lefschetz count & \cref{thm:deq}\\
\bottomrule
\end{longtable}
}
\chapter{Computations}\label{app:computations}

The script \texttt{verify-parametrized-networks.py} uses only the Python standard library. It implements small feedforward
networks (widths $(2,3,1)$, $(2,4,4,1)$ and a few others) with forward evaluation, forward-mode
differentiation by dual numbers and reverse-mode differentiation by backpropagation, a Runge--Kutta
integrator, a Jacobi eigenvalue routine and a flood-fill component counter. It performs 49 checks in about
three seconds, each printing \texttt{PASS} or \texttt{FAIL}, and exits with status zero if and only if all
pass. The checks recompute the examples in the text and test numerical instances of the structural
statements on random parameters with a fixed seed. They are not proofs, and a statement about almost every
parameter is tested only at the parameters sampled.

{\small
\begin{longtable}{@{}p{0.13\textwidth}p{0.79\textwidth}@{}}
\toprule
Chapter & What is checked\\
\midrule
\endhead
\ref{ch:families} & parameter counts (\cref{exer:count}); associativity of composition in $\Para$; the
derivative of a weight-tied network (\cref{exer:sharing}); the family $\theta+x^3$ (\cref{exer:notrich})\\
\ref{ch:backprop} & backpropagation against forward mode and finite differences for two architectures, and
for the input covector (\cref{thm:backprop}); the tangent kernel is positive semidefinite of rank $\rk J$;
the output dynamics $\dot u=-K(u-y)$ (\cref{prop:ntk}); the loss gradient; the Hessian as Gauss--Newton plus
curvature term (\cref{exer:hess})\\
\ref{ch:symmetry} & all $48$ symmetries of the $\tanh$ network $(2,3,1)$ and the size of an orbit
(\cref{prop:syms}); the scaling symmetry for $\relu$; rank $P=13$ at a random parameter and ranks $10$ and
$9$ at reducible ones (\cref{thm:ident}, \cref{exer:rankdrop}); for $\relu$, the generators $\xi_j$ in
$\ker J$ and $\rk J\le P-H$ (\cref{prop:relu}); a path of irreducible networks exchanging two hidden units
(\cref{prop:moduli})\\
\ref{ch:decision} & $\partial F/\partial c=1$; degenerate zeros of a one-dimensional network occur only at
the critical values (\cref{thm:boundary}(a)); the two chambers of $x_1^2-x_2^2+c$ (\cref{exer:crossex});
three sectors at a triple point (\cref{exer:triple}); the submersion of \cref{prop:walls}\\
\ref{ch:paths} & component counts, critical points and indices in \cref{ex:twoblobs}; the Euler
characteristic formula of \cref{prop:chi} for $d\le6$; the Hessian of the height function on the trace
(\cref{exer:hessgraph}); births and deaths of boundary points on a circle (\cref{exer:d1path})\\
\ref{ch:loss} & $\rk J=N$ and the dimension $P-N$ of the solution set; the Hessian $J^{\mathsf T}J$ with $4$
positive and $9$ zero eigenvalues (\cref{thm:interp}); the inequality of \cref{prop:PL}; sublevel
components, the barrier and the Hessians of \cref{ex:hyperbola}; \cref{prop:notproper}; \cref{exer:deeplin};
invariance of the loss under the symmetries\\
\ref{ch:isotopy} & random first layers of width $3$ embed a circle in $\R^5$ (\cref{prop:firstlayer}), and a
coordinate projection that does not (\cref{exer:proj}); the convex hull property of \cref{thm:obstruction}
for random injective networks; order preservation on the line; positivity of the Jacobian determinant and
Liouville's formula for a planar neural ODE (\cref{prop:node}); the swap of \cref{exer:swap}; invariance of
the linking number under a neural ODE in $\R^3$ (\cref{cor:nodeobst}); separation of nested circles after
augmentation (\cref{cor:separate})\\
\ref{ch:degree} & convergence of the fixed-point inversion and the determinant bound for a residual block
(\cref{prop:iresnet}, \cref{exer:det}); a non-proper local diffeomorphism (\cref{thm:hadamard}); odd numbers
of equilibria with signed count $1$ in one and two dimensions (\cref{thm:deq})\\
\bottomrule
\end{longtable}
}
\chapter{Solutions to the Exercises}\label{app:solutions}

Solutions to all exercises follow, by chapter. Where a solution relies on a computation, the computation is
also carried out by \texttt{verify-parametrized-networks.py}.
\section*{Chapter 1}

\sol{1.1} $(2,3,1)$: $P=3\cdot3+1\cdot4=13$, $H=3$. $(2,4,4,1)$: $P=4\cdot3+4\cdot5+1\cdot5=37$, $H=8$.
$(d,k,1)$: $P=k(d+1)+(k+1)=k(d+2)+1$, $H=k$.

\sol{1.2} Let $(\Theta,F)\colon X\to Y$, $(\Theta',G)\colon Y\to Z$, $(\Theta'',K)\colon Z\to V$. Both
bracketings have underlying function $(\theta,\theta',\theta'',x)\mapsto K(\theta'',G(\theta',F(\theta,x)))$,
on the parameter spaces $(\Theta\times\Theta')\times\Theta''$ and $\Theta\times(\Theta'\times\Theta'')$
respectively. The reassociation diffeomorphism $r$ satisfies $F_{\text{left}}=F_{\text{right}}\circ(r\times
\id_X)$, so $r$ and $r^{-1}$ are $2$-cells, mutually inverse.

\sol{1.3} The diagonal $\delta\colon\Theta\to\Theta\times\Theta$ gives a $2$-cell
$(\Theta,(F\circ F)\circ(\delta\times\id))\Rightarrow(\Theta\times\Theta,F\circ F)$, and its source is
$(\theta,x)\mapsto F(\theta,F(\theta,x))$, the weight-tied network. By the chain rule, the derivative of
$\theta\mapsto G(\delta(\theta))$ is $\partial_1G+\partial_2G$ evaluated on the diagonal, for
$G(\theta_1,\theta_2)=F(\theta_2,F(\theta_1,x))$.

\sol{1.4} $F(\theta,x)=\theta+x^3$. Every member has the degenerate critical point $x=0$. A function is
Morse when its $1$-jet prolongation is transverse to the set of jets with vanishing derivative; the family
of prolongations $(\theta,x)\mapsto(x,\theta+x^3,3x^2)$ is not transverse to $\{\text{third coordinate}=0\}$,
since the third coordinate has vanishing derivative in both $\theta$ and $x$ at $x=0$. The parameter moves
the value of the function but not its derivative, so the family is rich for conditions on values
(\cref{prin:rich}) and not for conditions on first derivatives. Adding a parameter $\theta_1x$ repairs this:
$\theta_0+\theta_1x+x^3$ is Morse for $\theta_1\ne0$.
\section*{Chapter 2}

\sol{2.1} Let $z=Wx+b$ and $h=\sigma(z)$. The lens of the layer has forward part $(W,b,x)\mapsto h$ and
backward part
$(W,b,x,w)\mapsto\bigl(\delta x^{\mathsf T},\ \delta,\ W^{\mathsf T}\delta\bigr)$ with
$\delta=\sigma'(z)\odot w$: the three components are the covectors for $W$, $b$ and $x$. The output layer
$(a,c;h)\mapsto a^{\mathsf T}h+c$ has backward part $(a,c,h,w)\mapsto(wh,\ w,\ wa)$. The loss
$\frac12(u-y)^2$ has gradient $e=u-y$ in $u$. Composing: the covector entering the output layer is $e$, so
$\nabla_a=e\,h$, $\nabla_c=e$, and the covector passed back is $e\,a$; then
$\delta=\sigma'(z)\odot(e\,a)$, $\nabla_W=\delta x^{\mathsf T}$, $\nabla_b=\delta$.

\sol{2.2} $\ker JJ^{\mathsf T}=\ker J^{\mathsf T}$ since $\langle v,JJ^{\mathsf T}v\rangle=|J^{\mathsf T}v|^2$;
so $\rk K=\rk J^{\mathsf T}=\rk J$. For scalar output $K$ is $n\times n$, invertible if and only if $\rk J=n$,
that is, if and only if the columns of $J$ span $\R^n$; the $k$-th column is the vector
$\bigl(\partial F/\partial\theta_k(\theta,x_i)\bigr)_{i}$.

\sol{2.3} By \DT{Prop.~2.9} the $2$-jet of $\lambda\circ\ev$ at $\theta$ has linear part $C A$ and quadratic part
$C\circ B+E\circ(A\times A)$, where $(A,B)$ is the $2$-jet of $\ev$ (so $A=J$ and $B=(\nabla^2\ev_k)_k$) and
$(C,E)$ that of $\lambda$ (so $C=\nabla\lambda^{\mathsf T}$, $E=\nabla^2\lambda$). The quadratic part is
$\sum_k\partial_k\lambda\,\nabla^2\ev_k+J^{\mathsf T}\nabla^2\lambda\,J$.

\sol{2.4} $\ev(\theta)=\Phi\theta$ with $\Phi$ the $n\times P$ matrix of rows $\phi(x_i)^{\mathsf T}$; so
$J=\Phi$ for all $\theta$ and $K=\Phi\Phi^{\mathsf T}$, the Gram matrix of the features. The fibres of $\ev$
are the affine subspaces $\theta_0+\ker\Phi$, of dimension $P-\rk\Phi$. The realization map is linear with
kernel $\{\theta:\langle\theta,\phi(x)\rangle=0\ \forall x\}$, so the functional dimension is the dimension
of the span of $\phi(X)$.
\section*{Chapter 3}

\sol{3.1} $F(\theta,x)=\sum_ja_j\sigma(\langle w_j,x\rangle+b_j)+c$. (i) Permuting the index $j$ reorders the
sum. (ii) If $\sigma$ is odd, $(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.
(iii) If $\sigma(\lambda s)=\lambda\sigma(s)$ for $\lambda>0$, then
$(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.

\sol{3.2} The columns of $J$ belonging to $(w_j,b_j)$ are
$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i$ and
$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i$; if $a_j=0$ these $d+1$ columns vanish, so
$\rk J\le P-(d+1)$. If $(w_j,b_j)=(w_k,b_k)$, the columns for $a_j$ and $a_k$ are both
$\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i$, so two columns coincide and $\rk J\le P-1$. (In the second
case the columns for $(w_j,b_j)$ and $(w_k,b_k)$ are also proportional, with ratio $a_j/a_k$, so the rank
drops by $d+2$ in all.)

\sol{3.3} $\theta=(w,b,a,c)$ and $\xi=(w,b,-a,0)$. Where $wx+b>0$, $F=a(wx+b)+c$ and
$D_\theta F=(ax,\ a,\ wx+b,\ 1)$, so $D_\theta F\cdot\xi=awx+ab-a(wx+b)=0$. Where $wx+b<0$, $F=c$ and
$D_\theta F=(0,0,0,1)$, again orthogonal to $\xi$. Here $P=4$, $H=1$, so the functional dimension is at most
$3$; it equals $3$ when $a\ne0$ and $w\ne0$, since the function $x\mapsto a\,\relu(wx+b)+c$ is determined by
its breakpoint $-b/w$, the slope $aw$ on the active side, and $c$ (and the side, which is discrete).

\sol{3.4} $(2,3,1)$: one hidden layer of $3$ units, $|G|=2^3\cdot3!=48$. $(2,4,4,1)$: two hidden layers of
$4$ units, $|G|=(2^4\cdot4!)^2=384^2=147456$.
\section*{Chapter 4}

\sol{4.1} By \cref{thm:boundary}(c), for almost every $\theta$ the set $B_\theta$ is a compact neat
$0$-dimensional submanifold of $[-R,R]$: a finite set of interior points at which $f_\theta'\ne0$, with
$f_\theta(\pm R)\ne0$. At each such point $f_\theta$ changes sign. So the number of zeros is even if
$f_\theta(-R)$ and $f_\theta(R)$ have the same sign and odd otherwise.

\sol{4.2} A compact neat $1$-dimensional submanifold of the disc: a finite disjoint union of circles in the
interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such
configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints
by non-crossing arcs.

\sol{4.3} $d_xF=(2x_1,-2x_2)$ vanishes only at $x=0$, where $F=c$. So $\Sigma=\{(0,0)\}$ and $\Delta=\{0\}$.
There are two chambers, $c>0$ and $c<0$. In each the boundary is a hyperbola; for $c<0$ its branches open
along the $x_1$-axis ($x_1^2=x_2^2-c\ge-c$) and for $c>0$ along the $x_2$-axis. The region $\{f<0\}$ is connected for
$c<0$ (it contains the origin) and has two components for $c>0$. (The plane is not compact; the statement about chambers holds here
because the family is a product near infinity up to scaling. Restricted to a disc the same description
holds.)

\sol{4.4} At a triple point $p$, $f_\theta$ is transverse to $A_{\{1,2,3\}}$, the line $y_1=y_2=y_3$ in $\R^3$,
of codimension $2$. So the map $x\mapsto(f_1-f_2,f_2-f_3)$ from the plane to $\R^2$ is a local
diffeomorphism at $p$, and in the coordinates $(u,v)=(f_1-f_2,f_2-f_3)$ the decision regions are
$\{u>0,\ u+v>0\}$, $\{u<0,\ v>0\}$ and $\{v<0,\ u+v<0\}$: three sectors bounded by the rays
$\{u=0,v>0\}$, $\{v=0,u>0\}$ and $\{u+v=0,u<0\}$.
\section*{Chapter 5}

\sol{5.1} $g=(x_1^2-1)^2+x_2^2$ has $\nabla g=(4x_1(x_1^2-1),2x_2)$, vanishing at $(0,0)$ and $(\pm1,0)$. The
Hessian is $\operatorname{diag}(12x_1^2-4,\,2)$: at $(\pm1,0)$ it is $\operatorname{diag}(8,2)$, index $0$,
value $0$; at the origin $\operatorname{diag}(-4,2)$, index $1$, value $1$. The sublevel set $\{g\le t\}$ is
empty for $t<0$, two discs around $(\pm1,0)$ for $0<t<1$ (on the line $x_2=0$ it is
$\{|x_1^2-1|\le\sqrt t\}$, two intervals when $t<1$), and one disc for $t>1$. Its Euler characteristic is
$0,2,1$, consistent with attaching two $0$-cells and then one $1$-cell.

\sol{5.2} Here $B_t$ is a finite set and $W_\gamma$ is a compact $1$-manifold with boundary $B_0\sqcup B_1$.
A critical point of $\tau$ on a $1$-manifold is a local minimum or maximum of $t$ along a curve: two points
of $B_t$ appear together or disappear together (surgery with $\lambda=0$ or $1$ on a $0$-manifold: add or
remove an $S^0$). Between critical times the points move continuously. If $X$ is a circle, $f_t$ changes
sign at each point of $B_t$ and returns to its original sign after one circuit, so $\#B_t$ is even for every
regular $t$.

\sol{5.3} Two. By \cref{prop:chi} with $d=3$, the Euler characteristic of the region changes by
$\pm(-1)^k$ at each surgery; the solid torus has $\chi=0$ and needs at least one $0$-handle to be non-empty,
so at least one more handle of odd index is needed. A $0$-handle (a ball appears, $k=0$) followed by a
$1$-handle attached to it (the ball becomes a solid torus, $k=1$) suffices. The boundary goes from empty to
a sphere to a torus.

\sol{5.4} Write $\Phi=\Phi_\gamma$ and let $t(x)$ be defined by $\Phi(t(x),x)=0$ near $x_i$, with
$t(x_i)=t_i$. Differentiating once, $\Phi_t\,\partial_jt+\partial_j\Phi=0$, so $\partial_jt(x_i)=0$ because
$d_x\Phi=0$ there. Differentiating again and evaluating at $x_i$, where the terms containing first
derivatives of $t$ vanish, gives $\Phi_t\,\partial_j\partial_kt+\partial_j\partial_k\Phi=0$. Hence
$\nabla^2t(x_i)=-\nabla^2_x\Phi(t_i,x_i)/\Phi_t(t_i,x_i)$.
\section*{Chapter 6}

\sol{6.1} $\ev(\theta)=\Phi\theta$ with $\Phi$ of rank $n$. Every $y$ is a regular value;
$M_y=\theta^*+\ker\Phi$ is an affine subspace of dimension $P-n$, where $\theta^*$ is any solution, for
instance the minimum-norm one $\Phi^{\mathsf T}(\Phi\Phi^{\mathsf T})^{-1}y$. The Hessian is the constant
matrix $\Phi^{\mathsf T}\Phi$, of rank $n$, with kernel $\ker\Phi$.

\sol{6.2} $\nabla\loss=(ab-1)(b,a)$, which vanishes when $ab=1$ or $(a,b)=0$. The Hessian is
$\bigl(\begin{smallmatrix}b^2&2ab-1\\2ab-1&a^2\end{smallmatrix}\bigr)$. On $ab=1$ it is
$\bigl(\begin{smallmatrix}b^2&1\\1&a^2\end{smallmatrix}\bigr)$, of determinant $a^2b^2-1=0$ and trace
$a^2+b^2>0$: rank $1$, with kernel spanned by $(a,-b)$, the tangent to the hyperbola. At the origin it is
$\bigl(\begin{smallmatrix}0&-1\\-1&0\end{smallmatrix}\bigr)$, with eigenvalues $\pm1$: a saddle of index
$1$ with value $\frac12$. The sublevel set is $\{1-\sqrt{2c}\le ab\le1+\sqrt{2c}\}$. For $c<\frac12$ the
lower bound is positive, and the set lies in the two quadrants $ab>0$, one component in each. For
$c\ge\frac12$ it contains the axes and is connected. Any path from $(1,1)$ to $(-1,-1)$ must cross the set
$ab=0$, where $\loss=\frac12$.

\sol{6.3} The flow line $\gamma$ has length at most $\ell=\sqrt{2\loss(\theta_0)/\mu}$ while it stays in $U$.
If the closed ball of radius $\ell$ about $\theta_0$ lies in $U$, the flow line cannot leave $U$: at the
first exit time it would be at distance at most its length, which is at most $\ell$, from $\theta_0$,
hence still inside. So the flow line is defined for all time, has finite length, and therefore converges; the
limit $\theta_\infty$ satisfies $|\theta_\infty-\theta_0|\le\ell$ and $\loss(\theta_\infty)=\lim
\loss(\gamma(t))=0$.

\sol{6.4} $\loss=\frac12(\prod a_i-1)^2$ and $M_y=\{\prod a_i=1\}$, the preimage of the regular value $1$ of
the product map (its gradient at a point with all $a_i\ne0$ is non-zero); so $\dim M_y=k-1$. On $M_y$ every
$a_i\ne0$ and the number of negative $a_i$ is even; each admissible sign pattern gives one component (a
graph over the positive orthant in the first $k-1$ coordinates), and there are $2^{k-1}$ sign patterns with
an even number of minus signs. The gradient is $(\prod a_i-1)\bigl(\prod_{j\ne i}a_j\bigr)_i$, which vanishes
at the origin for $k\ge2$; the Hessian at the origin has entries
$-\partial_i\partial_j\prod a_l$, which are zero for $k\ge3$. So the origin is a critical point with zero
Hessian.
\section*{Chapter 7}

\sol{7.1} Let $\varphi^t$ be the flow. If $x<y$ then $\varphi^t(x)\ne\varphi^t(y)$ for all $t$ by uniqueness
of solutions, so $\varphi^t(y)-\varphi^t(x)$ is continuous in $t$, never zero, and positive at $t=0$; hence
positive at $t=1$.

\sol{7.2} Let $\rho\colon[0,\infty)\to[0,1]$ be smooth with $\rho=1$ on $[0,2]$ and $\rho=0$ on $[3,\infty)$,
and let $v(x)=\pi\rho(|x|)\,(-x_2,x_1)$. The flow preserves $|x|$, since $v$ is tangent to circles, and on
$\{|x|\le2\}$ it is rotation with angular speed $\pi$. At time one the points $(\pm1,0)$ have been rotated
by the angle $\pi$ and are exchanged.

\sol{7.3} Let $\varphi_t$ be the diffeotopy and $K_t=\varphi_t(K)$, $L_t=\varphi_t(L)$; they are disjoint for
every $t$ because $\varphi_t$ is injective. The Gauss maps
$G_t\colon K\times L\to S^2$, $G_t(x,y)=(\varphi_t(y)-\varphi_t(x))/|\varphi_t(y)-\varphi_t(x)|$, form a
homotopy, and the degree is a homotopy invariant by \DT{Thm.~7.2}.

\sol{7.4} The first projection is $t\mapsto(\cos t,\sin t)$, the standard embedding of the circle. The second
is $t\mapsto(\cos2t,\sin2t)$, which takes the same value at $t$ and $t+\pi$; it is an immersion but not
injective, a double cover of the circle.
\section*{Chapter 8}

\sol{8.1} For fixed $y$ the map $T(x)=y-g(x)$ satisfies $|T(x)-T(x')|\le L|x-x'|$, so it has a unique fixed
point $x^*$, which is the unique solution of $x+g(x)=y$, and $|x_j-x^*|=|T^j(x_0)-T^j(x^*)|\le L^j|x_0-x^*|$.
So $f$ is a bijection. And $|f(x)-f(x')|\ge|x-x'|-|g(x)-g(x')|\ge(1-L)|x-x'|$, which says that $f^{-1}$ is
Lipschitz with constant $(1-L)^{-1}$.

\sol{8.2} The singular values of $1+Dg$ are at least $1-L$, since $|(1+Dg)v|\ge|v|-|Dg\,v|\ge(1-L)|v|$. The
absolute value of the determinant is the product of the singular values, so it is at least $(1-L)^d$; and
the determinant is positive by \cref{prop:iresnet}(b).

\sol{8.3} $z=0$ is always a fixed point, with $1-g'(0)=1-a$. For $a\le1$ it is the only one: $\tanh(az)$ is
strictly concave for $z>0$ with slope $a\le1$ at $0$, so $\tanh(az)<z$ for $z>0$, and symmetrically for
$z<0$. For $a>1$, $\tanh(az)>z$ for small $z>0$ and $\tanh(az)<1<z$ for $z>1$, so there is a fixed point
$z^*\in(0,1)$, unique by concavity, and $-z^*$ by symmetry. At $z^*$ the graph of $\tanh(az)$ crosses the
diagonal from above, so $g'(z^*)<1$ and the sign is $+$; at $0$ the sign of $1-a$ is $-$. The signed count is
$1-1+1=1$. For $a<1$ it is the single $+1$. At $a=1$ the fixed point $0$ is degenerate ($1-g'(0)=0$): this
is the exceptional parameter at which the pair $\pm z^*$ is born, a pitchfork, which is not a generic
birth--death because of the symmetry $z\mapsto-z$ present when $u=0$.

\sol{8.4} A point $y$ lies in $\operatorname{supp}(h_*\mu)$ if and only if every open neighbourhood $V$ of $y$
has $\mu(h^{-1}(V))>0$. As $h$ is a homeomorphism, the sets $h^{-1}(V)$ are exactly the open neighbourhoods of
$h^{-1}(y)$; so the condition is $h^{-1}(y)\in\operatorname{supp}\mu$. If $h_*\mu=\nu$ then
$h$ restricts to a homeomorphism from $\operatorname{supp}\mu$ to $\operatorname{supp}\nu$.
\backmatter
\chapter{References}
\begingroup\small
\renewcommand{\chapter}[2]{}
\begin{thebibliography}{CGGWZ22}\setlength{\itemsep}{1pt}
\newcommand{\bi}[1]{\bibitem[#1]{#1}}
\bi{BGC19} J.~Behrmann, W.~Grathwohl, R.~T.~Q.~Chen, D.~Duvenaud, J.-H.~Jacobsen, Invertible residual networks, in \emph{ICML 2019}, PMLR 97, 573--582.
\bi{BS14} M.~Bianchini, F.~Scarselli, On the complexity of neural network classifiers: a comparison between shallow and deep architectures, \emph{IEEE Trans.\ Neural Netw.\ Learn.\ Syst.} 25 (2014), 1553--1565.
\bi{CCDD20} R.~Cornish, A.~Caterini, G.~Deligiannidis, A.~Doucet, Relaxing bijectivity constraints with continuously indexed normalising flows, in \emph{ICML 2020}, PMLR 119.
\bi{CCG20} R.~Cockett, G.~Cruttwell, J.~Gallagher, J.-S.~P.~Lemay, B.~MacAdam, G.~Plotkin, D.~Pronk, Reverse derivative categories, in \emph{Computer Science Logic 2020}, LIPIcs 152, 18:1--18:16.
\bi{CGGWZ22} G.~S.~H.~Cruttwell, B.~Gavranovi\'c, N.~Ghani, P.~Wilson, F.~Zanasi, Categorical foundations of gradient-based learning, in \emph{ESOP 2022}, Lecture Notes in Computer Science 13240, Springer, 1--28.
\bi{Coo21} Y.~Cooper, Global minima of overparameterized neural networks, \emph{SIAM J.~Math.\ Data Sci.} 3 (2021), 676--691.
\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, in \emph{NeurIPS 2018}.
\bi{DDT19} E.~Dupont, A.~Doucet, Y.~W.~Teh, Augmented neural ODEs, in \emph{NeurIPS 2019}.
\bi{DTHC} Anonymous, \emph{Differential Topology in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.
\bi{DVSH18} F.~Draxler, K.~Veschgini, M.~Salmhofer, F.~Hamprecht, Essentially no barriers in neural network energy landscape, in \emph{ICML 2018}, PMLR 80.
\bi{EKSN22} R.~Entezari, H.~Sedghi, O.~Saukh, B.~Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, in \emph{ICLR 2022}.
\bi{FA00} K.~Fukumizu, S.~Amari, Local minima and plateaus in hierarchical structures of multilayer perceptrons, \emph{Neural Networks} 13 (2000), 317--327.
\bi{Fef94} C.~Fefferman, Reconstructing a neural net from its output, \emph{Rev.\ Mat.\ Iberoamericana} 10 (1994), 507--555.
\bi{FMN16} C.~Fefferman, S.~Mitter, H.~Narayanan, Testing the manifold hypothesis, \emph{J.~Amer.\ Math.\ Soc.} 29 (2016), 983--1049.
\bi{FST19} B.~Fong, D.~I.~Spivak, R.~Tuy\'eras, Backprop as functor: a compositional perspective on supervised learning, in \emph{LICS 2019}, IEEE, 2019.
\bi{Fuk96} K.~Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network, \emph{Neural Networks} 9 (1996), 871--879.
\bi{GIP18} T.~Garipov, P.~Izmailov, D.~Podoprikhin, D.~Vetrov, A.~G.~Wilson, Loss surfaces, mode connectivity, and fast ensembling of DNNs, in \emph{NeurIPS 2018}.
\bi{GL22} J.~E.~Grigsby, K.~Lindsey, On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks, \emph{SIAM J.~Appl.\ Algebra Geom.} 6 (2022), 216--242.
\bi{GLMW22} J.~E.~Grigsby, K.~Lindsey, R.~Meyerhoff, C.~Wu, Functional dimension of feedforward ReLU neural networks, preprint, arXiv:2209.04036.
\bi{Hir76} M.~W.~Hirsch, \emph{Differential Topology}, Graduate Texts in Mathematics 33, Springer, 1976.
\bi{Hor91} K.~Hornik, Approximation capabilities of multilayer feedforward networks, \emph{Neural Networks} 4 (1991), 251--257.
\bi{HS17} B.~Hanin, M.~Sellke, Approximating continuous functions by ReLU nets of minimal width, preprint, arXiv:1710.11278.
\bi{JGH18} A.~Jacot, F.~Gabriel, C.~Hongler, Neural tangent kernel: convergence and generalization in neural networks, in \emph{NeurIPS 2018}.
\bi{Joh19} J.~Johnson, Deep, skinny neural networks are not universal approximators, in \emph{ICLR 2019}.
\bi{LZB22} C.~Liu, L.~Zhu, M.~Belkin, Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, \emph{Appl.\ Comput.\ Harmon.\ Anal.} 59 (2022), 85--116.
\bi{Ngu19} Q.~Nguyen, On connected sublevel sets in deep learning, in \emph{ICML 2019}, PMLR 97.
\bi{NZL20} G.~Naitzat, A.~Zhitnikov, L.-H.~Lim, Topology of deep neural networks, \emph{J.~Mach.\ Learn.\ Res.} 21 (2020), no.~184, 1--40.
\bi{PL20} M.~Phuong, C.~H.~Lampert, Functional vs.\ parametric equivalence of ReLU networks, in \emph{ICLR 2020}.
\bi{SGJ21} B.~\c{S}im\c{s}ek, F.~Ged, A.~Jacot, F.~Spadaro, C.~Hongler, W.~Gerstner, J.~Brea, Geometry of the loss landscape in overparameterized neural networks: symmetries and invariances, in \emph{ICML 2021}, PMLR 139.
\bi{Sus92} H.~J.~Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, \emph{Neural Networks} 5 (1992), 589--593.
\bi{Wat09} S.~Watanabe, \emph{Algebraic Geometry and Statistical Learning Theory}, Cambridge University Press, 2009.
\bi{ZGUA20} H.~Zhang, X.~Gao, J.~Unterman, T.~Arodz, Approximation capabilities of neural ODEs and invertible residual networks, in \emph{ICML 2020}, PMLR 119.
\end{thebibliography}
\endgroup
\end{document}

End of manuscript · Back to top ↑