Independent scholarship · Open accessThe monthly digest

Full manuscript · browser edition

Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates

Continuous, single-column reading. Equations, tables, proofs, references and appendices stay in document order.

Contents · 17 sections

Partial Differential Equations in Machine Learning and AI:
Naturality, Well-Posedness, and Spectral Rates

Abstract. Convolutional layers are difference operators, depth is a time variable, generative diffusion is the heat equation, message passing is a graph Laplacian, and a kernel is a Green’s function. We organize these identifications by three questions that the theory of partial differential equations asks of any construction. Is it natural, that is, does it commute with the maps of domains, resolutions and symmetries along which one wants to move it? Is it well-posed, that is, does the output depend continuously on the input? At what rate does it act on each mode? Each qualitative answer is paired with a computed one. Naturality: half-sample reflection models the Neumann condition. Circular layers preserve the reflected subspace exactly when the periodized kernels are symmetric. zero padding is the Dirichlet condition only for three-point stencils; Fourier multipliers commute with resampling, stencils do so to second order, squaring needs more than 3K3K points for band KK, and a rectifier is aliased at order N−2N^{-2} on every grid. Well-posedness: reversing diffusion amplifies mode 1010 by 5×10215\times10^{21}, while the reverse flow driven by the score moves samples by 0.260.26 per unit of score error, the difficulty reappearing as a score of slope 1/t21/t^2. Forward Euler amplifies nonzero oscillatory modes. In the stable range 0<hω<20<h\omega<2, the leapfrog condition number is at most (1+hω/2)/(1−hω/2)(1+h\omega/2)/(1-h\omega/2). A strong-form loss squares the condition number of the energy loss, 414414 against 171,682171{,}682 on 3131 unknowns. Rates: oversmoothing proceeds at 1−τλ21-\tau\lambda_2 per layer, with depth scales 415415, 55 and 11 on a path, a hypercube and a complete graph of 3232 nodes; sheaf diffusion converges to global sections, which exist for 1616 of the 128128 sign sheaves on a small graph; kernel regression learns mode kk at rate 1/(1+k2)1/(1+k^2) plus aliases; and the spectral gap of Langevin dynamics in a double well approaches the Kramers rate at low noise. All numerical statements are recomputed by a companion script with 4242 checks. No new theorem is claimed; the contribution is the organization, the exact constants, and the three questions asked uniformly.

Introduction

The companion volume [36] retells an introductory course on partial differential equations [42] around one question: which constructions are natural for maps of domains, of equations and of function spaces. This paper asks the same question of machine learning, where the same equations have been rediscovered piecemeal. A convolutional layer with a three-point kernel is a difference operator, and its padding rule is a boundary condition [43]. A residual network is a difference scheme in a time variable called depth [14, 18, 10]. A diffusion model noises data with the heat equation and samples by reversing it [44, 19, 40]. Message passing on a graph is a discrete diffusion [11], or a discrete wave equation [38], or a diffusion of sections of a sheaf [17, 7]. Kernel regression inverts an elliptic operator, physics-informed networks minimize the residual of an equation [39], and operator learning approximates its solution map [31, 25, 30]. A recent survey organizes the last two families [13].

Three questions organize the theory of the equations themselves, and we ask each of them of each construction.

Naturality. A domain can be reflected, translated, refined or glued, and a construction is natural if it commutes with the corresponding maps. Equivariance [9, 4] is naturality for symmetries; discretization invariance and the absence of aliasing [23, 6] are naturality for changes of resolution.

Well-posedness. Hadamard’s criterion asks that a solution exist, be unique, and depend continuously on the data. Learning problems fail the third condition in characteristic ways: by running a parabolic equation backwards, by discretizing a conservative system with a dissipative scheme or the reverse, or by squaring an operator in the loss.

Rates. For linear constructions the answer to “how fast” is a spectrum. Oversmoothing, spectral bias, mixing of stochastic gradient descent and the rank of a learned operator are all statements about eigenvalues of a Laplacian.

What is and is not claimed.

Every statement below carries a status tag: classical, literature, reformulation, observation or programme. Almost all are classical or known; where the form or the constant seems not to be standard we say “observation”. The checks, tagged like [N3], are performed by the companion script verify-pde-learning.py, which uses only the Python standard library and a fixed seed, and are listed in 15. They recompute instances; they are evidence and not proofs. Most models are linear and one-dimensional, so that every number is exact or accurate to many digits; 14 says what this leaves out. The references were assembled from memory, except for [13, 6, 7], which were checked online on 17 November 2026.

Setting

Let Dom\mathsf{Dom} be a category of domains: intervals, circles, grids, graphs, with the maps of interest between them (inclusions, coverings, symmetries, refinements). A signal space is a functor F ⁣:Domop→VectF\colon\mathsf{Dom}^{\mathrm{op}}\to\mathsf{Vect}, such as functions on the domain with restriction or pullback. A layer is a family of maps TD ⁣:F(D)→F(D)T_D\colon F(D)\to F(D), not necessarily linear.

Definition 1. The layer TT is natural for a map f ⁣:D→D′f\colon D\to D' if the square image commutes. Its defect on an input xx is ∥f∗TD′x−TDf∗x∥\lVert f^*T_{D'}x-T_Df^*x\rVert, relative to the norm of either term.

For ff a symmetry of DD this is equivariance. For ff the inclusion of a coarse grid in a fine one, with FF restricted to band-limited signals, it is discretization invariance. For ff a covering map and FF replaced by its invariant part, it is the method of images, with which we begin. A depth semigroup is a family TtT^t with Tt+s=TtTsT^{t+s}=T^tT^s, and a network of NN layers of step hh is a discretization of TNhT^{Nh}; naturality in tt is consistency of a scheme, and well-posedness is its stability [36, Ch. 12].

Padding is a boundary condition

Let x∈Rnx\in\mathbb{R}^n and let w=(w−r,…,wr)w=(w_{-r},\dots,w_r) be a kernel. The layer is (Cwx)i=∑jwjxi+j(C_wx)_i=\sum_jw_jx_{i+j}, and a padding rule supplies the values xix_i for ii outside 0,…,n−10,\dots,n-1: zero (xi=0x_i=0), circular (xi+n=xix_{i+n}=x_i) or reflect (x−1−i=xix_{-1-i}=x_i and xn+i=xn−1−ix_{n+i}=x_{n-1-i}).

Proposition 2. [classical; [43]] With the kernel (−1,2,−1)(-1,2,-1) the three rules give the Dirichlet, periodic and Neumann difference Laplacians. They are diagonalized by the sine transform sin⁡πk(j+1)n+1\sin\frac{\pi k(j+1)}{n+1}, the discrete Fourier transform, and the cosine transform cos⁡πk(j+1/2)n\cos\frac{\pi k(j+1/2)}n, with eigenvalues 4sin⁡2πk2(n+1)4\sin^2\frac{\pi k}{2(n+1)}, 4sin⁡2πkn4\sin^2\frac{\pi k}n and 4sin⁡2πk2n4\sin^2\frac{\pi k}{2n}. [N1, N2]

Four ways of continuing a signal of six samples (shaded) beyond its ends. Reflect padding is the even extension about the half-sample points (dashed), which is the Neumann condition. The odd extension about the neighbouring grid points is the Dirichlet condition; zero padding agrees with it only on the first sample outside. Circular padding is the periodic extension.

The method of images [36, Thm. 3.5] explains this construction. In 1, the padding reflects about half-sample points, repeating the end samples. Let E ⁣:Rn→R2nE\colon\mathbb{R}^n\to\mathbb{R}^{2n} be the even extension Ex=(x0,…,xn−1,xn−1,…,x0)Ex=(x_0,\dots,x_{n-1},x_{n-1},\dots,x_0), regarded as a signal on the cycle Z/2n\mathbb{Z}/2n, and let RR be the restriction to the first nn entries. The image of EE is the space of signals invariant under the reflection ρ ⁣:i↦2n−1−i\rho\colon i\mapsto2n-1-i.

Theorem 3 (Reflect padding is the Neumann condition). [reformulation; cf. [43]; [36], Thm. 3.5] Write CwreflC^{\mathrm{refl}}_w and CwcircC^{\mathrm{circ}}_w for the layer with reflect padding on Rn\mathbb{R}^n and circular padding on R2n\mathbb{R}^{2n}.

  1. Cwrefl=R CwcircEC^{\mathrm{refl}}_w=R\,C^{\mathrm{circ}}_wE for every kernel.

  2. ρ Cwcircρ=Cwˇcirc\rho\,C^{\mathrm{circ}}_w\rho=C^{\mathrm{circ}}_{\check w} with wˇj=w−j\check w_j=w_{-j}. Hence if ww is symmetric, CwcircE=E CwreflC^{\mathrm{circ}}_wE=E\,C^{\mathrm{refl}}_w, and a network of symmetric kernels, reflect padding and pointwise nonlinearities is the restriction to invariant signals of a circular network on the doubled cycle.

  3. The circular layer preserves the reflected subspace exactly when the kernel, reduced modulo 2n2n by summing coefficients with the same residue, is symmetric. An asymmetric periodized kernel sends some even input outside that subspace.

Proof. (a) The reflect rule is the definition of EE read at indices outside 0,…,n−10,\dots,n-1. (b) Substituting i↦2n−1−ii\mapsto2n-1-i in the sum reverses the kernel. Pointwise maps commute with EE and RR. For (c), work with the periodized kernel and set a=w−wˇa=w-\check w. If the circular layer preserves the reflected subspace, then CacircC^{\mathrm{circ}}_a annihilates the even input δ0+δ−1\delta_0+\delta_{-1}, by (b). Its output at ii is ai+ai+1=0a_i+a_{i+1}=0. Since a0=0a_0=0, all aia_i vanish. ◻

For a random input and the kernels (0.2,−0.5,1,−0.5,0.2)(0.2,-0.5,1,-0.5,0.2) and (0.2,−0.5,1,0.7,−0.3)(0.2,-0.5,1,0.7,-0.3): identity (a) holds to rounding for both; the output on the double is invariant to rounding for the first and has an anti-invariant part of relative size 1.181.18 for the second; and two layers on the double differ from two reflect-padded layers by 00 when the symmetric kernel comes first and by 0.1000.100 when the other does. [N3] The circular model with symmetric kernels is equivariant under the dihedral group of the doubled cycle. The half-sample construction restricts it to signals fixed by the chosen reflection. The Dirichlet construction uses the odd extension about the neighbouring grid points.

Proposition 4 (Zero padding is Dirichlet only for short kernels). [observation]Let O ⁣:Rn→R2n+2O\colon\mathbb{R}^n\to\mathbb{R}^{2n+2} be the odd extension (0,x,0,−xrev)(0,x,0,-x^{\mathrm{rev}}). For a symmetric kernel of radius one, zero padding equals circular convolution on OxOx restricted to the original positions. For radius two or more it does not: the image construction supplies −x0-x_0 beyond the boundary where zero padding supplies 00. Pointwise maps commute with OO only if they are odd.

The relative difference is 00 for (0.3,1,0.3)(0.3,1,0.3) and 0.1600.160 for the five-point symmetric kernel above. [N4] The Dirichlet image construction can differ from zero padding for wider kernels. A rectifier can also destroy the odd symmetry required by the construction.

Proposition 5 (What padding breaks and keeps). [classical; ](a) Circular convolution commutes with translation. With zero padding the commutator with a shift is supported within the kernel radius of the boundary. (b) A layer with a non-negative symmetric kernel of sum one and reflect padding is doubly stochastic: it conserves the mean and obeys the maximum principle. With zero padding it is substochastic, and the mass remaining after mm layers is the probability that a random walk with steps distributed by the kernel has not left the interval.

For a five-point kernel on 2020 positions the commutator is supported on positions 00, 1818 and 1919. [N5] Position information therefore enters a zero-padded network at the boundary and spreads inwards by one kernel radius per layer, which is the mechanism behind the observation that such networks learn absolute position [21]. For the kernel (14,12,14)(\frac14,\frac12,\frac14) on 3232 positions, 200200 layers with reflect padding change the total mass by 2×10−152\times10^{-15}, and with zero padding leave 0.53200.5320 of it, in agreement with the sum over the sine basis. [N6]

Resolution: what commutes with resampling

Let BKB_K be the trigonometric polynomials of degree at most KK on the circle and SN ⁣:BK→RNS_N\colon B_K\to\mathbb{R}^N the sampling map on NN equally spaced points, injective for N>2KN>2K. An operator TT on functions with discretizations TNT_N is natural for refinement (1) when SNT=TNSNS_NT=T_NS_N on BKB_K; this is the requirement of representation equivalence in [6].

Theorem 6 (Three kinds of layer). [reformulation; cf. [6, 23]] On BKB_K and for N>2KN>2K:

  1. A Fourier multiplier m(D)m(D), discretized by the discrete Fourier transform, is natural exactly.

  2. The three-point stencil for −∂x2-\partial_x^2 has symbol k2sinc⁡2(kh/2)k^2\operatorname{sinc}^2(kh/2), h=2π/Nh=2\pi/N: it is natural to second order.

  3. A polynomial nonlinearity of degree dd followed by projection to BKB_K is natural if N>(d+1)KN>(d+1)K, and for generic inputs only then.

  4. A rectifier followed by projection to BKB_K is natural for no NN; the defect is of order N−2N^{-2}.

Proof. (a) Sampling intertwines the characters eikxe^{ikx}, ∣k∣≤K|k|\le K, with the discrete ones. (b) Apply the stencil to eikxe^{ikx}. (c) The power has band dKdK, and a frequency k′k' is sampled onto k≠k′k\ne k' when NN divides k′−kk'-k; the largest ∣k′−k∣|k'-k| with ∣k∣≤K|k|\le K is (d+1)K(d+1)K. (d) A function with a corner has Fourier coefficients of order k−2k^{-2}, and the sum of those at k+mNk+mN, m≠0m\neq0, is of order N−2N^{-2}. ◻

Aliasing error of a rectifier against the number of grid points, on logarithmic scales: the closed-form case of 6(d) and the largest in-band coefficient error for a random input of band 44. Both follow the slope −2-2.

For a random input of band 55, the multiplier (1+k2)−1(1+k^2)^{-1} computed on 1616 and on 6464 points agrees with the exact result to 10−1410^{-14}; the stencil is in relative error by 0.2210.221 and 0.01530.0153, its symbol at k=5k=5 being 0.71730.7173 and 0.98010.9801 of the true one. [A1] For band 44 the in-band coefficients of the square are exact on 1313 points and wrong by 1.41.4 on 1212: the “two-thirds rule” of spectral methods [35]. [A2] For the rectifier one case is closed-form: the mean of relu⁡(sin⁡x)\operatorname{relu}(\sin x) sampled on NN points is 1Ncot⁡πN\frac1N\cot\frac\pi N instead of 1π\frac1\pi, an error of −π/3N2-\pi/3N^2 to leading order; N2N^2 times the error is −1.058-1.058, −1.050-1.050, −1.048-1.048, −1.047-1.047 for N=8,…,64N=8,\dots,64. [A3] For a random input of band 44 the largest in-band error of the rectified signal is 7.4×10−27.4\times10^{-2}, 9.8×10−39.8\times10^{-3}, 2.1×10−32.1\times10^{-3}, 4.2×10−44.2\times10^{-4}, 1.4×10−41.4\times10^{-4} on 16,…,25616,\dots,256 points (2). [A4] This is the quantitative content of the practice of applying nonlinearities on an oversampled grid and filtering [23]: oversampling by a factor MM buys M2M^{2}, and no finite factor buys exactness. An architecture built from (a) alone is resolution-independent [31]; one with stencils is so to the order of the stencil; one with rectifiers is so only approximately, at every resolution.

Diffusion models

Let data X0∼p0X_0\sim p_0 on the line be noised by Xt=X0+WtX_t=X_0+W_t, so that the density solves the heat equation ∂tp=12∂x2p\partial_tp=\frac12\partial_x^2p, and write s=∂xlog⁡pts=\partial_x\log p_t for the score. Our example is a mixture of two Gaussians, for which everything is closed-form. [S1]

Proposition 7. [classical; [15, 20, 45]] (a) E[X0∣Xt=x]=x+t s(x,t)\mathbb{E}[X_0\mid X_t=x]=x+t\,s(x,t). (b) For every candidate σ\sigma, E∣σ(Xt)+Xt−X0t∣2=E∣σ(Xt)−s(Xt,t)∣2+1t−J(pt),\begin{gathered} \mathbb{E}\bigl|\sigma(X_t)+\tfrac{X_t-X_0}t\bigr|^2\\=\mathbb{E}|\sigma(X_t)-s(X_t,t)|^2+\tfrac1t-J(p_t), \end{gathered} where JJ is the Fisher information: denoising and explicit score matching differ by a constant.

Both hold to rounding in the example; for two-point data at t=12t=\frac12 the constant is 0.9240730.924073 for each of two test functions. [S2, S8] Since 12∂x2p=−∂x(p⋅(−12s))\frac12\partial_x^2p=-\partial_x\bigl(p\cdot(-\frac12s)\bigr), the heat equation is also a continuity equation, and its characteristics are the probability flow x˙=−12s(x,t)\dot x=-\frac12s(x,t) [40].

Theorem 8 (Where the ill-posedness goes). [reformulation; cf. [36], Ex. 2.5; [1, 40]]

  1. The backward heat equation multiplies the kk-th Fourier mode of pTp_T by ek2T/2e^{k^2T/2}: recovering p0p_0 from pTp_T is ill-posed in every Sobolev norm.

  2. Given the score on [t0,T][t_0,T] with t0>0t_0>0, the probability flow is an ordinary differential equation with a Lipschitz field; it transports pTp_T to pt0p_{t_0}, and an error ϵ\epsilon in the score, in the supremum norm, moves each sample by at most C(t0)ϵC(t_0)\epsilon.

  3. For data on the two points ±a\pm a, ∂xs(0,t)=−1/t+a2/t2\partial_xs(0,t)=-1/t+a^2/t^2. The Lipschitz constant of the field is not integrable at t=0t=0, and C(t0)→∞C(t_0)\to\infty as t0→0t_0\to0.

Proof. (a) is Fourier analysis. (b) is the uniqueness theorem for characteristics together with Grönwall’s inequality. (c) Here s=−x/t+(a/t)tanh⁡(ax/t)s=-x/t+(a/t)\tanh(ax/t). ◻

The score s(x,t)=−x/t+tanh⁡(x/t)/ts(x,t)=-x/t+\tanh(x/t)/t of data on the two points ±1\pm1 (dashed), at three noise levels. As t→0t\to0 it becomes a restoring force towards each point, joined by a transition of slope 1/t21/t^2 at the midpoint.

For T=1T=1 and k=10k=10 the factor in (a) is 5.2×10215.2\times10^{21}. In the example the flow from T=1T=1 to 00 preserves the value of the distribution function along five trajectories to 10−1410^{-14}, and a perturbation ϵsin⁡3x\epsilon\sin3x of the score displaces the samples by 0.264 ϵ0.264\,\epsilon. [S3, S4] The slope in (c) is 9090 at t=0.1t=0.1 and 99009900 at t=0.01t=0.01 (3). [S5] The three statements together say that a diffusion model does not solve the backward heat equation. It replaces an ill-posed linear problem for the density by a well-posed nonlinear transport problem for samples, whose coefficient, the score, carries the whole difficulty, concentrated at small times and near the boundaries between modes. Statement (c) is the sample-level form of Hadamard’s example.

Proposition 9 (Entropy). [classical; [41, 2]] (a) Under the heat flow ddth(pt)=12J(pt)\frac d{dt}h(p_t)=\frac12J(p_t). (b) Under the Ornstein–Uhlenbeck flow dX=−X dt+2 dWdX=-X\,dt+\sqrt2\,dW, with Gaussian limit γ\gamma, D(pt ∥ γ)≤e−2tD(p0 ∥ γ)D(p_t\,\Vert\,\gamma)\le e^{-2t}D(p_0\,\Vert\,\gamma), and e2tD(pt ∥ γ)→12(EX0)2e^{2t}D(p_t\,\Vert\,\gamma)\to\frac12(\mathbb{E}X_0)^2.

In the example dh/dt=0.535607dh/dt=0.535607 and J/2=0.535606J/2=0.535606 at t=0.3t=0.3. [S6] The ratio D(pt ∥ γ)/e−2tD(p0 ∥ γ)D(p_t\,\Vert\,\gamma)/e^{-2t}D(p_0\,\Vert\,\gamma) is 0.3530.353, 0.2040.204, 0.1190.119, 0.0880.088 at t=14,12,1,2t=\frac14,\frac12,1,2, and e2tD(pt ∥ γ)=0.03128e^{2t}D(p_t\,\Vert\,\gamma)=0.03128 at t=4t=4 against the limit 0.031250.03125. [S7] The rate 22 is twice the spectral gap of the generator, and it fixes the noising time needed to make the terminal law Gaussian to a given accuracy: 12log⁡(D(p0 ∥ γ)/δ)\frac12\log(D(p_0\,\Vert\,\gamma)/\delta).

Flows and characteristics

A continuous normalizing flow [10] moves samples by x˙=v(x,t)\dot x=v(x,t); the density then solves the continuity equation ∂tp+∂x(pv)=0\partial_tp+\partial_x(pv)=0, of which the trajectories are the characteristics [36, Prop. 1.8].

Proposition 10. [classical; [10]] Along a trajectory ddtlog⁡p(x(t),t)=−∂xv(x(t),t)\frac d{dt}\log p(x(t),t)=-\partial_xv(x(t),t).

For v=0.8tanh⁡(1.3x+0.2)v=0.8\tanh(1.3x+0.2) the exponential of the integrated divergence equals the derivative of the flow map to 10−810^{-8}. [F1] Flow matching [27, 28] chooses a coupling of X0∼p0X_0\sim p_0 and X1∼p1X_1\sim p_1, sets Xt=(1−t)X0+tX1X_t=(1-t)X_0+tX_1, and regresses onto v(x,t)=E[X1−X0∣Xt=x]v(x,t)=\mathbb{E}[X_1-X_0\mid X_t=x].

Theorem 11 (One dimension). [observation]On the line, if vv is regular enough for trajectories to be unique, the flow of the flow-matching field is the increasing rearrangement of p0p_0 onto p1p_1, whatever the coupling.

Proof. The marginal field transports p0p_0 to p1p_1 by construction. Trajectories of a scalar equation cannot cross, so the flow map is increasing, and an increasing map carrying one atomless law to another is unique. ◻

For p0=N(0,1)p_0=N(0,1), p1=N(2,14)p_1=N(2,\frac14) and the independent coupling the field is linear in xx with coefficients rational in tt, and its flow is x↦2+12xx\mapsto2+\frac12x to 10−1310^{-13}. [F2] The conditional paths are straight lines that cross; the marginal characteristics do not. In higher dimensions non-crossing no longer determines the map, and the flow depends on the coupling.

Proposition 12 (A pitchfork in time). [observation]For p0=N(0,1)p_0=N(0,1) and p1p_1 uniform on ±1\pm1, with the independent coupling, v(x,t)=11−t[tanh⁡tx(1−t)2−x].v(x,t)=\frac1{1-t}\Bigl[\tanh\frac{tx}{(1-t)^2}-x\Bigr]. The midpoint x=0x=0 is an attracting equilibrium for t<t∗=3−52=0.382t<t^*=\frac{3-\sqrt5}2=0.382 and a repelling one afterwards, and ∂xv(0,t)∼(1−t)−3\partial_xv(0,t)\sim(1-t)^{-3} as t→1t\to1.

Characteristics of the flow-matching field from N(0,1)N(0,1) to the two points ±1\pm1 (12). The midpoint is attracting until t∗=0.382t^*=0.382 and repelling afterwards; the dotted line is the unstable trajectory. No two characteristics cross.

The field satisfies the continuity equation with ptp_t to 10−1310^{-13}; the slope at the midpoint changes sign at 0.3819660.381966 and is 890890 at t=0.9t=0.9 (4). [F3] This is the transport-side picture of 8(c): mass begins to be committed to one of the two points at a definite time t∗t^*, by the bifurcation that [36, Thm. 14.7] describes for a buckling rod, and the field then steepens without bound as the characteristics focus. Such transition times have been studied for diffusion models in high dimension [5].

Graph networks

Let LL be the Laplacian of a connected graph with eigenvalues 0=λ1<λ2≤⋯≤λn0=\lambda_1<\lambda_2\le\dots\le\lambda_n. The linear part of a message-passing layer is X↦(I−τL)XX\mapsto(I-\tau L)X, one explicit Euler step of the heat equation on the graph [11].

Theorem 13 (Oversmoothing is the spectral gap). [classical; [29, 34]] For 0<τ≤1/λn0<\tau\le1/\lambda_n the non-constant part of XX is multiplied by at most 1−τλ21-\tau\lambda_2 per layer, with equality in the limit, and the Dirichlet energy XTLXX^{\mathsf{T}}LX by at most (1−τλ2)2(1-\tau\lambda_2)^2. The depth at which features become indistinguishable is of order 1/τλ21/\tau\lambda_2.

On a random connected graph with 1212 nodes, λ2=0.6251\lambda_2=0.6251, λn=8.2655\lambda_n=8.2655 and the measured factor is 0.924370.92437, as predicted; 99%99\% of the Dirichlet energy of a random input is gone after 1515 layers, the bound from λ2\lambda_2 alone being 2929. [G1] The depth scale varies by orders of magnitude with the graph: on 3232 nodes, with τ\tau at its largest stable value, it is 415415 layers for a path (λ2=0.00963\lambda_2=0.00963), 55 for the hypercube (λ2=2\lambda_2=2, λn=10\lambda_n=10) and 11 for the complete graph, where a single layer returns the mean exactly (5). [G2] Graphs that are good for propagating information are the ones that oversmooth fastest; the two are the same eigenvalue.

Dirichlet energy of a random signal on 3232 nodes against depth, relative to its initial value. Heat-type layers lose it at a rate set by the spectral gap, slowly on a path and quickly on a hypercube; a wave-type layer on the path (14) keeps it.

A layer of second order in depth behaves as a wave equation [38].

Theorem 14 (Wave layers). [reformulation; cf. [36], Thm. 12.3] The recursion Xm+1=2Xm−Xm−1−τ2LXmX_{m+1}=2X_m-X_{m-1}-\tau^2LX_m conserves τ−2∥Xm+1−Xm∥2+Xm+1TLXm,\tau^{-2}\lVert X_{m+1}-X_m\rVert^2+X_{m+1}^{\mathsf{T}}LX_m, which is positive definite on non-constant signals for τ2λn<4\tau^2\lambda_n<4. For τ2λn>4\tau^2\lambda_n>4 the top mode grows per layer by the modulus of the larger root of ξ2+(τ2λn−2)ξ+1=0\xi^2+(\tau^2\lambda_n-2)\xi+1=0.

Over 500500 layers at τ2λn=3.6\tau^2\lambda_n=3.6 the invariant drifts by 2×10−152\times10^{-15} and the mean Dirichlet energy of the last hundred layers is 1.0011.001 times that of the first hundred: nothing is smoothed. At τ2λn=4.4\tau^2\lambda_n=4.4 the growth factor is 1.86331.8633. [G3] The bound is the Courant–Friedrichs–Lewy condition, and the dichotomy between the two theorems is that of [36, Ch. 4]: a Markov semigroup forgets, a symplectic group does not.

Both layers assume that neighbouring nodes should agree. A cellular sheaf [17] replaces agreement by agreement after transport: each edge carries linear maps from the data at its ends, and the sheaf Laplacian LFL_{\mathcal F} penalizes the mismatch. Its kernel is the space H0H^0 of global sections. The simplest case is a sign σe=±1\sigma_e=\pm1 on each edge, with xTLFx=∑e=uv(xu−σexv)2x^{\mathsf{T}}L_{\mathcal F}x=\sum_{e=uv}(x_u-\sigma_ex_v)^2.

Theorem 15 (Sheaf diffusion). [literature; [17, 7]] Diffusion by LFL_{\mathcal F} converges to the orthogonal projection of the input onto H0H^0. For a sign sheaf on a connected graph, H0H^0 has dimension one if the signs are a coboundary, σuv=ℓuℓv\sigma_{uv}=\ell_u\ell_v, in which case it is spanned by ℓ\ell; otherwise H0=0H^0=0. The obstruction is the class of σ\sigma in H1(G;Z/2)H^1(G;\mathbb{Z}/2), a group of order 2E−V+12^{E-V+1}.

On a random graph with ten nodes, two classes, and signs ++ within and −- between classes, sheaf diffusion converges to the class labelling times ⟨ℓ,X0⟩/n\langle\ell,X_0\rangle/n, to rounding, while ordinary diffusion converges to a constant and loses the classes. Reversing one edge that lies on a cycle makes the smallest eigenvalue 0.16060.1606 and the limit zero; for a cycle of length 77 with one negative edge it is 4sin⁡2π14=0.198064\sin^2\frac\pi{14}=0.19806, the antiperiodic Laplacian. [G4] On a graph with 55 nodes and 77 edges, exactly 16=2V−116=2^{V-1} of the 128128 sign assignments have a non-zero section, and they are exactly those whose Laplacian is singular over the rationals; the 128128 assignments fall into 8=2E−V+18=2^{E-V+1} classes. [G5] This is the same first cohomology that classifies flat gauge fields in [36, Prop. 13.1]: choosing the transport maps of a sheaf network is choosing a connection, and heterophily is non-trivial holonomy. A sign sheaf whose class is non-zero cannot be made consistent by relabelling nodes, and its diffusion has no non-trivial limit.

Depth as time

A residual layer x↦x+hf(x)x\mapsto x+hf(x) is one step of forward Euler for x˙=f(x)\dot x=f(x) [14, 18]. The type of the linearization of ff decides which schemes are admissible, as for partial differential equations [36, Ch. 12].

Theorem 16 (Layers by type). [reformulation; cf. [18, 8]; [36], Thm. 12.2, 12.3]

  1. Parabolic, f=−Axf=-Ax with AA positive definite: forward Euler is stable if and only if h≤2/λmax⁡(A)h\le2/\lambda_{\max}(A).

  2. Hyperbolic, f=Jxf=Jx with JJ antisymmetric with eigenvalues ±iω\pm i\omega: forward Euler multiplies the norm by 1+h2ω2\sqrt{1+h^2\omega^2} per layer, for every hh.

  3. For h,ω>0h,\omega>0, the leapfrog layer q↦q+hpq\mapsto q+hp followed by p↦p−hω2qp\mapsto p-h\omega^2q, with the new qq, has determinant one, conserves p2+ω2q2+hω2pqp^2+\omega^2q^2+h\omega^2pq, and has bounded powers exactly when hω<2h\omega<2.

  4. In this stable range, the Jacobian of NN leapfrog layers in coordinates (ωq,p)(\omega q,p) has condition number at most (1+hω2)/(1−hω2)(1+\frac{h\omega}2)/(1-\frac{h\omega}2), for every NN. The Jacobian of NN parabolic layers with hλmax⁡<1h\lambda_{\max}<1 has condition number ((1−hλmin⁡)/(1−hλmax⁡))N\bigl((1-h\lambda_{\min})/(1-h\lambda_{\max})\bigr)^N.

Proof. (a) and (b) follow from the eigenvalues of I+hDfI+hDf. For (c), direct computation gives the invariant, which is positive definite when hω<2h\omega<2. The update matrix has trace 2−(hω)22-(h\omega)^2 and determinant one. At hω=2h\omega=2 it has a nontrivial Jordan block at −1-1, giving linear growth for some data. For hω>2h\omega>2 an eigenvalue has modulus greater than one. (d): a map that preserves a positive definite quadratic form QQ is conjugate to an orthogonal one by Q1/2Q^{1/2}, so its powers have condition number at most that of QQ. ◻

For a random positive definite AA of size 66 the per-layer factor is 0.98140.9814 at h=1.9/λmax⁡h=1.9/\lambda_{\max} and 1.10001.1000 at 2.1/λmax⁡2.1/\lambda_{\max}. [R1] Forward Euler on a rotation grows by 1.64461.6446 over time 1010 with h=0.1h=0.1 and by 1.05131.0513 with h=0.01h=0.01, that is by ehω2T/2e^{h\omega^2T/2}: the instability vanishes only in the limit of infinitely many layers. [R2] Leapfrog conserves its invariant to 10−1410^{-14} at hω=1.9h\omega=1.9 and grows by 1.87731.8773 per layer at 2.12.1. [R3] With hω=12h\omega=\frac12 the condition number of the product of 5050, 200200 and 800800 leapfrog Jacobians is 1.071.07, 1.301.30 and 1.531.53, below the bound 1.6671.667; for fifty Euler layers on eigenvalues 11 and 1010 with h=0.05h=0.05 it is 8.7×10138.7\times10^{13}. [R4] Since back-propagation multiplies by the transposed Jacobians, (d) is a statement about vanishing and exploding gradients: a symplectic architecture has neither, by the conservation law that a wave equation has and a heat equation lacks. The price is 14: it does not forget either.

Kernels and losses

In the kernel regime, gradient descent on the squared loss moves the function by f↦f−ηK(f−y)f\mapsto f-\eta K(f-y) for a positive kernel operator KK [22]. For a translation-invariant kernel whose Fourier coefficients decay like k−2k^{-2}, as for the simplest rectifier networks on the circle [37], the model case is K=(1−∂x2)−1K=(1-\partial_x^2)^{-1}, the Green’s operator of an elliptic equation.

Theorem 17 (Spectral bias, with aliases). [reformulation; cf. [37, 22]] (a) The residual in mode kk after mm steps is (1−ημk)m(1-\eta\mu_k)^m times its initial value, with μk=(1+k2)−1\mu_k=(1+k^2)^{-1}. (b) On NN equally spaced samples the eigenvalue is μkN=∑m∈Z11+(k+mN)2=πNsinh⁡2πNcosh⁡2πN−cos⁡2πkN.\mu^N_k=\sum_{m\in\mathbb{Z}}\frac1{1+(k+mN)^2}=\frac{\frac\pi N\sinh\frac{2\pi}N}{\cosh\frac{2\pi}N-\cos\frac{2\pi k}N}. (c) The interpolant u=∑iciK(⋅−xi)u=\sum_ic_iK(\cdot-x_i) solves u−u′′=∑iciδxiu-u''=\sum_ic_i\delta_{x_i}.

Statement (b) is the Poisson summation formula of [36, Thm. 4.6]: sampling in space is summation over the dual lattice. On 6464 points μkN/μk\mu^N_k/\mu_k is 1.00161.0016, 1.08521.0852 and 1.23461.2346 at k=1,10,16k=1,10,16. [K1] With η=1\eta=1 and target cos⁡x+cos⁡8x\cos x+\cos8x, after 200200 steps mode 11 is fitted to rounding and 0.03800.0380 of mode 88 remains; the steps needed to reach one per cent are 6.66.6, 282282 and 956956 for k=1,8,16k=1,8,16 (6). [K2] For (c), with five data points the interpolant satisfies the homogeneous equation away from the data to 10−910^{-9} and its derivative jumps by −ci-c_i at xix_i: the representer theorem is the statement that the minimizer is a combination of fundamental solutions. [K5]

Spectral bias in the kernel regime: the remaining fraction of mode kk of the target against the number of gradient steps, for the sampled H1H^1 kernel on 6464 points (17).

A physics-informed loss [39] for −u′′=f-u''=f is the squared residual ∥Au−f∥2\lVert Au-f\rVert^2 with A=−∂x2A=-\partial_x^2; the energy, or Ritz, loss [16] is 12⟨u,Au⟩−⟨f,u⟩\frac12\langle u,Au\rangle-\langle f,u\rangle.

Theorem 18 (Squaring the operator). [classical; cf. [24, 46]] (a) The Hessian of the strong-form loss is ATAA^{\mathsf{T}}A and that of the energy loss is AA: the condition numbers are κ2\kappa^2 and κ\kappa, with κ∼4N2/π2\kappa\sim4N^2/\pi^2 for NN unknowns. (b) Composed with the kernel of 17, the strong-form loss has mode rates μkk4=k4/(1+k2)\mu_kk^4=k^4/(1+k^2), increasing in kk.

On 3131 unknowns κ=414.3\kappa=414.3 and κ2=171,682\kappa^2=171{,}682; gradient descent with the optimal step needs 28622862 and 1,185,9361{,}185{,}936 iterations to gain six digits. [K3] On 6464 points the regression rates at k=1k=1 and 1616 are 0.50080.5008 and 0.00480.0048, a ratio of 1/1041/104, and the strong-form rates are 0.50000.5000 and 206.8206.8, a ratio of 414414. [K4] The bias is reversed and the spread of rates is four times as large: the low modes, which carry the solution, are now the slow ones relative to the step that the high modes allow. This is a linear model of the training difficulties reported for physics-informed networks [24, 46], and it is the reason the weak formulation is the standard one in numerical analysis [36, Ch. 12].

Noise

Gradient descent with small isotropic noise on a loss VV is modelled by the Langevin equation dX=−V′(X) dt+2/β dWdX=-V'(X)\,dt+\sqrt{2/\beta}\,dW, whose density solves a Fokker–Planck equation with stationary solution proportional to e−βVe^{-\beta V} [32].

Proposition 19. [classical; [26, 2]] (a) For V=12λx2V=\frac12\lambda x^2 the spectral gap is λ\lambda for every β\beta. (b) For a symmetric double well with minima ±a\pm a and barrier ΔV\Delta V at 00, the gap is 1πV′′(a)∣V′′(0)∣ e−βΔV(1+O(β−1))\frac1\pi\sqrt{V''(a)|V''(0)|}\,e^{-\beta\Delta V}(1+O(\beta^{-1})). (c) Constant-step stochastic gradient descent on 12λx2\frac12\lambda x^2 with gradient noise of variance σ2\sigma^2 has stationary variance ησ2/λ(2−ηλ)\eta\sigma^2/\lambda(2-\eta\lambda).

A reversible discretization has the Gibbs density as its exact stationary vector, and for λ=1\lambda=1 its gap is 0.99970.9997 at β=1\beta=1 and at β=4\beta=4. [L1] For V=(x2−1)2V=(x^2-1)^2 the gap divided by 32πe−β\frac{\sqrt{32}}\pi e^{-\beta} is 0.9030.903, 0.9280.928, 0.9470.947, 0.9590.959 at β=4,6,8,10\beta=4,6,8,10, and the gap at β=10\beta=10 is 7.8×10−57.8\times10^{-5}. [L2] The recursion for the variance converges to 0.06250.0625 for η=0.1\eta=0.1, λ=2\lambda=2, σ=1.5\sigma=1.5; the first-order value ησ2/2λ\eta\sigma^2/2\lambda is 0.056250.05625, corresponding to the temperature β−1=ησ2/2\beta^{-1}=\eta\sigma^2/2. [L3] Within a basin, mixing is as fast as the curvature and does not depend on the noise; between basins it is exponentially slow in the ratio of barrier height to noise. The exponent in (b) is where a claim that noise selects wide minima has to be made quantitative.

Control

Proposition 20. [classical; [12, 3]] The Bellman operator TV=min⁡a(ca+γPaV)TV=\min_a(c_a+\gamma P_aV) is monotone and a contraction of factor γ\gamma in the supremum norm. A monotone, stable and consistent scheme for a Hamilton–Jacobi equation converges to its viscosity solution.

For a random problem with 66 states and 33 actions, monotonicity and the contraction hold on 300300 random pairs, with largest ratio 0.70720.7072 at γ=0.9\gamma=0.9, and value iteration obeys the geometric bound. [Q1] For ∣u′∣=1|u'|=1 on the unit interval with zero boundary values the scheme ui=min⁡(ui−1,ui+1)+hu_i=\min(u_{i-1},u_{i+1})+h converges to the distance to the boundary exactly, and has residual 2h2h on a sawtooth and on minus the distance, which satisfy the equation almost everywhere. [Q2] Monotonicity is the discrete maximum principle: the Bellman operator is to control what a Markov kernel is to diffusion, and the selection of the viscosity solution is the selection of the entropy solution of a conservation law [36, Thm. 14.3].

How many numbers an operator needs

Operator learning approximates a solution map by a finite-rank one, composed with nonlinear maps [30, 25]. For linear problems the best rank-rr error is the (r+1)(r+1)-st singular value.

Theorem 21 (Rank by type). [classical; ]On the unit interval with Dirichlet conditions:

  1. Elliptic. (−∂x2)−1(-\partial_x^2)^{-1} has singular values (πk)−2(\pi k)^{-2}: relative accuracy ϵ\epsilon needs ϵ−1/2\epsilon^{-1/2} modes.

  2. Parabolic. et∂x2e^{t\partial_x^2} has singular values e−π2k2te^{-\pi^2k^2t}: accuracy ϵ\epsilon needs 1+log⁡(1/ϵ)/π2t\sqrt{1+\log(1/\epsilon)/\pi^2t} modes.

  3. Hyperbolic. The wave propagator is an isometry of the energy: every mode is needed.

  4. Translates of a step function have covariance eigenvalues ((k−12)π)−2((k-\frac12)\pi)^{-2}, so the best rr-dimensional linear approximation of the family has relative error ∼(π2r/2)−1/2\sim(\pi^2r/2)^{-1/2}.

At ϵ=10−6\epsilon=10^{-6} this is 10001000 modes for the elliptic inverse and 1111 for the heat flow at t=0.01t=0.01 (1212 on a grid of 6363 points, whose stencil underestimates the higher eigenvalues as in 6(b)); the discrete wave propagator preserves the energy form to 10−1510^{-15}. [O1] For the steps, the first three eigenvalues are 0.405310.40531, 0.045050.04505, 0.016230.01623 on 6060 points, and the relative error is 0.1420.142 with ten modes and 0.0450.045 with a hundred (7). [O2] A linear-reduction architecture is therefore well matched to parabolic problems, adequate for elliptic ones, and badly matched to transport, where the natural coordinates are the characteristics of 6 and not a fixed basis. This is the Kolmogorov barrier of reduced-order modelling [33], read off the type of the equation.

Singular values of solution operators on the unit interval, relative to the first, by type (21): none decay for the wave equation, they decay algebraically for the elliptic inverse and for translates of a step, and faster than exponentially for the heat flow at t=0.01t=0.01.

Summary of results

Qualitative statements with their quantitative counterparts. Every number is recomputed by the companion script.
Topic Qualitative statement Quantitative statement Checks
Padding reflect == Neumann by images, iff kernels symmetric; zero == Dirichlet only for radius one defects 1.181.18, 0.1000.100, 0.1600.160; mass 0.5320.532 left after 200200 zero-padded layers N1–N6
Resolution multipliers natural; stencils to second order; polynomials iff N>(d+1)KN>(d+1)K; rectifiers never symbol 0.7170.717, 0.9800.980 of k2k^2; rectifier error −π/3N2-\pi/3N^2 A1–A4
Diffusion models reversal is ill-posed for densities, well-posed for samples given the score 5.2×10215.2\times10^{21} against 0.2640.264; score slope −1/t+a2/t2-1/t+a^2/t^2 S3–S5
Score matching, entropy denoising == explicit ++ constant; de Bruijn; contraction constant 1/t−J=0.9241/t-J=0.924; rate 22, limit 12(EX0)2\frac12(\mathbb{E}X_0)^2 S2, S6–S8
Flow matching in one dimension, the monotone map for any coupling; commitment by a pitchfork error 10−1310^{-13}; t∗=0.382t^*=0.382; slope 890890 at t=0.9t=0.9 F1–F3
Message passing oversmoothing is the spectral gap factor 0.924370.92437; depth scales 415415, 55, 11 G1, G2
Wave layers conserve an energy; CFL bound drift 2×10−152\times10^{-15}; growth 1.86331.8633 beyond G3
Sheaf diffusion limit is the projection to H0H^0; obstruction in H1(G;Z/2)H^1(G;\mathbb{Z}/2) 1616 of 128128; gaps 0.16060.1606, 0.198060.19806 G4, G5
Residual layers Euler unstable on antisymmetric layers; leapfrog bounded gradients 1.64461.6446, 1.05131.0513; condition ≤1.667\le1.667 against 8.7×10138.7\times10^{13} R1–R4
Kernel regression low modes first; sampled spectrum is a Poisson sum ratios 1.00161.0016, 1.08521.0852, 1.23461.2346; steps 6.66.6, 282282, 956956 K1, K2, K5
Physics-informed loss squares the condition number and reverses the bias 414414 against 171,682171{,}682; rate ratios 1/1041/104 and 414414 K3, K4
Noise gap == curvature in a basin, Kramers between basins 0.99970.9997; ratios 0.903,…,0.9590.903,\dots,0.959 L1–L3
Control Bellman operator monotone contraction; monotone scheme selects the viscosity solution ratio ≤0.7072\le0.7072; residual 2h2h on the wrong solutions Q1, Q2
Operator rank algebraic, exponential, none, by type 10001000, 1111, all; steps: 0.1420.142 with ten modes O1, O2

[tab:summary] pairs the statements. Read by column of the title, the entries on padding, resolution, flow matching and sheaves are about naturality; those on diffusion models, residual layers, physics-informed losses and control are about well-posedness; and those on message passing, kernels, noise and rank are about rates. The divisions are not sharp, because in each case the three questions are answered by the same object: a group acting on a domain and the spectrum of an operator that commutes with it.

Limits

Linearity. Most results concern the linear part of a layer or the kernel regime of training. Nonlinear layers enter only where they can be treated exactly: pointwise maps commute with even extension (3), and their aliasing is computed (6). Nothing is proved about the training dynamics of a finite network.

Dimension. The examples are one-dimensional or on small graphs. Theorem 3 extends to boxes and to the other chambers of reflection groups [36, Rem. 3.6]; 11 does not extend, and says so.

The score is assumed. Theorem 8 takes the score as given with a uniform error. How well a network estimates it, and in which norm, is the statistical question that the theorem isolates and does not answer.

Models of noise. Proposition 19 models stochastic gradient descent by isotropic Langevin dynamics, which is known to be a rough approximation: real gradient noise is anisotropic and state-dependent.

Checks. The script verifies instances with a fixed seed. A check that passes on a random instance supports a statement and does not prove it; the proofs are those given or cited.

Novelty. We know of no result here that a specialist in the relevant area would find new. The statements tagged as observations (4, 11, 12) are elementary and may well be in the literature in other words.

What is higher. As in [36], the categorical content is modest: naturality squares, a group action with its characters, and cohomology in degree one for sheaves on graphs. The one place where a higher structure is forced, gauge fields forming a groupoid [36, Rem. 13.2], has a counterpart here in 15, where relabelling nodes is a gauge transformation; we have not developed it.

The companion script

verify-pde-learning.py runs in about a second and prints one line per check. The checks are: N1–N6 padding rules, their eigenbases, the image constructions and their failures, mass and the maximum principle; A1–A4 resampling of multipliers, stencils, squares and rectifiers; S1–S8 the heat equation for a Gaussian mixture, Tweedie’s formula, the probability flow, sensitivity to the score, the two-point score, de Bruijn’s identity, contraction of relative entropy, and the score-matching constant; F1–F3 change of variables along characteristics, flow matching between Gaussians, and the two-point field with its bifurcation time; G1–G5 oversmoothing rates, depth scales on three graphs, the wave layer, sheaf diffusion, and the count of balanced sign sheaves; R1–R4 stability of Euler and leapfrog layers and the conditioning of their Jacobians; K1–K5 the sampled kernel spectrum, kernel gradient descent, the two condition numbers, the reversed rates, and the interpolant as a sum of fundamental solutions; L1–L3 Fokker–Planck gaps and the variance of stochastic gradient descent; Q1–Q2 the Bellman operator and the monotone scheme for the eikonal equation; O1–O2 ranks of solution operators and the spectrum of translated steps. Random inputs are drawn with seed 2026111720261117. Tolerances are stated in the script next to each check. The curves in the figures were computed from the formulas of the text by a separate plotting script and are embedded in the source as coordinates; the points of 2 for the random input are the values printed by check A4, and the signal of 5 is a random one drawn for the figure.

References

  1. B. D. O. Anderson, Reverse-time diffusion equation models, Stochastic Process. Appl. 12 (1982), 313–326.

  2. D. Bakry, M. Émery, Diffusions hypercontractives, in Séminaire de Probabilités XIX, Lecture Notes in Math. 1123, Springer, 1985, 177–206.

  3. G. Barles, P. E. Souganidis, Convergence of approximation schemes for fully nonlinear second order equations, Asymptotic Anal. 4 (1991), 271–283.

  4. M. M. Bronstein, J. Bruna, T. Cohen, P. Veličković, Geometric deep learning: grids, groups, graphs, geodesics, and gauges, arXiv:2104.13478, 2021.

  5. G. Biroli, T. Bonnaire, V. de Bortoli, M. Mézard, Dynamical regimes of diffusion models, Nature Commun. 15 (2024).

  6. F. Bartolucci, E. de Bézenac, B. Raonić, R. Molinaro, S. Mishra, R. Alaifari, Representation equivalent neural operators: a framework for alias-free operator learning, Adv. Neural Inf. Process. Syst. 36 (2023); arXiv:2305.19913. Checked.

  7. C. Bodnar, F. Di Giovanni, B. P. Chamberlain, P. Liò, M. M. Bronstein, Neural sheaf diffusion: a topological perspective on heterophily and oversmoothing in GNNs, Adv. Neural Inf. Process. Syst. 35 (2022); arXiv:2202.04579. Checked.

  8. B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, E. Holtham, Reversible architectures for arbitrarily deep residual neural networks, Proc. AAAI 32 (2018).

  9. T. Cohen, M. Welling, Group equivariant convolutional networks, Proc. ICML 2016, 2990–2999.

  10. R. T. Q. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud, Neural ordinary differential equations, Adv. Neural Inf. Process. Syst. 31 (2018).

  11. B. P. Chamberlain, J. Rowbottom, M. Gorinova, S. Webb, E. Rossi, M. M. Bronstein, GRAND: graph neural diffusion, Proc. ICML 2021, 1407–1418.

  12. M. G. Crandall, P.-L. Lions, Viscosity solutions of Hamilton–Jacobi equations, Trans. Amer. Math. Soc. 277 (1983), 1–42.

  13. Dai, Chen, Wang, Jia, Xie, Kumar, Yu, Learning PDE solvers with physics and data: a unifying view of physics-informed neural networks and neural operators, arXiv:2601.14517, 2026. Checked; surnames only were recorded.

  14. W. E, A proposal on machine learning via dynamical systems, Commun. Math. Stat. 5 (2017), 1–11.

  15. B. Efron, Tweedie’s formula and selection bias, J. Amer. Statist. Assoc. 106 (2011), 1602–1614.

  16. W. E, B. Yu, The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems, Commun. Math. Stat. 6 (2018), 1–12.

  17. J. Hansen, R. Ghrist, Toward a spectral theory of cellular sheaves, J. Appl. Comput. Topol. 3 (2019), 315–358.

  18. E. Haber, L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems 34 (2017), 014004.

  19. J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, Adv. Neural Inf. Process. Syst. 33 (2020).

  20. A. Hyvärinen, Estimation of non-normalized statistical models by score matching, J. Mach. Learn. Res. 6 (2005), 695–709.

  21. M. A. Islam, S. Jia, N. D. B. Bruce, How much position information do convolutional neural networks encode?, Proc. ICLR 2020.

  22. A. Jacot, F. Gabriel, C. Hongler, Neural tangent kernel: convergence and generalization in neural networks, Adv. Neural Inf. Process. Syst. 31 (2018).

  23. T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, T. Aila, Alias-free generative adversarial networks, Adv. Neural Inf. Process. Syst. 34 (2021).

  24. A. S. Krishnapriyan, A. Gholami, S. Zhe, R. M. Kirby, M. W. Mahoney, Characterizing possible failure modes in physics-informed neural networks, Adv. Neural Inf. Process. Syst. 34 (2021).

  25. N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, A. Anandkumar, Neural operator: learning maps between function spaces with applications to PDEs, J. Mach. Learn. Res. 24 (2023), no. 89.

  26. H. A. Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, Physica 7 (1940), 284–304.

  27. Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, M. Le, Flow matching for generative modeling, Proc. ICLR 2023.

  28. X. Liu, C. Gong, Q. Liu, Flow straight and fast: learning to generate and transfer data with rectified flow, Proc. ICLR 2023.

  29. Q. Li, Z. Han, X.-M. Wu, Deeper insights into graph convolutional networks for semi-supervised learning, Proc. AAAI 32 (2018).

  30. L. Lu, P. Jin, G. Pang, Z. Zhang, G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Mach. Intell. 3 (2021), 218–229.

  31. Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, A. Anandkumar, Fourier neural operator for parametric partial differential equations, Proc. ICLR 2021.

  32. S. Mandt, M. D. Hoffman, D. M. Blei, Stochastic gradient descent as approximate Bayesian inference, J. Mach. Learn. Res. 18 (2017), no. 134.

  33. M. Ohlberger, S. Rave, Reduced basis methods: success, limitations and future challenges, Proc. ALGORITMY 2016, 1–12.

  34. K. Oono, T. Suzuki, Graph neural networks exponentially lose expressive power for node classification, Proc. ICLR 2020.

  35. S. A. Orszag, On the elimination of aliasing in finite-difference schemes by filtering high-wavenumber components, J. Atmos. Sci. 28 (1971), 1074.

  36. Anonymous, Partial Differential Equations in Higher Categories, Higher Categorical Semantics, 2026. Companion volume.

  37. N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y. Bengio, A. Courville, On the spectral bias of neural networks, Proc. ICML 2019, 5301–5310.

  38. T. K. Rusch, B. P. Chamberlain, J. Rowbottom, S. Mishra, M. M. Bronstein, Graph-coupled oscillator networks, Proc. ICML 2022, 18888–18909.

  39. M. Raissi, P. Perdikaris, G. E. Karniadakis, Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys. 378 (2019), 686–707.

  40. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, B. Poole, Score-based generative modeling through stochastic differential equations, Proc. ICLR 2021.

  41. A. J. Stam, Some inequalities satisfied by the quantities of information of Fisher and Shannon, Inform. and Control 2 (1959), 101–112.

  42. W. A. Strauss, Partial Differential Equations: An Introduction, 2nd ed., Wiley, 2008.

  43. G. Strang, The discrete cosine transform, SIAM Rev. 41 (1999), 135–147.

  44. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, S. Ganguli, Deep unsupervised learning using nonequilibrium thermodynamics, Proc. ICML 2015, 2256–2265.

  45. P. Vincent, A connection between score matching and denoising autoencoders, Neural Comput. 23 (2011), 1661–1674.

  46. S. Wang, Y. Teng, P. Perdikaris, Understanding and mitigating gradient flow pathologies in physics-informed neural networks, SIAM J. Sci. Comput. 43 (2021), A3055–A3081.

Complete flattened LaTeX source
\documentclass[10pt,twocolumn]{article}
\usepackage[T1]{fontenc}

\usepackage[utf8]{inputenc}

\usepackage[sc]{mathpazo}

\usepackage[scaled=0.9]{helvet}

\usepackage[letterpaper,margin=0.78in,columnsep=0.3in]{geometry}

\usepackage{amsmath,amssymb,amsthm,mathtools}

\usepackage{microtype}

\usepackage{xcolor}

\definecolor{accent}{HTML}{7A1F2B}
\definecolor{ink2}{HTML}{55595F}
\definecolor{fwd}{HTML}{1F4E79}
\definecolor{bwd}{HTML}{5B3A7A}
\usepackage{tikz,tikz-cd}

\usetikzlibrary{arrows.meta,positioning}
\usepackage{booktabs,array}

\usepackage{enumitem}

\usepackage[font=small,labelfont={bf,sf}]{caption}

\usepackage[colorlinks=true,linkcolor=accent,citecolor=accent,urlcolor=accent,
  pdftitle={Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates},
  pdfauthor={Anonymous},
  pdfsubject={Partial differential equations as a framework for convolution, generative diffusion, graph networks and learning dynamics},
  pdfkeywords={partial differential equations, machine learning, boundary conditions, generative diffusion, graph networks, numerical stability, spectral rates}]{hyperref}

\usepackage[nameinlink,capitalise,noabbrev]{cleveref}

\theoremstyle{plain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{definition}[theorem]{Definition}
\newtheorem{example}[theorem]{Example}
\theoremstyle{remark}
\newtheorem{remark}[theorem]{Remark}
\newcommand{\stag}[2]{{\footnotesize\sffamily\upshape\color{#1}[#2]}\enspace\ignorespaces}
\newcommand{\Scl}[1][]{\stag{ink2}{classical\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Sli}[1][]{\stag{fwd}{literature\if\relax\detokenize{#1}\relax\else; #1\fi}}
\newcommand{\Srf}[1][]{\stag{accent}{reformulation\if\relax\detokenize{#1}\relax\else; cf.\ #1\fi}}
\newcommand{\Sob}{\stag{accent}{observation}}
\newcommand{\Spg}{\stag{bwd}{programme}}
\newcommand{\chk}[1]{{\footnotesize\sffamily\upshape\color{accent}[#1]}}
\newcommand{\cat}[1]{\mathsf{#1}}
\newcommand{\R}{\mathbb{R}}
\newcommand{\Z}{\mathbb{Z}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\one}{\mathbf{1}}
\newcommand{\T}{\mathsf{T}}
\newcommand{\Lap}{\Delta}
\newcommand{\pd}{\partial}
\newcommand{\KL}[2]{D(#1\,\Vert\,#2)}
\newcommand{\Var}{\operatorname{Var}}
\newcommand{\relu}{\operatorname{relu}}
\newcommand{\sinc}{\operatorname{sinc}}
\newcommand{\term}[1]{\textbf{\itshape #1}}
\setlength{\parskip}{0pt}
\frenchspacing
\raggedbottom
\title{\textbf{Partial Differential Equations in Machine Learning and AI:\\ Naturality, Well-Posedness, and Spectral Rates}}
\author{Anonymous}
\date{Higher Categorical Semantics \quad$\cdot$\quad Version 1.0, 18 November 2026}
\begin{document}
\twocolumn[
\maketitle
\begin{center}\begin{minipage}{0.86\textwidth}\small
\textbf{Abstract.} Convolutional layers are difference operators, depth is a time variable, generative diffusion is
the heat equation, message passing is a graph Laplacian, and a kernel is a Green's function. We organize these
identifications by three questions that the theory of partial differential equations asks of any construction.
\emph{Is it natural}, that is, does it commute with the maps of domains, resolutions and symmetries along which
one wants to move it? \emph{Is it well-posed}, that is, does the output depend continuously on the input?
\emph{At what rate} does it act on each mode? Each qualitative answer is paired with a computed one. Naturality:
half-sample reflection models the Neumann condition. Circular layers preserve the reflected subspace exactly
when the periodized kernels are symmetric.
zero padding is the Dirichlet condition only for three-point stencils; Fourier multipliers commute with
resampling, stencils do so to second order, squaring needs more than $3K$ points for band $K$, and a rectifier
is aliased at order $N^{-2}$ on every grid. Well-posedness: reversing diffusion amplifies mode $10$ by
$5\times10^{21}$, while the reverse flow driven by the score moves samples by $0.26$ per unit of score error, the
difficulty reappearing as a score of slope $1/t^2$. Forward Euler amplifies nonzero oscillatory modes. In the
stable range $0<h\omega<2$, the leapfrog condition number is at most $(1+h\omega/2)/(1-h\omega/2)$. A strong-form loss squares the
condition number of the energy loss, $414$ against $171{,}682$ on $31$ unknowns. Rates: oversmoothing proceeds
at $1-\tau\lambda_2$ per layer, with depth scales $415$, $5$ and $1$ on a path, a hypercube and a complete graph
of $32$ nodes; sheaf diffusion converges to global sections, which exist for $16$ of the $128$ sign sheaves on a
small graph; kernel regression learns mode $k$ at rate $1/(1+k^2)$ plus aliases; and the spectral gap of
Langevin dynamics in a double well approaches the Kramers rate at low noise. All numerical statements are recomputed by a companion
script with $42$ checks. No new theorem is claimed; the contribution is the organization, the exact constants,
and the three questions asked uniformly.
\end{minipage}\end{center}
\vspace{1.2em}
]

\section{Introduction}

The companion volume \cite{PDEHC} retells an introductory course on partial differential equations
\cite{Str08} around one question: which constructions are natural for maps of domains, of equations and of
function spaces. This paper asks the same question of machine learning, where the same equations have been
rediscovered piecemeal. A convolutional layer with a three-point kernel is a difference operator, and its
padding rule is a boundary condition \cite{Str99}. A residual network is a difference scheme in a time variable
called depth \cite{E17,HaRu17,CRBD18}. A diffusion model noises data with the heat equation and samples by
reversing it \cite{SWMG15,HJA20,SSKKEP21}. Message passing on a graph is a discrete diffusion
\cite{CRGBB21}, or a discrete wave equation \cite{RCRMB22}, or a diffusion of sections of a sheaf
\cite{HaGh19,BDCLB22}. Kernel regression inverts an elliptic operator, physics-informed networks minimize the
residual of an equation \cite{RPK19}, and operator learning approximates its solution map
\cite{LKALBSA21,KLLABSA23,LJPZK21}. A recent survey organizes the last two families \cite{DCWJXKY26}.

Three questions organize the theory of the equations themselves, and we ask each of them of each construction.

\emph{Naturality.} A domain can be reflected, translated, refined or glued, and a construction is natural if it
commutes with the corresponding maps. Equivariance \cite{CoWe16,BBCV21} is naturality for symmetries;
discretization invariance and the absence of aliasing \cite{KALHHLA21,BBRMMA23} are naturality for changes of
resolution.

\emph{Well-posedness.} Hadamard's criterion asks that a solution exist, be unique, and depend continuously on
the data. Learning problems fail the third condition in characteristic ways: by running a parabolic equation
backwards, by discretizing a conservative system with a dissipative scheme or the reverse, or by squaring an
operator in the loss.

\emph{Rates.} For linear constructions the answer to ``how fast'' is a spectrum. Oversmoothing, spectral bias,
mixing of stochastic gradient descent and the rank of a learned operator are all statements about eigenvalues
of a Laplacian.

\paragraph{What is and is not claimed.} Every statement below carries a status tag: classical, literature,
reformulation, observation or programme. Almost all are classical or known; where the form or the constant
seems not to be standard we say ``observation''. The checks, tagged like \chk{N3}, are performed by the companion
script \texttt{verify-pde-learning.py}, which uses only the Python standard library and a fixed seed, and are
listed in \cref{app:checks}. They recompute instances; they are evidence and not proofs. Most models are linear
and one-dimensional, so that every number is exact or accurate to many digits; \cref{sec:limits} says what this
leaves out. The references were assembled from memory, except for \cite{DCWJXKY26,BBRMMA23,BDCLB22}, which were
checked online on 17 November 2026.

\section{Setting}\label{sec:setting}

Let $\cat{Dom}$ be a category of domains: intervals, circles, grids, graphs, with the maps of interest between
them (inclusions, coverings, symmetries, refinements). A \term{signal space} is a functor
$F\colon\cat{Dom}^{\mathrm{op}}\to\cat{Vect}$, such as functions on the domain with restriction or pullback. A
\term{layer} is a family of maps $T_D\colon F(D)\to F(D)$, not necessarily linear.

\begin{definition}\label{def:natural}
The layer $T$ is \term{natural} for a map $f\colon D\to D'$ if the square
\[\begin{tikzcd}[column sep=large]
F(D')\arrow[r,"T_{D'}"]\arrow[d,"f^*"'] & F(D')\arrow[d,"f^*"]\\
F(D)\arrow[r,"T_D"'] & F(D)
\end{tikzcd}\]
commutes. Its \term{defect} on an input $x$ is $\lVert f^*T_{D'}x-T_Df^*x\rVert$, relative to the norm of either
term.
\end{definition}

\noindent For $f$ a symmetry of $D$ this is equivariance. For $f$ the inclusion of a coarse grid in a fine one,
with $F$ restricted to band-limited signals, it is discretization invariance. For $f$ a covering map and $F$
replaced by its invariant part, it is the method of images, with which we begin. A \term{depth semigroup} is a
family $T^t$ with $T^{t+s}=T^tT^s$, and a network of $N$ layers of step $h$ is a discretization of $T^{Nh}$;
naturality in $t$ is consistency of a scheme, and well-posedness is its stability \cite[Ch.~12]{PDEHC}.

\section{Padding is a boundary condition}\label{sec:padding}

Let $x\in\R^n$ and let $w=(w_{-r},\dots,w_r)$ be a kernel. The layer is $(C_wx)_i=\sum_jw_jx_{i+j}$, and a
\term{padding} rule supplies the values $x_i$ for $i$ outside $0,\dots,n-1$: \emph{zero} ($x_i=0$),
\emph{circular} ($x_{i+n}=x_i$) or \emph{reflect} ($x_{-1-i}=x_i$ and $x_{n+i}=x_{n-1-i}$).

\begin{proposition}\label{prop:laplacians}
\Scl[\cite{Str99}] With the kernel $(-1,2,-1)$ the three rules give the Dirichlet, periodic and Neumann
difference Laplacians. They are diagonalized by the sine transform $\sin\frac{\pi k(j+1)}{n+1}$, the discrete
Fourier transform, and the cosine transform $\cos\frac{\pi k(j+1/2)}n$, with eigenvalues
$4\sin^2\frac{\pi k}{2(n+1)}$, $4\sin^2\frac{\pi k}n$ and $4\sin^2\frac{\pi k}{2n}$. \chk{N1, N2}
\end{proposition}

\begin{figure}[t]
\centering
\begin{tikzpicture}
\fill[fwd!8] (-0.2,-0.72) rectangle (2.2,0.78);
\draw[ink2] (-3.04,0)--(5.04,0);
\draw[accent,densely dashed] (-0.2,-0.72)--(-0.2,0.78);
\draw[accent,densely dashed] (2.2,-0.72)--(2.2,0.78);
\draw[thick,bwd] (-2.8,0)--(-2.8,0.66);
\fill[bwd] (-2.8,0.66) circle (1.3pt);
\draw[thick,bwd] (-2.4,0)--(-2.4,0.66);
\fill[bwd] (-2.4,0.66) circle (1.3pt);
\draw[thick,bwd] (-2,0)--(-2,0.363);
\fill[bwd] (-2,0.363) circle (1.3pt);
\draw[thick,bwd] (-1.6,0)--(-1.6,0.198);
\fill[bwd] (-1.6,0.198) circle (1.3pt);
\draw[thick,bwd] (-1.2,0)--(-1.2,0.462);
\fill[bwd] (-1.2,0.462) circle (1.3pt);
\draw[thick,bwd] (-0.8,0)--(-0.8,0.594);
\fill[bwd] (-0.8,0.594) circle (1.3pt);
\draw[thick,bwd] (-0.4,0)--(-0.4,0.33);
\fill[bwd] (-0.4,0.33) circle (1.3pt);
\draw[thick,fwd] (0,0)--(0,0.33);
\fill[fwd] (0,0.33) circle (1.3pt);
\draw[thick,fwd] (0.4,0)--(0.4,0.594);
\fill[fwd] (0.4,0.594) circle (1.3pt);
\draw[thick,fwd] (0.8,0)--(0.8,0.462);
\fill[fwd] (0.8,0.462) circle (1.3pt);
\draw[thick,fwd] (1.2,0)--(1.2,0.198);
\fill[fwd] (1.2,0.198) circle (1.3pt);
\draw[thick,fwd] (1.6,0)--(1.6,0.363);
\fill[fwd] (1.6,0.363) circle (1.3pt);
\draw[thick,fwd] (2,0)--(2,0.66);
\fill[fwd] (2,0.66) circle (1.3pt);
\draw[thick,bwd] (2.4,0)--(2.4,0.66);
\fill[bwd] (2.4,0.66) circle (1.3pt);
\draw[thick,bwd] (2.8,0)--(2.8,0.363);
\fill[bwd] (2.8,0.363) circle (1.3pt);
\draw[thick,bwd] (3.2,0)--(3.2,0.198);
\fill[bwd] (3.2,0.198) circle (1.3pt);
\draw[thick,bwd] (3.6,0)--(3.6,0.462);
\fill[bwd] (3.6,0.462) circle (1.3pt);
\draw[thick,bwd] (4,0)--(4,0.594);
\fill[bwd] (4,0.594) circle (1.3pt);
\draw[thick,bwd] (4.4,0)--(4.4,0.33);
\fill[bwd] (4.4,0.33) circle (1.3pt);
\draw[thick,bwd] (4.8,0)--(4.8,0.33);
\fill[bwd] (4.8,0.33) circle (1.3pt);
\node[anchor=west] at (-3.04,1.02) {\scriptsize reflect: even extension};
\fill[fwd!8] (-0.2,-2.72) rectangle (2.2,-1.22);
\draw[ink2] (-3.04,-2)--(5.04,-2);
\draw[accent,densely dashed] (-0.4,-2.72)--(-0.4,-1.22);
\draw[accent,densely dashed] (2.4,-2.72)--(2.4,-1.22);
\draw[thick,bwd] (-2.8,-2)--(-2.8,-2.66);
\fill[bwd] (-2.8,-2.66) circle (1.3pt);
\draw[thick,bwd] (-2.4,-2)--(-2.4,-2.363);
\fill[bwd] (-2.4,-2.363) circle (1.3pt);
\draw[thick,bwd] (-2,-2)--(-2,-2.198);
\fill[bwd] (-2,-2.198) circle (1.3pt);
\draw[thick,bwd] (-1.6,-2)--(-1.6,-2.462);
\fill[bwd] (-1.6,-2.462) circle (1.3pt);
\draw[thick,bwd] (-1.2,-2)--(-1.2,-2.594);
\fill[bwd] (-1.2,-2.594) circle (1.3pt);
\draw[thick,bwd] (-0.8,-2)--(-0.8,-2.33);
\fill[bwd] (-0.8,-2.33) circle (1.3pt);
\draw[thick,bwd] (-0.4,-2)--(-0.4,-2);
\fill[bwd] (-0.4,-2) circle (1.3pt);
\draw[thick,fwd] (0,-2)--(0,-1.67);
\fill[fwd] (0,-1.67) circle (1.3pt);
\draw[thick,fwd] (0.4,-2)--(0.4,-1.406);
\fill[fwd] (0.4,-1.406) circle (1.3pt);
\draw[thick,fwd] (0.8,-2)--(0.8,-1.538);
\fill[fwd] (0.8,-1.538) circle (1.3pt);
\draw[thick,fwd] (1.2,-2)--(1.2,-1.802);
\fill[fwd] (1.2,-1.802) circle (1.3pt);
\draw[thick,fwd] (1.6,-2)--(1.6,-1.637);
\fill[fwd] (1.6,-1.637) circle (1.3pt);
\draw[thick,fwd] (2,-2)--(2,-1.34);
\fill[fwd] (2,-1.34) circle (1.3pt);
\draw[thick,bwd] (2.4,-2)--(2.4,-2);
\fill[bwd] (2.4,-2) circle (1.3pt);
\draw[thick,bwd] (2.8,-2)--(2.8,-2.66);
\fill[bwd] (2.8,-2.66) circle (1.3pt);
\draw[thick,bwd] (3.2,-2)--(3.2,-2.363);
\fill[bwd] (3.2,-2.363) circle (1.3pt);
\draw[thick,bwd] (3.6,-2)--(3.6,-2.198);
\fill[bwd] (3.6,-2.198) circle (1.3pt);
\draw[thick,bwd] (4,-2)--(4,-2.462);
\fill[bwd] (4,-2.462) circle (1.3pt);
\draw[thick,bwd] (4.4,-2)--(4.4,-2.594);
\fill[bwd] (4.4,-2.594) circle (1.3pt);
\draw[thick,bwd] (4.8,-2)--(4.8,-2.33);
\fill[bwd] (4.8,-2.33) circle (1.3pt);
\node[anchor=west] at (-3.04,-0.98) {\scriptsize odd extension (Dirichlet images)};
\fill[fwd!8] (-0.2,-4.72) rectangle (2.2,-3.22);
\draw[ink2] (-3.04,-4)--(5.04,-4);
\draw[thick,bwd] (-2.8,-4)--(-2.8,-4);
\fill[bwd] (-2.8,-4) circle (1.3pt);
\draw[thick,bwd] (-2.4,-4)--(-2.4,-4);
\fill[bwd] (-2.4,-4) circle (1.3pt);
\draw[thick,bwd] (-2,-4)--(-2,-4);
\fill[bwd] (-2,-4) circle (1.3pt);
\draw[thick,bwd] (-1.6,-4)--(-1.6,-4);
\fill[bwd] (-1.6,-4) circle (1.3pt);
\draw[thick,bwd] (-1.2,-4)--(-1.2,-4);
\fill[bwd] (-1.2,-4) circle (1.3pt);
\draw[thick,bwd] (-0.8,-4)--(-0.8,-4);
\fill[bwd] (-0.8,-4) circle (1.3pt);
\draw[thick,bwd] (-0.4,-4)--(-0.4,-4);
\fill[bwd] (-0.4,-4) circle (1.3pt);
\draw[thick,fwd] (0,-4)--(0,-3.67);
\fill[fwd] (0,-3.67) circle (1.3pt);
\draw[thick,fwd] (0.4,-4)--(0.4,-3.406);
\fill[fwd] (0.4,-3.406) circle (1.3pt);
\draw[thick,fwd] (0.8,-4)--(0.8,-3.538);
\fill[fwd] (0.8,-3.538) circle (1.3pt);
\draw[thick,fwd] (1.2,-4)--(1.2,-3.802);
\fill[fwd] (1.2,-3.802) circle (1.3pt);
\draw[thick,fwd] (1.6,-4)--(1.6,-3.637);
\fill[fwd] (1.6,-3.637) circle (1.3pt);
\draw[thick,fwd] (2,-4)--(2,-3.34);
\fill[fwd] (2,-3.34) circle (1.3pt);
\draw[thick,bwd] (2.4,-4)--(2.4,-4);
\fill[bwd] (2.4,-4) circle (1.3pt);
\draw[thick,bwd] (2.8,-4)--(2.8,-4);
\fill[bwd] (2.8,-4) circle (1.3pt);
\draw[thick,bwd] (3.2,-4)--(3.2,-4);
\fill[bwd] (3.2,-4) circle (1.3pt);
\draw[thick,bwd] (3.6,-4)--(3.6,-4);
\fill[bwd] (3.6,-4) circle (1.3pt);
\draw[thick,bwd] (4,-4)--(4,-4);
\fill[bwd] (4,-4) circle (1.3pt);
\draw[thick,bwd] (4.4,-4)--(4.4,-4);
\fill[bwd] (4.4,-4) circle (1.3pt);
\draw[thick,bwd] (4.8,-4)--(4.8,-4);
\fill[bwd] (4.8,-4) circle (1.3pt);
\node[anchor=west] at (-3.04,-2.98) {\scriptsize zero};
\fill[fwd!8] (-0.2,-6.72) rectangle (2.2,-5.22);
\draw[ink2] (-3.04,-6)--(5.04,-6);
\draw[accent,densely dashed] (-0.2,-6.72)--(-0.2,-5.22);
\draw[accent,densely dashed] (2.2,-6.72)--(2.2,-5.22);
\draw[thick,bwd] (-2.8,-6)--(-2.8,-5.34);
\fill[bwd] (-2.8,-5.34) circle (1.3pt);
\draw[thick,bwd] (-2.4,-6)--(-2.4,-5.67);
\fill[bwd] (-2.4,-5.67) circle (1.3pt);
\draw[thick,bwd] (-2,-6)--(-2,-5.406);
\fill[bwd] (-2,-5.406) circle (1.3pt);
\draw[thick,bwd] (-1.6,-6)--(-1.6,-5.538);
\fill[bwd] (-1.6,-5.538) circle (1.3pt);
\draw[thick,bwd] (-1.2,-6)--(-1.2,-5.802);
\fill[bwd] (-1.2,-5.802) circle (1.3pt);
\draw[thick,bwd] (-0.8,-6)--(-0.8,-5.637);
\fill[bwd] (-0.8,-5.637) circle (1.3pt);
\draw[thick,bwd] (-0.4,-6)--(-0.4,-5.34);
\fill[bwd] (-0.4,-5.34) circle (1.3pt);
\draw[thick,fwd] (0,-6)--(0,-5.67);
\fill[fwd] (0,-5.67) circle (1.3pt);
\draw[thick,fwd] (0.4,-6)--(0.4,-5.406);
\fill[fwd] (0.4,-5.406) circle (1.3pt);
\draw[thick,fwd] (0.8,-6)--(0.8,-5.538);
\fill[fwd] (0.8,-5.538) circle (1.3pt);
\draw[thick,fwd] (1.2,-6)--(1.2,-5.802);
\fill[fwd] (1.2,-5.802) circle (1.3pt);
\draw[thick,fwd] (1.6,-6)--(1.6,-5.637);
\fill[fwd] (1.6,-5.637) circle (1.3pt);
\draw[thick,fwd] (2,-6)--(2,-5.34);
\fill[fwd] (2,-5.34) circle (1.3pt);
\draw[thick,bwd] (2.4,-6)--(2.4,-5.67);
\fill[bwd] (2.4,-5.67) circle (1.3pt);
\draw[thick,bwd] (2.8,-6)--(2.8,-5.406);
\fill[bwd] (2.8,-5.406) circle (1.3pt);
\draw[thick,bwd] (3.2,-6)--(3.2,-5.538);
\fill[bwd] (3.2,-5.538) circle (1.3pt);
\draw[thick,bwd] (3.6,-6)--(3.6,-5.802);
\fill[bwd] (3.6,-5.802) circle (1.3pt);
\draw[thick,bwd] (4,-6)--(4,-5.637);
\fill[bwd] (4,-5.637) circle (1.3pt);
\draw[thick,bwd] (4.4,-6)--(4.4,-5.34);
\fill[bwd] (4.4,-5.34) circle (1.3pt);
\draw[thick,bwd] (4.8,-6)--(4.8,-5.67);
\fill[bwd] (4.8,-5.67) circle (1.3pt);
\node[anchor=west] at (-3.04,-4.98) {\scriptsize circular};
\end{tikzpicture}
\caption{Four ways of continuing a signal of six samples (shaded) beyond its ends. Reflect padding is the even extension about the half-sample points (dashed), which is the Neumann condition. The odd extension about the neighbouring grid points is the Dirichlet condition; zero padding agrees with it only on the first sample outside. Circular padding is the periodic extension.}\label{fig:pad}
\end{figure}

The method of images \cite[Thm.~3.5]{PDEHC} explains this construction. In \cref{fig:pad}, the padding reflects
about half-sample points, repeating the end samples. Let $E\colon\R^n\to\R^{2n}$ be the even extension
$Ex=(x_0,\dots,x_{n-1},x_{n-1},\dots,x_0)$, regarded as a signal on the cycle $\Z/2n$, and let $R$ be the
restriction to the first $n$ entries. The image of $E$ is the space of signals invariant under the reflection
$\rho\colon i\mapsto2n-1-i$.

\begin{theorem}[Reflect padding is the Neumann condition]\label{thm:reflect}
\Srf[\cite{Str99}; \cite{PDEHC}, Thm.~3.5] Write $C^{\mathrm{refl}}_w$ and $C^{\mathrm{circ}}_w$ for the layer with
reflect padding on $\R^n$ and circular padding on $\R^{2n}$.
\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]
\item $C^{\mathrm{refl}}_w=R\,C^{\mathrm{circ}}_wE$ for every kernel.
\item $\rho\,C^{\mathrm{circ}}_w\rho=C^{\mathrm{circ}}_{\check w}$ with $\check w_j=w_{-j}$. Hence if $w$ is symmetric,
$C^{\mathrm{circ}}_wE=E\,C^{\mathrm{refl}}_w$, and a network of symmetric kernels, reflect padding and pointwise
nonlinearities is the restriction to invariant signals of a circular network on the doubled cycle.
\item The circular layer preserves the reflected subspace exactly when the kernel, reduced modulo $2n$ by
summing coefficients with the same residue, is symmetric. An asymmetric periodized kernel sends some
even input outside that subspace.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) The reflect rule is the definition of $E$ read at indices outside $0,\dots,n-1$. (b) Substituting
$i\mapsto2n-1-i$ in the sum reverses the kernel. Pointwise maps commute with $E$ and $R$.
For (c), work with the periodized kernel and set $a=w-\check w$. If the circular layer preserves the
reflected subspace, then $C^{\mathrm{circ}}_a$ annihilates the even input $\delta_0+\delta_{-1}$,
by (b). Its output at $i$ is $a_i+a_{i+1}=0$. Since $a_0=0$, all $a_i$ vanish.
\end{proof}

\noindent For a random input and the kernels $(0.2,-0.5,1,-0.5,0.2)$ and $(0.2,-0.5,1,0.7,-0.3)$: identity (a)
holds to rounding for both; the output on the double is invariant to rounding for the first and has an
anti-invariant part of relative size $1.18$ for the second; and two layers on the double differ from two
reflect-padded layers by $0$ when the symmetric kernel comes first and by $0.100$ when the other does.
\chk{N3} The circular model with symmetric kernels is equivariant under the dihedral group of the doubled
cycle. The half-sample construction restricts it to signals fixed by the chosen reflection. The Dirichlet
construction uses the odd extension about the neighbouring grid points.

\begin{proposition}[Zero padding is Dirichlet only for short kernels]\label{prop:zero}
\Sob Let $O\colon\R^n\to\R^{2n+2}$ be the odd extension $(0,x,0,-x^{\mathrm{rev}})$. For a symmetric kernel of radius
one, zero padding equals circular convolution on $Ox$ restricted to the original positions. For radius two or
more it does not: the image construction supplies $-x_0$ beyond the boundary where zero padding supplies $0$.
Pointwise maps commute with $O$ only if they are odd.
\end{proposition}

\noindent The relative difference is $0$ for $(0.3,1,0.3)$ and $0.160$ for the five-point symmetric kernel above.
\chk{N4} The Dirichlet image construction can differ from zero padding for wider kernels. A rectifier can
also destroy the odd symmetry required by the construction.

\begin{proposition}[What padding breaks and keeps]\label{prop:mass}
\Scl \emph{(a)} Circular convolution commutes with translation. With zero padding the commutator with a shift is
supported within the kernel radius of the boundary. \emph{(b)} A layer with a non-negative symmetric kernel of
sum one and reflect padding is doubly stochastic: it conserves the mean and obeys the maximum principle. With
zero padding it is substochastic, and the mass remaining after $m$ layers is the probability that a random
walk with steps distributed by the kernel has not left the interval.
\end{proposition}

\noindent For a five-point kernel on $20$ positions the commutator is supported on positions $0$, $18$ and $19$.
\chk{N5} Position information therefore enters a zero-padded network at the boundary and spreads inwards by one
kernel radius per layer, which is the mechanism behind the observation that such networks learn absolute
position \cite{ILB20}. For the kernel $(\frac14,\frac12,\frac14)$ on $32$ positions, $200$ layers with reflect
padding change the total mass by $2\times10^{-15}$, and with zero padding leave $0.5320$ of it, in agreement
with the sum over the sine basis. \chk{N6}

\section{Resolution: what commutes with resampling}\label{sec:resolution}

Let $B_K$ be the trigonometric polynomials of degree at most $K$ on the circle and
$S_N\colon B_K\to\R^N$ the sampling map on $N$ equally spaced points, injective for $N>2K$. An operator $T$ on
functions with discretizations $T_N$ is natural for refinement (\cref{def:natural}) when $S_NT=T_NS_N$ on $B_K$;
this is the requirement of representation equivalence in \cite{BBRMMA23}.

\begin{theorem}[Three kinds of layer]\label{thm:alias}
\Srf[\cite{BBRMMA23,KALHHLA21}] On $B_K$ and for $N>2K$:
\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]
\item A Fourier multiplier $m(D)$, discretized by the discrete Fourier transform, is natural exactly.
\item The three-point stencil for $-\pd_x^2$ has symbol $k^2\sinc^2(kh/2)$, $h=2\pi/N$: it is natural to second
order.
\item A polynomial nonlinearity of degree $d$ followed by projection to $B_K$ is natural if $N>(d+1)K$, and
for generic inputs only then.
\item A rectifier followed by projection to $B_K$ is natural for no $N$; the defect is of order $N^{-2}$.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) Sampling intertwines the characters $e^{ikx}$, $|k|\le K$, with the discrete ones. (b) Apply the stencil to
$e^{ikx}$. (c) The power has band $dK$, and a frequency $k'$ is sampled onto $k\ne k'$ when $N$ divides $k'-k$;
the largest $|k'-k|$ with $|k|\le K$ is $(d+1)K$. (d) A function with a corner has Fourier coefficients of order
$k^{-2}$, and the sum of those at $k+mN$, $m\neq0$, is of order $N^{-2}$.
\end{proof}

\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{scope}[shift={(0,0)}]
\draw[ink2] (0,3.5)--(0,0)--(6.3,0);
\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $8$};
\draw[ink2] (1.26,0)--(1.26,-0.07) node[below,inner sep=1.5pt]{\scriptsize $16$};
\draw[ink2] (2.52,0)--(2.52,-0.07) node[below,inner sep=1.5pt]{\scriptsize $32$};
\draw[ink2] (3.78,0)--(3.78,-0.07) node[below,inner sep=1.5pt]{\scriptsize $64$};
\draw[ink2] (5.04,0)--(5.04,-0.07) node[below,inner sep=1.5pt]{\scriptsize $128$};
\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $256$};
\draw[ink2] (0,0)--(-0.07,0) node[left,inner sep=1.5pt]{\scriptsize $10^{-5}$};
\draw[ink2] (0,0.814)--(-0.07,0.814) node[left,inner sep=1.5pt]{\scriptsize $10^{-4}$};
\draw[ink2] (0,1.628)--(-0.07,1.628) node[left,inner sep=1.5pt]{\scriptsize $10^{-3}$};
\draw[ink2] (0,2.441)--(-0.07,2.441) node[left,inner sep=1.5pt]{\scriptsize $10^{-2}$};
\draw[ink2] (0,3.255)--(-0.07,3.255) node[left,inner sep=1.5pt]{\scriptsize $10^{-1}$};
\node[below] at (3.15,-0.38) {\scriptsize grid points $N$};
\draw[ink2,densely dashed] plot coordinates {(0,3.319) (6.3,0.87)};
\draw[thick,fwd] plot coordinates {(0,2.619) (1.26,2.126) (2.52,1.636) (3.78,1.146) (5.04,0.656) (6.3,0.166)};
\draw[thick,bwd] plot coordinates {(1.26,3.149) (2.52,2.434) (3.78,1.89) (5.04,1.321) (6.3,0.933)};
\fill[fwd] (0,2.619) circle (1.6pt);
\fill[fwd] (1.26,2.126) circle (1.6pt);
\fill[fwd] (2.52,1.636) circle (1.6pt);
\fill[fwd] (3.78,1.146) circle (1.6pt);
\fill[fwd] (5.04,0.656) circle (1.6pt);
\fill[fwd] (6.3,0.166) circle (1.6pt);
\fill[bwd] (1.26,3.149) circle (1.6pt);
\fill[bwd] (2.52,2.434) circle (1.6pt);
\fill[bwd] (3.78,1.89) circle (1.6pt);
\fill[bwd] (5.04,1.321) circle (1.6pt);
\fill[bwd] (6.3,0.933) circle (1.6pt);
\node[fwd] at (1.666,1.092) {\scriptsize mean of $\relu(\sin x)$};
\node[bwd] at (4.923,2.261) {\scriptsize random input, band $4$};
\node[ink2] at (2.926,2.751) {\scriptsize slope $-2$};
\end{scope}
\end{tikzpicture}
\caption{Aliasing error of a rectifier against the number of grid points, on logarithmic scales: the closed-form case of \cref{thm:alias}(d) and the largest in-band coefficient error for a random input of band $4$. Both follow the slope $-2$.}\label{fig:alias}
\end{figure}

\noindent For a random input of band $5$, the multiplier $(1+k^2)^{-1}$ computed on $16$ and on $64$ points agrees
with the exact result to $10^{-14}$; the stencil is in relative error by $0.221$ and $0.0153$, its symbol at $k=5$
being $0.7173$ and $0.9801$ of the true one. \chk{A1} For band $4$ the in-band coefficients of the square are
exact on $13$ points and wrong by $1.4$ on $12$: the ``two-thirds rule'' of spectral methods \cite{Ors71}.
\chk{A2} For the rectifier one case is closed-form: the mean of $\relu(\sin x)$ sampled on $N$ points is
$\frac1N\cot\frac\pi N$ instead of $\frac1\pi$, an error of $-\pi/3N^2$ to leading order; $N^2$ times the error is
$-1.058$, $-1.050$, $-1.048$, $-1.047$ for $N=8,\dots,64$. \chk{A3} For a random input of band $4$ the largest
in-band error of the rectified signal is $7.4\times10^{-2}$, $9.8\times10^{-3}$, $2.1\times10^{-3}$,
$4.2\times10^{-4}$, $1.4\times10^{-4}$ on $16,\dots,256$ points (\cref{fig:alias}). \chk{A4} This is the quantitative content of the
practice of applying nonlinearities on an oversampled grid and filtering \cite{KALHHLA21}: oversampling by a
factor $M$ buys $M^{2}$, and no finite factor buys exactness. An architecture built from (a) alone is
resolution-independent \cite{LKALBSA21}; one with stencils is so to the order of the stencil; one with
rectifiers is so only approximately, at every resolution.

\section{Diffusion models}\label{sec:diffusion}

Let data $X_0\sim p_0$ on the line be noised by $X_t=X_0+W_t$, so that the density solves the heat equation
$\pd_tp=\frac12\pd_x^2p$, and write $s=\pd_x\log p_t$ for the \term{score}. Our example is a mixture of two
Gaussians, for which everything is closed-form. \chk{S1}

\begin{proposition}\label{prop:tweedie}
\Scl[\cite{Efr11,Hyv05,Vin11}] \emph{(a)} $\E[X_0\mid X_t=x]=x+t\,s(x,t)$. \emph{(b)} For every candidate $\sigma$,
\begin{multline*}
\E\bigl|\sigma(X_t)+\tfrac{X_t-X_0}t\bigr|^2\\=\E|\sigma(X_t)-s(X_t,t)|^2+\tfrac1t-J(p_t),
\end{multline*}
where $J$ is the Fisher information: denoising and explicit score matching differ by a constant.
\end{proposition}

\noindent Both hold to rounding in the example; for two-point data at $t=\frac12$ the constant is $0.924073$ for
each of two test functions. \chk{S2, S8} Since $\frac12\pd_x^2p=-\pd_x\bigl(p\cdot(-\frac12s)\bigr)$, the heat equation
is also a continuity equation, and its characteristics are the \term{probability flow}
$\dot x=-\frac12s(x,t)$ \cite{SSKKEP21}.

\begin{theorem}[Where the ill-posedness goes]\label{thm:illposed}
\Srf[\cite{PDEHC}, Ex.~2.5; \cite{And82,SSKKEP21}]
\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]
\item The backward heat equation multiplies the $k$-th Fourier mode of $p_T$ by $e^{k^2T/2}$: recovering $p_0$
from $p_T$ is ill-posed in every Sobolev norm.
\item Given the score on $[t_0,T]$ with $t_0>0$, the probability flow is an ordinary differential equation with
a Lipschitz field; it transports $p_T$ to $p_{t_0}$, and an error $\epsilon$ in the score, in the supremum norm,
moves each sample by at most $C(t_0)\epsilon$.
\item For data on the two points $\pm a$, $\pd_xs(0,t)=-1/t+a^2/t^2$. The Lipschitz constant of the field is not
integrable at $t=0$, and $C(t_0)\to\infty$ as $t_0\to0$.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) is Fourier analysis. (b) is the uniqueness theorem for characteristics together with Gr\"onwall's
inequality. (c) Here $s=-x/t+(a/t)\tanh(ax/t)$.
\end{proof}

\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{scope}[shift={(0,0)}]
\draw[ink2] (0,3.4)--(0,0)--(6,0);
\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $-2$};
\draw[ink2] (1.5,0)--(1.5,-0.07) node[below,inner sep=1.5pt]{\scriptsize $-1$};
\draw[ink2] (3,0)--(3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $0$};
\draw[ink2] (4.5,0)--(4.5,-0.07) node[below,inner sep=1.5pt]{\scriptsize $1$};
\draw[ink2] (6,0)--(6,-0.07) node[below,inner sep=1.5pt]{\scriptsize $2$};
\draw[ink2] (0,0.283)--(-0.07,0.283) node[left,inner sep=1.5pt]{\scriptsize $-10$};
\draw[ink2] (0,1.7)--(-0.07,1.7) node[left,inner sep=1.5pt]{\scriptsize $0$};
\draw[ink2] (0,3.117)--(-0.07,3.117) node[left,inner sep=1.5pt]{\scriptsize $10$};
\node[below] at (3,-0.38) {\scriptsize $x$};
\draw[ink2,very thin] (0,1.7)--(6,1.7);
\draw[thick,ink2] plot coordinates {(0,1.847) (0.015,1.845) (0.03,1.844) (0.045,1.843) (0.06,1.842) (0.075,1.84) (0.09,1.839) (0.105,1.838) (0.12,1.836) (0.135,1.835) (0.15,1.834) (0.165,1.832) (0.18,1.831) (0.195,1.83) (0.21,1.829) (0.225,1.827) (0.24,1.826) (0.255,1.825) (0.27,1.823) (0.285,1.822) (0.3,1.821) (0.315,1.82) (0.33,1.818) (0.345,1.817) (0.36,1.816) (0.375,1.815) (0.39,1.813) (0.405,1.812) (0.42,1.811) (0.435,1.81) (0.45,1.808) (0.465,1.807) (0.48,1.806) (0.495,1.805) (0.51,1.803) (0.525,1.802) (0.54,1.801) (0.555,1.8) (0.57,1.799) (0.585,1.797) (0.6,1.796) (0.615,1.795) (0.63,1.794) (0.645,1.793) (0.66,1.791) (0.675,1.79) (0.69,1.789) (0.705,1.788) (0.72,1.787) (0.735,1.785) (0.75,1.784) (0.765,1.783) (0.78,1.782) (0.795,1.781) (0.81,1.78) (0.825,1.779) (0.84,1.777) (0.855,1.776) (0.87,1.775) (0.885,1.774) (0.9,1.773) (0.915,1.772) (0.93,1.771) (0.945,1.77) (0.96,1.769) (0.975,1.767) (0.99,1.766) (1.005,1.765) (1.02,1.764) (1.035,1.763) (1.05,1.762) (1.065,1.761) (1.08,1.76) (1.095,1.759) (1.11,1.758) (1.125,1.757) (1.14,1.756) (1.155,1.755) (1.17,1.754) (1.185,1.753) (1.2,1.752) (1.215,1.751) (1.23,1.75) (1.245,1.749) (1.26,1.748) (1.275,1.747) (1.29,1.746) (1.305,1.745) (1.32,1.744) (1.335,1.743) (1.35,1.742) (1.365,1.742) (1.38,1.741) (1.395,1.74) (1.41,1.739) (1.425,1.738) (1.44,1.737) (1.455,1.736) (1.47,1.735) (1.485,1.735) (1.5,1.734) (1.515,1.733) (1.53,1.732) (1.545,1.731) (1.56,1.731) (1.575,1.73) (1.59,1.729) (1.605,1.728) (1.62,1.727) (1.635,1.727) (1.65,1.726) (1.665,1.725) (1.68,1.725) (1.695,1.724) (1.71,1.723) (1.725,1.723) (1.74,1.722) (1.755,1.721) (1.77,1.721) (1.785,1.72) (1.8,1.719) (1.815,1.719) (1.83,1.718) (1.845,1.717) (1.86,1.717) (1.875,1.716) (1.89,1.716) (1.905,1.715) (1.92,1.715) (1.935,1.714) (1.95,1.714) (1.965,1.713) (1.98,1.713) (1.995,1.712) (2.01,1.712) (2.025,1.711) (2.04,1.711) (2.055,1.71) (2.07,1.71) (2.085,1.709) (2.1,1.709) (2.115,1.709) (2.13,1.708) (2.145,1.708) (2.16,1.707) (2.175,1.707) (2.19,1.707) (2.205,1.706) (2.22,1.706) (2.235,1.706) (2.25,1.705) (2.265,1.705) (2.28,1.705) (2.295,1.705) (2.31,1.704) (2.325,1.704) (2.34,1.704) (2.355,1.703) (2.37,1.703) (2.385,1.703) (2.4,1.703) (2.415,1.703) (2.43,1.702) (2.445,1.702) (2.46,1.702) (2.475,1.702) (2.49,1.702) (2.505,1.702) (2.52,1.701) (2.535,1.701) (2.55,1.701) (2.565,1.701) (2.58,1.701) (2.595,1.701) (2.61,1.701) (2.625,1.701) (2.64,1.701) (2.655,1.701) (2.67,1.7) (2.685,1.7) (2.7,1.7) (2.715,1.7) (2.73,1.7) (2.745,1.7) (2.76,1.7) (2.775,1.7) (2.79,1.7) (2.805,1.7) (2.82,1.7) (2.835,1.7) (2.85,1.7) (2.865,1.7) (2.88,1.7) (2.895,1.7) (2.91,1.7) (2.925,1.7) (2.94,1.7) (2.955,1.7) (2.97,1.7) (2.985,1.7) (3,1.7) (3.015,1.7) (3.03,1.7) (3.045,1.7) (3.06,1.7) (3.075,1.7) (3.09,1.7) (3.105,1.7) (3.12,1.7) (3.135,1.7) (3.15,1.7) (3.165,1.7) (3.18,1.7) (3.195,1.7) (3.21,1.7) (3.225,1.7) (3.24,1.7) (3.255,1.7) (3.27,1.7) (3.285,1.7) (3.3,1.7) (3.315,1.7) (3.33,1.7) (3.345,1.699) (3.36,1.699) (3.375,1.699) (3.39,1.699) (3.405,1.699) (3.42,1.699) (3.435,1.699) (3.45,1.699) (3.465,1.699) (3.48,1.699) (3.495,1.698) (3.51,1.698) (3.525,1.698) (3.54,1.698) (3.555,1.698) (3.57,1.698) (3.585,1.697) (3.6,1.697) (3.615,1.697) (3.63,1.697) (3.645,1.697) (3.66,1.696) (3.675,1.696) (3.69,1.696) (3.705,1.695) (3.72,1.695) (3.735,1.695) (3.75,1.695) (3.765,1.694) (3.78,1.694) (3.795,1.694) (3.81,1.693) (3.825,1.693) (3.84,1.693) (3.855,1.692) (3.87,1.692) (3.885,1.691) (3.9,1.691) (3.915,1.691) (3.93,1.69) (3.945,1.69) (3.96,1.689) (3.975,1.689) (3.99,1.688) (4.005,1.688) (4.02,1.687) (4.035,1.687) (4.05,1.686) (4.065,1.686) (4.08,1.685) (4.095,1.685) (4.11,1.684) (4.125,1.684) (4.14,1.683) (4.155,1.683) (4.17,1.682) (4.185,1.681) (4.2,1.681) (4.215,1.68) (4.23,1.679) (4.245,1.679) (4.26,1.678) (4.275,1.677) (4.29,1.677) (4.305,1.676) (4.32,1.675) (4.335,1.675) (4.35,1.674) (4.365,1.673) (4.38,1.673) (4.395,1.672) (4.41,1.671) (4.425,1.67) (4.44,1.669) (4.455,1.669) (4.47,1.668) (4.485,1.667) (4.5,1.666) (4.515,1.665) (4.53,1.665) (4.545,1.664) (4.56,1.663) (4.575,1.662) (4.59,1.661) (4.605,1.66) (4.62,1.659) (4.635,1.658) (4.65,1.658) (4.665,1.657) (4.68,1.656) (4.695,1.655) (4.71,1.654) (4.725,1.653) (4.74,1.652) (4.755,1.651) (4.77,1.65) (4.785,1.649) (4.8,1.648) (4.815,1.647) (4.83,1.646) (4.845,1.645) (4.86,1.644) (4.875,1.643) (4.89,1.642) (4.905,1.641) (4.92,1.64) (4.935,1.639) (4.95,1.638) (4.965,1.637) (4.98,1.636) (4.995,1.635) (5.01,1.634) (5.025,1.633) (5.04,1.631) (5.055,1.63) (5.07,1.629) (5.085,1.628) (5.1,1.627) (5.115,1.626) (5.13,1.625) (5.145,1.624) (5.16,1.623) (5.175,1.621) (5.19,1.62) (5.205,1.619) (5.22,1.618) (5.235,1.617) (5.25,1.616) (5.265,1.615) (5.28,1.613) (5.295,1.612) (5.31,1.611) (5.325,1.61) (5.34,1.609) (5.355,1.607) (5.37,1.606) (5.385,1.605) (5.4,1.604) (5.415,1.603) (5.43,1.601) (5.445,1.6) (5.46,1.599) (5.475,1.598) (5.49,1.597) (5.505,1.595) (5.52,1.594) (5.535,1.593) (5.55,1.592) (5.565,1.59) (5.58,1.589) (5.595,1.588) (5.61,1.587) (5.625,1.585) (5.64,1.584) (5.655,1.583) (5.67,1.582) (5.685,1.58) (5.7,1.579) (5.715,1.578) (5.73,1.577) (5.745,1.575) (5.76,1.574) (5.775,1.573) (5.79,1.571) (5.805,1.57) (5.82,1.569) (5.835,1.568) (5.85,1.566) (5.865,1.565) (5.88,1.564) (5.895,1.562) (5.91,1.561) (5.925,1.56) (5.94,1.558) (5.955,1.557) (5.97,1.556) (5.985,1.555) (6,1.553)};
\draw[thick,fwd] plot coordinates {(0,2.172) (0.015,2.168) (0.03,2.163) (0.045,2.158) (0.06,2.153) (0.075,2.149) (0.09,2.144) (0.105,2.139) (0.12,2.134) (0.135,2.13) (0.15,2.125) (0.165,2.12) (0.18,2.116) (0.195,2.111) (0.21,2.106) (0.225,2.101) (0.24,2.097) (0.255,2.092) (0.27,2.087) (0.285,2.083) (0.3,2.078) (0.315,2.073) (0.33,2.068) (0.345,2.064) (0.36,2.059) (0.375,2.054) (0.39,2.049) (0.405,2.045) (0.42,2.04) (0.435,2.035) (0.45,2.031) (0.465,2.026) (0.48,2.021) (0.495,2.016) (0.51,2.012) (0.525,2.007) (0.54,2.002) (0.555,1.998) (0.57,1.993) (0.585,1.988) (0.6,1.983) (0.615,1.979) (0.63,1.974) (0.645,1.969) (0.66,1.964) (0.675,1.96) (0.69,1.955) (0.705,1.95) (0.72,1.946) (0.735,1.941) (0.75,1.936) (0.765,1.931) (0.78,1.927) (0.795,1.922) (0.81,1.917) (0.825,1.913) (0.84,1.908) (0.855,1.903) (0.87,1.898) (0.885,1.894) (0.9,1.889) (0.915,1.884) (0.93,1.88) (0.945,1.875) (0.96,1.87) (0.975,1.865) (0.99,1.861) (1.005,1.856) (1.02,1.851) (1.035,1.847) (1.05,1.842) (1.065,1.837) (1.08,1.832) (1.095,1.828) (1.11,1.823) (1.125,1.818) (1.14,1.814) (1.155,1.809) (1.17,1.804) (1.185,1.799) (1.2,1.795) (1.215,1.79) (1.23,1.785) (1.245,1.781) (1.26,1.776) (1.275,1.771) (1.29,1.767) (1.305,1.762) (1.32,1.757) (1.335,1.753) (1.35,1.748) (1.365,1.743) (1.38,1.738) (1.395,1.734) (1.41,1.729) (1.425,1.724) (1.44,1.72) (1.455,1.715) (1.47,1.71) (1.485,1.706) (1.5,1.701) (1.515,1.697) (1.53,1.692) (1.545,1.687) (1.56,1.683) (1.575,1.678) (1.59,1.673) (1.605,1.669) (1.62,1.664) (1.635,1.66) (1.65,1.655) (1.665,1.651) (1.68,1.646) (1.695,1.641) (1.71,1.637) (1.725,1.632) (1.74,1.628) (1.755,1.623) (1.77,1.619) (1.785,1.615) (1.8,1.61) (1.815,1.606) (1.83,1.601) (1.845,1.597) (1.86,1.593) (1.875,1.588) (1.89,1.584) (1.905,1.58) (1.92,1.575) (1.935,1.571) (1.95,1.567) (1.965,1.563) (1.98,1.559) (1.995,1.555) (2.01,1.551) (2.025,1.547) (2.04,1.543) (2.055,1.539) (2.07,1.535) (2.085,1.532) (2.1,1.528) (2.115,1.525) (2.13,1.521) (2.145,1.518) (2.16,1.514) (2.175,1.511) (2.19,1.508) (2.205,1.505) (2.22,1.502) (2.235,1.499) (2.25,1.496) (2.265,1.494) (2.28,1.491) (2.295,1.489) (2.31,1.487) (2.325,1.485) (2.34,1.483) (2.355,1.482) (2.37,1.48) (2.385,1.479) (2.4,1.478) (2.415,1.477) (2.43,1.477) (2.445,1.476) (2.46,1.476) (2.475,1.477) (2.49,1.477) (2.505,1.478) (2.52,1.479) (2.535,1.48) (2.55,1.482) (2.565,1.484) (2.58,1.486) (2.595,1.489) (2.61,1.492) (2.625,1.496) (2.64,1.5) (2.655,1.504) (2.67,1.509) (2.685,1.514) (2.7,1.519) (2.715,1.525) (2.73,1.531) (2.745,1.538) (2.76,1.545) (2.775,1.553) (2.79,1.56) (2.805,1.569) (2.82,1.577) (2.835,1.586) (2.85,1.595) (2.865,1.605) (2.88,1.615) (2.895,1.625) (2.91,1.635) (2.925,1.646) (2.94,1.656) (2.955,1.667) (2.97,1.678) (2.985,1.689) (3,1.7) (3.015,1.711) (3.03,1.722) (3.045,1.733) (3.06,1.744) (3.075,1.754) (3.09,1.765) (3.105,1.775) (3.12,1.785) (3.135,1.795) (3.15,1.805) (3.165,1.814) (3.18,1.823) (3.195,1.831) (3.21,1.84) (3.225,1.847) (3.24,1.855) (3.255,1.862) (3.27,1.869) (3.285,1.875) (3.3,1.881) (3.315,1.886) (3.33,1.891) (3.345,1.896) (3.36,1.9) (3.375,1.904) (3.39,1.908) (3.405,1.911) (3.42,1.914) (3.435,1.916) (3.45,1.918) (3.465,1.92) (3.48,1.921) (3.495,1.922) (3.51,1.923) (3.525,1.923) (3.54,1.924) (3.555,1.924) (3.57,1.923) (3.585,1.923) (3.6,1.922) (3.615,1.921) (3.63,1.92) (3.645,1.918) (3.66,1.917) (3.675,1.915) (3.69,1.913) (3.705,1.911) (3.72,1.909) (3.735,1.906) (3.75,1.904) (3.765,1.901) (3.78,1.898) (3.795,1.895) (3.81,1.892) (3.825,1.889) (3.84,1.886) (3.855,1.882) (3.87,1.879) (3.885,1.875) (3.9,1.872) (3.915,1.868) (3.93,1.865) (3.945,1.861) (3.96,1.857) (3.975,1.853) (3.99,1.849) (4.005,1.845) (4.02,1.841) (4.035,1.837) (4.05,1.833) (4.065,1.829) (4.08,1.825) (4.095,1.82) (4.11,1.816) (4.125,1.812) (4.14,1.807) (4.155,1.803) (4.17,1.799) (4.185,1.794) (4.2,1.79) (4.215,1.785) (4.23,1.781) (4.245,1.777) (4.26,1.772) (4.275,1.768) (4.29,1.763) (4.305,1.759) (4.32,1.754) (4.335,1.749) (4.35,1.745) (4.365,1.74) (4.38,1.736) (4.395,1.731) (4.41,1.727) (4.425,1.722) (4.44,1.717) (4.455,1.713) (4.47,1.708) (4.485,1.703) (4.5,1.699) (4.515,1.694) (4.53,1.69) (4.545,1.685) (4.56,1.68) (4.575,1.676) (4.59,1.671) (4.605,1.666) (4.62,1.662) (4.635,1.657) (4.65,1.652) (4.665,1.647) (4.68,1.643) (4.695,1.638) (4.71,1.633) (4.725,1.629) (4.74,1.624) (4.755,1.619) (4.77,1.615) (4.785,1.61) (4.8,1.605) (4.815,1.601) (4.83,1.596) (4.845,1.591) (4.86,1.586) (4.875,1.582) (4.89,1.577) (4.905,1.572) (4.92,1.568) (4.935,1.563) (4.95,1.558) (4.965,1.553) (4.98,1.549) (4.995,1.544) (5.01,1.539) (5.025,1.535) (5.04,1.53) (5.055,1.525) (5.07,1.52) (5.085,1.516) (5.1,1.511) (5.115,1.506) (5.13,1.502) (5.145,1.497) (5.16,1.492) (5.175,1.487) (5.19,1.483) (5.205,1.478) (5.22,1.473) (5.235,1.469) (5.25,1.464) (5.265,1.459) (5.28,1.454) (5.295,1.45) (5.31,1.445) (5.325,1.44) (5.34,1.436) (5.355,1.431) (5.37,1.426) (5.385,1.421) (5.4,1.417) (5.415,1.412) (5.43,1.407) (5.445,1.402) (5.46,1.398) (5.475,1.393) (5.49,1.388) (5.505,1.384) (5.52,1.379) (5.535,1.374) (5.55,1.369) (5.565,1.365) (5.58,1.36) (5.595,1.355) (5.61,1.351) (5.625,1.346) (5.64,1.341) (5.655,1.336) (5.67,1.332) (5.685,1.327) (5.7,1.322) (5.715,1.317) (5.73,1.313) (5.745,1.308) (5.76,1.303) (5.775,1.299) (5.79,1.294) (5.805,1.289) (5.82,1.284) (5.835,1.28) (5.85,1.275) (5.865,1.27) (5.88,1.266) (5.895,1.261) (5.91,1.256) (5.925,1.251) (5.94,1.247) (5.955,1.242) (5.97,1.237) (5.985,1.232) (6,1.228)};
\draw[thick,bwd] plot coordinates {(0,3.117) (0.015,3.102) (0.03,3.088) (0.045,3.074) (0.06,3.06) (0.075,3.046) (0.09,3.032) (0.105,3.017) (0.12,3.003) (0.135,2.989) (0.15,2.975) (0.165,2.961) (0.18,2.947) (0.195,2.933) (0.21,2.918) (0.225,2.904) (0.24,2.89) (0.255,2.876) (0.27,2.862) (0.285,2.848) (0.3,2.833) (0.315,2.819) (0.33,2.805) (0.345,2.791) (0.36,2.777) (0.375,2.763) (0.39,2.748) (0.405,2.734) (0.42,2.72) (0.435,2.706) (0.45,2.692) (0.465,2.678) (0.48,2.663) (0.495,2.649) (0.51,2.635) (0.525,2.621) (0.54,2.607) (0.555,2.593) (0.57,2.578) (0.585,2.564) (0.6,2.55) (0.615,2.536) (0.63,2.522) (0.645,2.508) (0.66,2.493) (0.675,2.479) (0.69,2.465) (0.705,2.451) (0.72,2.437) (0.735,2.423) (0.75,2.408) (0.765,2.394) (0.78,2.38) (0.795,2.366) (0.81,2.352) (0.825,2.338) (0.84,2.323) (0.855,2.309) (0.87,2.295) (0.885,2.281) (0.9,2.267) (0.915,2.253) (0.93,2.238) (0.945,2.224) (0.96,2.21) (0.975,2.196) (0.99,2.182) (1.005,2.168) (1.02,2.153) (1.035,2.139) (1.05,2.125) (1.065,2.111) (1.08,2.097) (1.095,2.083) (1.11,2.068) (1.125,2.054) (1.14,2.04) (1.155,2.026) (1.17,2.012) (1.185,1.998) (1.2,1.983) (1.215,1.969) (1.23,1.955) (1.245,1.941) (1.26,1.927) (1.275,1.913) (1.29,1.898) (1.305,1.884) (1.32,1.87) (1.335,1.856) (1.35,1.842) (1.365,1.828) (1.38,1.813) (1.395,1.799) (1.41,1.785) (1.425,1.771) (1.44,1.757) (1.455,1.743) (1.47,1.728) (1.485,1.714) (1.5,1.7) (1.515,1.686) (1.53,1.672) (1.545,1.658) (1.56,1.643) (1.575,1.629) (1.59,1.615) (1.605,1.601) (1.62,1.587) (1.635,1.573) (1.65,1.558) (1.665,1.544) (1.68,1.53) (1.695,1.516) (1.71,1.502) (1.725,1.488) (1.74,1.473) (1.755,1.459) (1.77,1.445) (1.785,1.431) (1.8,1.417) (1.815,1.403) (1.83,1.388) (1.845,1.374) (1.86,1.36) (1.875,1.346) (1.89,1.332) (1.905,1.318) (1.92,1.303) (1.935,1.289) (1.95,1.275) (1.965,1.261) (1.98,1.247) (1.995,1.233) (2.01,1.218) (2.025,1.204) (2.04,1.19) (2.055,1.176) (2.07,1.162) (2.085,1.148) (2.1,1.133) (2.115,1.119) (2.13,1.105) (2.145,1.091) (2.16,1.077) (2.175,1.063) (2.19,1.048) (2.205,1.034) (2.22,1.02) (2.235,1.006) (2.25,0.992) (2.265,0.978) (2.28,0.964) (2.295,0.949) (2.31,0.935) (2.325,0.921) (2.34,0.907) (2.355,0.893) (2.37,0.879) (2.385,0.865) (2.4,0.851) (2.415,0.837) (2.43,0.823) (2.445,0.809) (2.46,0.795) (2.475,0.782) (2.49,0.768) (2.505,0.755) (2.52,0.741) (2.535,0.728) (2.55,0.715) (2.565,0.703) (2.58,0.69) (2.595,0.679) (2.61,0.667) (2.625,0.656) (2.64,0.646) (2.655,0.637) (2.67,0.629) (2.685,0.623) (2.7,0.618) (2.715,0.614) (2.73,0.614) (2.745,0.616) (2.76,0.621) (2.775,0.63) (2.79,0.644) (2.805,0.663) (2.82,0.689) (2.835,0.722) (2.85,0.763) (2.865,0.813) (2.88,0.873) (2.895,0.943) (2.91,1.024) (2.925,1.116) (2.94,1.218) (2.955,1.33) (2.97,1.449) (2.985,1.573) (3,1.7) (3.015,1.827) (3.03,1.951) (3.045,2.07) (3.06,2.182) (3.075,2.284) (3.09,2.376) (3.105,2.457) (3.12,2.527) (3.135,2.587) (3.15,2.637) (3.165,2.678) (3.18,2.711) (3.195,2.737) (3.21,2.756) (3.225,2.77) (3.24,2.779) (3.255,2.784) (3.27,2.786) (3.285,2.786) (3.3,2.782) (3.315,2.777) (3.33,2.771) (3.345,2.763) (3.36,2.754) (3.375,2.744) (3.39,2.733) (3.405,2.721) (3.42,2.71) (3.435,2.697) (3.45,2.685) (3.465,2.672) (3.48,2.659) (3.495,2.645) (3.51,2.632) (3.525,2.618) (3.54,2.605) (3.555,2.591) (3.57,2.577) (3.585,2.563) (3.6,2.549) (3.615,2.535) (3.63,2.521) (3.645,2.507) (3.66,2.493) (3.675,2.479) (3.69,2.465) (3.705,2.451) (3.72,2.436) (3.735,2.422) (3.75,2.408) (3.765,2.394) (3.78,2.38) (3.795,2.366) (3.81,2.352) (3.825,2.337) (3.84,2.323) (3.855,2.309) (3.87,2.295) (3.885,2.281) (3.9,2.267) (3.915,2.252) (3.93,2.238) (3.945,2.224) (3.96,2.21) (3.975,2.196) (3.99,2.182) (4.005,2.167) (4.02,2.153) (4.035,2.139) (4.05,2.125) (4.065,2.111) (4.08,2.097) (4.095,2.082) (4.11,2.068) (4.125,2.054) (4.14,2.04) (4.155,2.026) (4.17,2.012) (4.185,1.997) (4.2,1.983) (4.215,1.969) (4.23,1.955) (4.245,1.941) (4.26,1.927) (4.275,1.912) (4.29,1.898) (4.305,1.884) (4.32,1.87) (4.335,1.856) (4.35,1.842) (4.365,1.827) (4.38,1.813) (4.395,1.799) (4.41,1.785) (4.425,1.771) (4.44,1.757) (4.455,1.742) (4.47,1.728) (4.485,1.714) (4.5,1.7) (4.515,1.686) (4.53,1.672) (4.545,1.657) (4.56,1.643) (4.575,1.629) (4.59,1.615) (4.605,1.601) (4.62,1.587) (4.635,1.572) (4.65,1.558) (4.665,1.544) (4.68,1.53) (4.695,1.516) (4.71,1.502) (4.725,1.487) (4.74,1.473) (4.755,1.459) (4.77,1.445) (4.785,1.431) (4.8,1.417) (4.815,1.402) (4.83,1.388) (4.845,1.374) (4.86,1.36) (4.875,1.346) (4.89,1.332) (4.905,1.317) (4.92,1.303) (4.935,1.289) (4.95,1.275) (4.965,1.261) (4.98,1.247) (4.995,1.232) (5.01,1.218) (5.025,1.204) (5.04,1.19) (5.055,1.176) (5.07,1.162) (5.085,1.147) (5.1,1.133) (5.115,1.119) (5.13,1.105) (5.145,1.091) (5.16,1.077) (5.175,1.062) (5.19,1.048) (5.205,1.034) (5.22,1.02) (5.235,1.006) (5.25,0.992) (5.265,0.977) (5.28,0.963) (5.295,0.949) (5.31,0.935) (5.325,0.921) (5.34,0.907) (5.355,0.892) (5.37,0.878) (5.385,0.864) (5.4,0.85) (5.415,0.836) (5.43,0.822) (5.445,0.807) (5.46,0.793) (5.475,0.779) (5.49,0.765) (5.505,0.751) (5.52,0.737) (5.535,0.722) (5.55,0.708) (5.565,0.694) (5.58,0.68) (5.595,0.666) (5.61,0.652) (5.625,0.637) (5.64,0.623) (5.655,0.609) (5.67,0.595) (5.685,0.581) (5.7,0.567) (5.715,0.552) (5.73,0.538) (5.745,0.524) (5.76,0.51) (5.775,0.496) (5.79,0.482) (5.805,0.467) (5.82,0.453) (5.835,0.439) (5.85,0.425) (5.865,0.411) (5.88,0.397) (5.895,0.382) (5.91,0.368) (5.925,0.354) (5.94,0.34) (5.955,0.326) (5.97,0.312) (5.985,0.297) (6,0.283)};
\draw[accent,very thin,densely dashed] (1.5,0)--(1.5,3.4);
\draw[accent,very thin,densely dashed] (4.5,0)--(4.5,3.4);
\node[bwd,right] at (6.075,0.283) {\scriptsize $t=0.1$};
\node[fwd,right] at (6.075,1.133) {\scriptsize $0.3$};
\node[ink2,right] at (6.075,1.615) {\scriptsize $1$};
\end{scope}
\end{tikzpicture}
\caption{The score $s(x,t)=-x/t+\tanh(x/t)/t$ of data on the two points $\pm1$ (dashed), at three noise levels. As $t\to0$ it becomes a restoring force towards each point, joined by a transition of slope $1/t^2$ at the midpoint.}\label{fig:score}
\end{figure}

\noindent For $T=1$ and $k=10$ the factor in (a) is $5.2\times10^{21}$. In the example the flow from $T=1$ to $0$
preserves the value of the distribution function along five trajectories to $10^{-14}$, and a perturbation
$\epsilon\sin3x$ of the score displaces the samples by $0.264\,\epsilon$. \chk{S3, S4} The slope in (c) is $90$ at
$t=0.1$ and $9900$ at $t=0.01$ (\cref{fig:score}). \chk{S5} The three statements together say that a diffusion model does not solve
the backward heat equation. It replaces an ill-posed linear problem for the density by a well-posed nonlinear
transport problem for samples, whose coefficient, the score, carries the whole difficulty, concentrated at small
times and near the boundaries between modes. Statement (c) is the sample-level form of Hadamard's example.

\begin{proposition}[Entropy]\label{prop:entropy}
\Scl[\cite{Sta59,BaEm85}] \emph{(a)} Under the heat flow $\frac d{dt}h(p_t)=\frac12J(p_t)$. \emph{(b)} Under the
Ornstein--Uhlenbeck flow $dX=-X\,dt+\sqrt2\,dW$, with Gaussian limit $\gamma$,
$\KL{p_t}{\gamma}\le e^{-2t}\KL{p_0}{\gamma}$, and $e^{2t}\KL{p_t}{\gamma}\to\frac12(\E X_0)^2$.
\end{proposition}

\noindent In the example $dh/dt=0.535607$ and $J/2=0.535606$ at $t=0.3$. \chk{S6} The ratio
$\KL{p_t}{\gamma}/e^{-2t}\KL{p_0}{\gamma}$ is $0.353$, $0.204$, $0.119$, $0.088$ at $t=\frac14,\frac12,1,2$, and
$e^{2t}\KL{p_t}{\gamma}=0.03128$ at $t=4$ against the limit $0.03125$. \chk{S7} The rate $2$ is twice the spectral
gap of the generator, and it fixes the noising time needed to make the terminal law Gaussian to a given
accuracy: $\frac12\log(\KL{p_0}{\gamma}/\delta)$.

\section{Flows and characteristics}\label{sec:flows}

A continuous normalizing flow \cite{CRBD18} moves samples by $\dot x=v(x,t)$; the density then solves the
continuity equation $\pd_tp+\pd_x(pv)=0$, of which the trajectories are the characteristics
\cite[Prop.~1.8]{PDEHC}.

\begin{proposition}\label{prop:cnf}
\Scl[\cite{CRBD18}] Along a trajectory $\frac d{dt}\log p(x(t),t)=-\pd_xv(x(t),t)$.
\end{proposition}

\noindent For $v=0.8\tanh(1.3x+0.2)$ the exponential of the integrated divergence equals the derivative of the
flow map to $10^{-8}$. \chk{F1} \term{Flow matching} \cite{LCBNL23,LGL23} chooses a coupling of $X_0\sim p_0$ and
$X_1\sim p_1$, sets $X_t=(1-t)X_0+tX_1$, and regresses onto $v(x,t)=\E[X_1-X_0\mid X_t=x]$.

\begin{theorem}[One dimension]\label{thm:fm}
\Sob On the line, if $v$ is regular enough for trajectories to be unique, the flow of the flow-matching field
is the increasing rearrangement of $p_0$ onto $p_1$, whatever the coupling.
\end{theorem}
\begin{proof}
The marginal field transports $p_0$ to $p_1$ by construction. Trajectories of a scalar equation cannot cross,
so the flow map is increasing, and an increasing map carrying one atomless law to another is unique.
\end{proof}

\noindent For $p_0=N(0,1)$, $p_1=N(2,\frac14)$ and the independent coupling the field is linear in $x$ with
coefficients rational in $t$, and its flow is $x\mapsto2+\frac12x$ to $10^{-13}$. \chk{F2} The conditional paths
are straight lines that cross; the marginal characteristics do not. In higher dimensions non-crossing no longer
determines the map, and the flow depends on the coupling.

\begin{proposition}[A pitchfork in time]\label{prop:pitchfork}
\Sob For $p_0=N(0,1)$ and $p_1$ uniform on $\pm1$, with the independent coupling,
\[ v(x,t)=\frac1{1-t}\Bigl[\tanh\frac{tx}{(1-t)^2}-x\Bigr]. \]
The midpoint $x=0$ is an attracting equilibrium for $t<t^*=\frac{3-\sqrt5}2=0.382$ and a repelling one
afterwards, and $\pd_xv(0,t)\sim(1-t)^{-3}$ as $t\to1$.
\end{proposition}

\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{scope}[shift={(0,0)}]
\draw[ink2] (0,3.8)--(0,0)--(6.3,0);
\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $0$};
\draw[ink2] (3.15,0)--(3.15,-0.07) node[below,inner sep=1.5pt]{\scriptsize $0.5$};
\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $1$};
\draw[ink2] (0,0.173)--(-0.07,0.173) node[left,inner sep=1.5pt]{\scriptsize $-2$};
\draw[ink2] (0,1.036)--(-0.07,1.036) node[left,inner sep=1.5pt]{\scriptsize $-1$};
\draw[ink2] (0,1.9)--(-0.07,1.9) node[left,inner sep=1.5pt]{\scriptsize $0$};
\draw[ink2] (0,2.764)--(-0.07,2.764) node[left,inner sep=1.5pt]{\scriptsize $1$};
\draw[ink2] (0,3.627)--(-0.07,3.627) node[left,inner sep=1.5pt]{\scriptsize $2$};
\node[below] at (3.15,-0.38) {\scriptsize $t$};
\node[above,inner sep=1pt] at (0,3.85) {\scriptsize $x$};
\draw[accent,densely dashed] (2.406,0)--(2.406,3.8);
\draw[fwd] plot coordinates {(0,0.173) (0.076,0.193) (0.151,0.214) (0.227,0.234) (0.302,0.254) (0.378,0.273) (0.454,0.292) (0.529,0.311) (0.605,0.33) (0.68,0.348) (0.756,0.366) (0.832,0.384) (0.907,0.401) (0.983,0.418) (1.058,0.434) (1.134,0.45) (1.21,0.466) (1.285,0.481) (1.361,0.496) (1.436,0.511) (1.512,0.525) (1.588,0.538) (1.663,0.551) (1.739,0.564) (1.814,0.576) (1.89,0.588) (1.966,0.6) (2.041,0.611) (2.117,0.621) (2.192,0.632) (2.268,0.642) (2.344,0.651) (2.419,0.66) (2.495,0.669) (2.57,0.678) (2.646,0.686) (2.722,0.694) (2.797,0.702) (2.873,0.71) (2.948,0.718) (3.024,0.725) (3.1,0.733) (3.175,0.74) (3.251,0.748) (3.326,0.755) (3.402,0.762) (3.478,0.769) (3.553,0.776) (3.629,0.784) (3.704,0.791) (3.78,0.798) (3.856,0.805) (3.931,0.812) (4.007,0.819) (4.082,0.827) (4.158,0.834) (4.234,0.841) (4.309,0.848) (4.385,0.855) (4.46,0.862) (4.536,0.869) (4.612,0.877) (4.687,0.884) (4.763,0.891) (4.838,0.898) (4.914,0.905) (4.99,0.912) (5.065,0.92) (5.141,0.927) (5.216,0.934) (5.292,0.941) (5.368,0.948) (5.443,0.955) (5.519,0.962) (5.594,0.97) (5.67,0.977) (5.746,0.984) (5.821,0.991) (5.897,0.998) (5.972,1.005) (6.048,1.013) (6.124,1.02)};
\draw[fwd] plot coordinates {(0,0.389) (0.076,0.407) (0.151,0.424) (0.227,0.442) (0.302,0.459) (0.378,0.476) (0.454,0.493) (0.529,0.51) (0.605,0.526) (0.68,0.542) (0.756,0.558) (0.832,0.573) (0.907,0.588) (0.983,0.603) (1.058,0.617) (1.134,0.631) (1.21,0.645) (1.285,0.658) (1.361,0.671) (1.436,0.684) (1.512,0.696) (1.588,0.707) (1.663,0.719) (1.739,0.729) (1.814,0.74) (1.89,0.75) (1.966,0.759) (2.041,0.768) (2.117,0.777) (2.192,0.785) (2.268,0.793) (2.344,0.801) (2.419,0.808) (2.495,0.814) (2.57,0.821) (2.646,0.827) (2.722,0.833) (2.797,0.838) (2.873,0.843) (2.948,0.848) (3.024,0.853) (3.1,0.858) (3.175,0.862) (3.251,0.867) (3.326,0.871) (3.402,0.876) (3.478,0.88) (3.553,0.884) (3.629,0.888) (3.704,0.892) (3.78,0.897) (3.856,0.901) (3.931,0.905) (4.007,0.909) (4.082,0.913) (4.158,0.918) (4.234,0.922) (4.309,0.926) (4.385,0.93) (4.46,0.934) (4.536,0.939) (4.612,0.943) (4.687,0.947) (4.763,0.951) (4.838,0.955) (4.914,0.96) (4.99,0.964) (5.065,0.968) (5.141,0.972) (5.216,0.976) (5.292,0.981) (5.368,0.985) (5.443,0.989) (5.519,0.993) (5.594,0.997) (5.67,1.001) (5.746,1.006) (5.821,1.01) (5.897,1.014) (5.972,1.018) (6.048,1.022) (6.124,1.027)};
\draw[fwd] plot coordinates {(0,0.605) (0.076,0.62) (0.151,0.635) (0.227,0.65) (0.302,0.665) (0.378,0.68) (0.454,0.694) (0.529,0.708) (0.605,0.722) (0.68,0.736) (0.756,0.749) (0.832,0.763) (0.907,0.775) (0.983,0.788) (1.058,0.8) (1.134,0.812) (1.21,0.824) (1.285,0.835) (1.361,0.846) (1.436,0.857) (1.512,0.867) (1.588,0.877) (1.663,0.886) (1.739,0.895) (1.814,0.904) (1.89,0.912) (1.966,0.92) (2.041,0.927) (2.117,0.934) (2.192,0.941) (2.268,0.947) (2.344,0.952) (2.419,0.957) (2.495,0.962) (2.57,0.966) (2.646,0.97) (2.722,0.973) (2.797,0.977) (2.873,0.979) (2.948,0.982) (3.024,0.984) (3.1,0.986) (3.175,0.988) (3.251,0.989) (3.326,0.991) (3.402,0.992) (3.478,0.994) (3.553,0.995) (3.629,0.996) (3.704,0.997) (3.78,0.999) (3.856,1) (3.931,1.001) (4.007,1.002) (4.082,1.003) (4.158,1.004) (4.234,1.005) (4.309,1.006) (4.385,1.008) (4.46,1.009) (4.536,1.01) (4.612,1.011) (4.687,1.012) (4.763,1.013) (4.838,1.014) (4.914,1.016) (4.99,1.017) (5.065,1.018) (5.141,1.019) (5.216,1.02) (5.292,1.021) (5.368,1.022) (5.443,1.024) (5.519,1.025) (5.594,1.026) (5.67,1.027) (5.746,1.028) (5.821,1.029) (5.897,1.03) (5.972,1.031) (6.048,1.033) (6.124,1.034)};
\draw[fwd] plot coordinates {(0,0.82) (0.076,0.833) (0.151,0.846) (0.227,0.859) (0.302,0.871) (0.378,0.883) (0.454,0.895) (0.529,0.907) (0.605,0.919) (0.68,0.93) (0.756,0.941) (0.832,0.952) (0.907,0.963) (0.983,0.973) (1.058,0.984) (1.134,0.993) (1.21,1.003) (1.285,1.012) (1.361,1.022) (1.436,1.03) (1.512,1.039) (1.588,1.047) (1.663,1.055) (1.739,1.062) (1.814,1.069) (1.89,1.075) (1.966,1.082) (2.041,1.087) (2.117,1.092) (2.192,1.097) (2.268,1.102) (2.344,1.106) (2.419,1.109) (2.495,1.112) (2.57,1.114) (2.646,1.116) (2.722,1.118) (2.797,1.119) (2.873,1.119) (2.948,1.119) (3.024,1.119) (3.1,1.119) (3.175,1.118) (3.251,1.117) (3.326,1.115) (3.402,1.114) (3.478,1.112) (3.553,1.11) (3.629,1.109) (3.704,1.107) (3.78,1.105) (3.856,1.103) (3.931,1.101) (4.007,1.099) (4.082,1.097) (4.158,1.095) (4.234,1.092) (4.309,1.09) (4.385,1.088) (4.46,1.086) (4.536,1.084) (4.612,1.082) (4.687,1.08) (4.763,1.078) (4.838,1.076) (4.914,1.074) (4.99,1.072) (5.065,1.07) (5.141,1.068) (5.216,1.066) (5.292,1.064) (5.368,1.062) (5.443,1.06) (5.519,1.058) (5.594,1.056) (5.67,1.053) (5.746,1.051) (5.821,1.049) (5.897,1.047) (5.972,1.045) (6.048,1.043) (6.124,1.041)};
\draw[fwd] plot coordinates {(0,1.036) (0.076,1.047) (0.151,1.057) (0.227,1.067) (0.302,1.077) (0.378,1.087) (0.454,1.096) (0.529,1.106) (0.605,1.115) (0.68,1.124) (0.756,1.133) (0.832,1.142) (0.907,1.15) (0.983,1.159) (1.058,1.167) (1.134,1.175) (1.21,1.182) (1.285,1.19) (1.361,1.197) (1.436,1.204) (1.512,1.211) (1.588,1.217) (1.663,1.223) (1.739,1.229) (1.814,1.234) (1.89,1.239) (1.966,1.244) (2.041,1.248) (2.117,1.252) (2.192,1.256) (2.268,1.258) (2.344,1.261) (2.419,1.263) (2.495,1.265) (2.57,1.265) (2.646,1.266) (2.722,1.266) (2.797,1.265) (2.873,1.264) (2.948,1.262) (3.024,1.26) (3.1,1.257) (3.175,1.254) (3.251,1.251) (3.326,1.247) (3.402,1.242) (3.478,1.238) (3.553,1.233) (3.629,1.228) (3.704,1.223) (3.78,1.217) (3.856,1.212) (3.931,1.207) (4.007,1.201) (4.082,1.196) (4.158,1.19) (4.234,1.185) (4.309,1.18) (4.385,1.174) (4.46,1.169) (4.536,1.163) (4.612,1.158) (4.687,1.152) (4.763,1.147) (4.838,1.142) (4.914,1.136) (4.99,1.131) (5.065,1.125) (5.141,1.12) (5.216,1.114) (5.292,1.109) (5.368,1.103) (5.443,1.098) (5.519,1.093) (5.594,1.087) (5.67,1.082) (5.746,1.076) (5.821,1.071) (5.897,1.065) (5.972,1.06) (6.048,1.054) (6.124,1.049)};
\draw[fwd] plot coordinates {(0,1.252) (0.076,1.26) (0.151,1.268) (0.227,1.275) (0.302,1.283) (0.378,1.29) (0.454,1.297) (0.529,1.304) (0.605,1.311) (0.68,1.318) (0.756,1.325) (0.832,1.331) (0.907,1.338) (0.983,1.344) (1.058,1.35) (1.134,1.356) (1.21,1.362) (1.285,1.367) (1.361,1.373) (1.436,1.378) (1.512,1.383) (1.588,1.388) (1.663,1.392) (1.739,1.396) (1.814,1.4) (1.89,1.404) (1.966,1.407) (2.041,1.41) (2.117,1.413) (2.192,1.415) (2.268,1.417) (2.344,1.419) (2.419,1.42) (2.495,1.42) (2.57,1.42) (2.646,1.42) (2.722,1.418) (2.797,1.417) (2.873,1.414) (2.948,1.412) (3.024,1.408) (3.1,1.404) (3.175,1.399) (3.251,1.393) (3.326,1.387) (3.402,1.38) (3.478,1.373) (3.553,1.366) (3.629,1.357) (3.704,1.349) (3.78,1.34) (3.856,1.332) (3.931,1.323) (4.007,1.314) (4.082,1.305) (4.158,1.295) (4.234,1.286) (4.309,1.277) (4.385,1.268) (4.46,1.259) (4.536,1.25) (4.612,1.241) (4.687,1.231) (4.763,1.222) (4.838,1.213) (4.914,1.204) (4.99,1.195) (5.065,1.186) (5.141,1.177) (5.216,1.167) (5.292,1.158) (5.368,1.149) (5.443,1.14) (5.519,1.131) (5.594,1.122) (5.67,1.113) (5.746,1.103) (5.821,1.094) (5.897,1.085) (5.972,1.076) (6.048,1.067) (6.124,1.058)};
\draw[fwd] plot coordinates {(0,1.468) (0.076,1.473) (0.151,1.478) (0.227,1.483) (0.302,1.488) (0.378,1.493) (0.454,1.498) (0.529,1.503) (0.605,1.507) (0.68,1.512) (0.756,1.516) (0.832,1.521) (0.907,1.525) (0.983,1.529) (1.058,1.533) (1.134,1.537) (1.21,1.541) (1.285,1.545) (1.361,1.548) (1.436,1.552) (1.512,1.555) (1.588,1.558) (1.663,1.561) (1.739,1.564) (1.814,1.567) (1.89,1.569) (1.966,1.571) (2.041,1.573) (2.117,1.575) (2.192,1.576) (2.268,1.577) (2.344,1.578) (2.419,1.578) (2.495,1.578) (2.57,1.578) (2.646,1.577) (2.722,1.576) (2.797,1.574) (2.873,1.571) (2.948,1.568) (3.024,1.564) (3.1,1.56) (3.175,1.554) (3.251,1.548) (3.326,1.541) (3.402,1.533) (3.478,1.525) (3.553,1.515) (3.629,1.505) (3.704,1.493) (3.78,1.482) (3.856,1.469) (3.931,1.457) (4.007,1.444) (4.082,1.431) (4.158,1.417) (4.234,1.404) (4.309,1.391) (4.385,1.377) (4.46,1.364) (4.536,1.35) (4.612,1.337) (4.687,1.323) (4.763,1.31) (4.838,1.296) (4.914,1.283) (4.99,1.269) (5.065,1.256) (5.141,1.243) (5.216,1.229) (5.292,1.216) (5.368,1.202) (5.443,1.189) (5.519,1.175) (5.594,1.162) (5.67,1.148) (5.746,1.135) (5.821,1.122) (5.897,1.108) (5.972,1.095) (6.048,1.081) (6.124,1.068)};
\draw[fwd] plot coordinates {(0,1.684) (0.076,1.687) (0.151,1.689) (0.227,1.692) (0.302,1.694) (0.378,1.697) (0.454,1.699) (0.529,1.701) (0.605,1.704) (0.68,1.706) (0.756,1.708) (0.832,1.71) (0.907,1.713) (0.983,1.715) (1.058,1.717) (1.134,1.719) (1.21,1.721) (1.285,1.722) (1.361,1.724) (1.436,1.726) (1.512,1.728) (1.588,1.729) (1.663,1.731) (1.739,1.732) (1.814,1.733) (1.89,1.734) (1.966,1.735) (2.041,1.736) (2.117,1.737) (2.192,1.738) (2.268,1.738) (2.344,1.739) (2.419,1.739) (2.495,1.739) (2.57,1.738) (2.646,1.738) (2.722,1.737) (2.797,1.735) (2.873,1.734) (2.948,1.732) (3.024,1.729) (3.1,1.726) (3.175,1.722) (3.251,1.718) (3.326,1.713) (3.402,1.707) (3.478,1.7) (3.553,1.691) (3.629,1.682) (3.704,1.671) (3.78,1.659) (3.856,1.645) (3.931,1.63) (4.007,1.614) (4.082,1.597) (4.158,1.579) (4.234,1.56) (4.309,1.541) (4.385,1.522) (4.46,1.503) (4.536,1.484) (4.612,1.465) (4.687,1.446) (4.763,1.426) (4.838,1.407) (4.914,1.388) (4.99,1.369) (5.065,1.35) (5.141,1.33) (5.216,1.311) (5.292,1.292) (5.368,1.273) (5.443,1.254) (5.519,1.235) (5.594,1.215) (5.67,1.196) (5.746,1.177) (5.821,1.158) (5.897,1.139) (5.972,1.119) (6.048,1.1) (6.124,1.081)};
\draw[bwd,densely dotted] plot coordinates {(0,1.9) (0.076,1.9) (0.151,1.9) (0.227,1.9) (0.302,1.9) (0.378,1.9) (0.454,1.9) (0.529,1.9) (0.605,1.9) (0.68,1.9) (0.756,1.9) (0.832,1.9) (0.907,1.9) (0.983,1.9) (1.058,1.9) (1.134,1.9) (1.21,1.9) (1.285,1.9) (1.361,1.9) (1.436,1.9) (1.512,1.9) (1.588,1.9) (1.663,1.9) (1.739,1.9) (1.814,1.9) (1.89,1.9) (1.966,1.9) (2.041,1.9) (2.117,1.9) (2.192,1.9) (2.268,1.9) (2.344,1.9) (2.419,1.9) (2.495,1.9) (2.57,1.9) (2.646,1.9) (2.722,1.9) (2.797,1.9) (2.873,1.9) (2.948,1.9) (3.024,1.9) (3.1,1.9) (3.175,1.9) (3.251,1.9) (3.326,1.9) (3.402,1.9) (3.478,1.9) (3.553,1.9) (3.629,1.9) (3.704,1.9) (3.78,1.9) (3.856,1.9) (3.931,1.9) (4.007,1.9) (4.082,1.9) (4.158,1.9) (4.234,1.9) (4.309,1.9) (4.385,1.9) (4.46,1.9) (4.536,1.9) (4.612,1.9) (4.687,1.9) (4.763,1.9) (4.838,1.9) (4.914,1.9) (4.99,1.9) (5.065,1.9) (5.141,1.9) (5.216,1.9) (5.292,1.9) (5.368,1.9) (5.443,1.9) (5.519,1.9) (5.594,1.9) (5.67,1.9) (5.746,1.9) (5.821,1.9) (5.897,1.9) (5.972,1.9) (6.048,1.9) (6.124,1.9)};
\draw[fwd] plot coordinates {(0,2.116) (0.076,2.113) (0.151,2.111) (0.227,2.108) (0.302,2.106) (0.378,2.103) (0.454,2.101) (0.529,2.099) (0.605,2.096) (0.68,2.094) (0.756,2.092) (0.832,2.09) (0.907,2.087) (0.983,2.085) (1.058,2.083) (1.134,2.081) (1.21,2.079) (1.285,2.078) (1.361,2.076) (1.436,2.074) (1.512,2.072) (1.588,2.071) (1.663,2.069) (1.739,2.068) (1.814,2.067) (1.89,2.066) (1.966,2.065) (2.041,2.064) (2.117,2.063) (2.192,2.062) (2.268,2.062) (2.344,2.061) (2.419,2.061) (2.495,2.061) (2.57,2.062) (2.646,2.062) (2.722,2.063) (2.797,2.065) (2.873,2.066) (2.948,2.068) (3.024,2.071) (3.1,2.074) (3.175,2.078) (3.251,2.082) (3.326,2.087) (3.402,2.093) (3.478,2.1) (3.553,2.109) (3.629,2.118) (3.704,2.129) (3.78,2.141) (3.856,2.155) (3.931,2.17) (4.007,2.186) (4.082,2.203) (4.158,2.221) (4.234,2.24) (4.309,2.259) (4.385,2.278) (4.46,2.297) (4.536,2.316) (4.612,2.335) (4.687,2.354) (4.763,2.374) (4.838,2.393) (4.914,2.412) (4.99,2.431) (5.065,2.45) (5.141,2.47) (5.216,2.489) (5.292,2.508) (5.368,2.527) (5.443,2.546) (5.519,2.565) (5.594,2.585) (5.67,2.604) (5.746,2.623) (5.821,2.642) (5.897,2.661) (5.972,2.681) (6.048,2.7) (6.124,2.719)};
\draw[fwd] plot coordinates {(0,2.332) (0.076,2.327) (0.151,2.322) (0.227,2.317) (0.302,2.312) (0.378,2.307) (0.454,2.302) (0.529,2.297) (0.605,2.293) (0.68,2.288) (0.756,2.284) (0.832,2.279) (0.907,2.275) (0.983,2.271) (1.058,2.267) (1.134,2.263) (1.21,2.259) (1.285,2.255) (1.361,2.252) (1.436,2.248) (1.512,2.245) (1.588,2.242) (1.663,2.239) (1.739,2.236) (1.814,2.233) (1.89,2.231) (1.966,2.229) (2.041,2.227) (2.117,2.225) (2.192,2.224) (2.268,2.223) (2.344,2.222) (2.419,2.222) (2.495,2.222) (2.57,2.222) (2.646,2.223) (2.722,2.224) (2.797,2.226) (2.873,2.229) (2.948,2.232) (3.024,2.236) (3.1,2.24) (3.175,2.246) (3.251,2.252) (3.326,2.259) (3.402,2.267) (3.478,2.275) (3.553,2.285) (3.629,2.295) (3.704,2.307) (3.78,2.318) (3.856,2.331) (3.931,2.343) (4.007,2.356) (4.082,2.369) (4.158,2.383) (4.234,2.396) (4.309,2.409) (4.385,2.423) (4.46,2.436) (4.536,2.45) (4.612,2.463) (4.687,2.477) (4.763,2.49) (4.838,2.504) (4.914,2.517) (4.99,2.531) (5.065,2.544) (5.141,2.557) (5.216,2.571) (5.292,2.584) (5.368,2.598) (5.443,2.611) (5.519,2.625) (5.594,2.638) (5.67,2.652) (5.746,2.665) (5.821,2.678) (5.897,2.692) (5.972,2.705) (6.048,2.719) (6.124,2.732)};
\draw[fwd] plot coordinates {(0,2.548) (0.076,2.54) (0.151,2.532) (0.227,2.525) (0.302,2.517) (0.378,2.51) (0.454,2.503) (0.529,2.496) (0.605,2.489) (0.68,2.482) (0.756,2.475) (0.832,2.469) (0.907,2.462) (0.983,2.456) (1.058,2.45) (1.134,2.444) (1.21,2.438) (1.285,2.433) (1.361,2.427) (1.436,2.422) (1.512,2.417) (1.588,2.412) (1.663,2.408) (1.739,2.404) (1.814,2.4) (1.89,2.396) (1.966,2.393) (2.041,2.39) (2.117,2.387) (2.192,2.385) (2.268,2.383) (2.344,2.381) (2.419,2.38) (2.495,2.38) (2.57,2.38) (2.646,2.38) (2.722,2.382) (2.797,2.383) (2.873,2.386) (2.948,2.388) (3.024,2.392) (3.1,2.396) (3.175,2.401) (3.251,2.407) (3.326,2.413) (3.402,2.42) (3.478,2.427) (3.553,2.434) (3.629,2.443) (3.704,2.451) (3.78,2.46) (3.856,2.468) (3.931,2.477) (4.007,2.486) (4.082,2.495) (4.158,2.505) (4.234,2.514) (4.309,2.523) (4.385,2.532) (4.46,2.541) (4.536,2.55) (4.612,2.559) (4.687,2.569) (4.763,2.578) (4.838,2.587) (4.914,2.596) (4.99,2.605) (5.065,2.614) (5.141,2.623) (5.216,2.633) (5.292,2.642) (5.368,2.651) (5.443,2.66) (5.519,2.669) (5.594,2.678) (5.67,2.687) (5.746,2.697) (5.821,2.706) (5.897,2.715) (5.972,2.724) (6.048,2.733) (6.124,2.742)};
\draw[fwd] plot coordinates {(0,2.764) (0.076,2.753) (0.151,2.743) (0.227,2.733) (0.302,2.723) (0.378,2.713) (0.454,2.704) (0.529,2.694) (0.605,2.685) (0.68,2.676) (0.756,2.667) (0.832,2.658) (0.907,2.65) (0.983,2.641) (1.058,2.633) (1.134,2.625) (1.21,2.618) (1.285,2.61) (1.361,2.603) (1.436,2.596) (1.512,2.589) (1.588,2.583) (1.663,2.577) (1.739,2.571) (1.814,2.566) (1.89,2.561) (1.966,2.556) (2.041,2.552) (2.117,2.548) (2.192,2.544) (2.268,2.542) (2.344,2.539) (2.419,2.537) (2.495,2.535) (2.57,2.535) (2.646,2.534) (2.722,2.534) (2.797,2.535) (2.873,2.536) (2.948,2.538) (3.024,2.54) (3.1,2.543) (3.175,2.546) (3.251,2.549) (3.326,2.553) (3.402,2.558) (3.478,2.562) (3.553,2.567) (3.629,2.572) (3.704,2.577) (3.78,2.583) (3.856,2.588) (3.931,2.593) (4.007,2.599) (4.082,2.604) (4.158,2.61) (4.234,2.615) (4.309,2.62) (4.385,2.626) (4.46,2.631) (4.536,2.637) (4.612,2.642) (4.687,2.648) (4.763,2.653) (4.838,2.658) (4.914,2.664) (4.99,2.669) (5.065,2.675) (5.141,2.68) (5.216,2.686) (5.292,2.691) (5.368,2.697) (5.443,2.702) (5.519,2.707) (5.594,2.713) (5.67,2.718) (5.746,2.724) (5.821,2.729) (5.897,2.735) (5.972,2.74) (6.048,2.746) (6.124,2.751)};
\draw[fwd] plot coordinates {(0,2.98) (0.076,2.967) (0.151,2.954) (0.227,2.941) (0.302,2.929) (0.378,2.917) (0.454,2.905) (0.529,2.893) (0.605,2.881) (0.68,2.87) (0.756,2.859) (0.832,2.848) (0.907,2.837) (0.983,2.827) (1.058,2.816) (1.134,2.807) (1.21,2.797) (1.285,2.788) (1.361,2.778) (1.436,2.77) (1.512,2.761) (1.588,2.753) (1.663,2.745) (1.739,2.738) (1.814,2.731) (1.89,2.725) (1.966,2.718) (2.041,2.713) (2.117,2.708) (2.192,2.703) (2.268,2.698) (2.344,2.694) (2.419,2.691) (2.495,2.688) (2.57,2.686) (2.646,2.684) (2.722,2.682) (2.797,2.681) (2.873,2.681) (2.948,2.681) (3.024,2.681) (3.1,2.681) (3.175,2.682) (3.251,2.683) (3.326,2.685) (3.402,2.686) (3.478,2.688) (3.553,2.69) (3.629,2.691) (3.704,2.693) (3.78,2.695) (3.856,2.697) (3.931,2.699) (4.007,2.701) (4.082,2.703) (4.158,2.705) (4.234,2.708) (4.309,2.71) (4.385,2.712) (4.46,2.714) (4.536,2.716) (4.612,2.718) (4.687,2.72) (4.763,2.722) (4.838,2.724) (4.914,2.726) (4.99,2.728) (5.065,2.73) (5.141,2.732) (5.216,2.734) (5.292,2.736) (5.368,2.738) (5.443,2.74) (5.519,2.742) (5.594,2.744) (5.67,2.747) (5.746,2.749) (5.821,2.751) (5.897,2.753) (5.972,2.755) (6.048,2.757) (6.124,2.759)};
\draw[fwd] plot coordinates {(0,3.195) (0.076,3.18) (0.151,3.165) (0.227,3.15) (0.302,3.135) (0.378,3.12) (0.454,3.106) (0.529,3.092) (0.605,3.078) (0.68,3.064) (0.756,3.051) (0.832,3.037) (0.907,3.025) (0.983,3.012) (1.058,3) (1.134,2.988) (1.21,2.976) (1.285,2.965) (1.361,2.954) (1.436,2.943) (1.512,2.933) (1.588,2.923) (1.663,2.914) (1.739,2.905) (1.814,2.896) (1.89,2.888) (1.966,2.88) (2.041,2.873) (2.117,2.866) (2.192,2.859) (2.268,2.853) (2.344,2.848) (2.419,2.843) (2.495,2.838) (2.57,2.834) (2.646,2.83) (2.722,2.827) (2.797,2.823) (2.873,2.821) (2.948,2.818) (3.024,2.816) (3.1,2.814) (3.175,2.812) (3.251,2.811) (3.326,2.809) (3.402,2.808) (3.478,2.806) (3.553,2.805) (3.629,2.804) (3.704,2.803) (3.78,2.801) (3.856,2.8) (3.931,2.799) (4.007,2.798) (4.082,2.797) (4.158,2.796) (4.234,2.795) (4.309,2.794) (4.385,2.792) (4.46,2.791) (4.536,2.79) (4.612,2.789) (4.687,2.788) (4.763,2.787) (4.838,2.786) (4.914,2.784) (4.99,2.783) (5.065,2.782) (5.141,2.781) (5.216,2.78) (5.292,2.779) (5.368,2.778) (5.443,2.776) (5.519,2.775) (5.594,2.774) (5.67,2.773) (5.746,2.772) (5.821,2.771) (5.897,2.77) (5.972,2.769) (6.048,2.767) (6.124,2.766)};
\draw[fwd] plot coordinates {(0,3.411) (0.076,3.393) (0.151,3.376) (0.227,3.358) (0.302,3.341) (0.378,3.324) (0.454,3.307) (0.529,3.29) (0.605,3.274) (0.68,3.258) (0.756,3.242) (0.832,3.227) (0.907,3.212) (0.983,3.197) (1.058,3.183) (1.134,3.169) (1.21,3.155) (1.285,3.142) (1.361,3.129) (1.436,3.116) (1.512,3.104) (1.588,3.093) (1.663,3.081) (1.739,3.071) (1.814,3.06) (1.89,3.05) (1.966,3.041) (2.041,3.032) (2.117,3.023) (2.192,3.015) (2.268,3.007) (2.344,2.999) (2.419,2.992) (2.495,2.986) (2.57,2.979) (2.646,2.973) (2.722,2.967) (2.797,2.962) (2.873,2.957) (2.948,2.952) (3.024,2.947) (3.1,2.942) (3.175,2.938) (3.251,2.933) (3.326,2.929) (3.402,2.924) (3.478,2.92) (3.553,2.916) (3.629,2.912) (3.704,2.908) (3.78,2.903) (3.856,2.899) (3.931,2.895) (4.007,2.891) (4.082,2.887) (4.158,2.882) (4.234,2.878) (4.309,2.874) (4.385,2.87) (4.46,2.866) (4.536,2.861) (4.612,2.857) (4.687,2.853) (4.763,2.849) (4.838,2.845) (4.914,2.84) (4.99,2.836) (5.065,2.832) (5.141,2.828) (5.216,2.824) (5.292,2.819) (5.368,2.815) (5.443,2.811) (5.519,2.807) (5.594,2.803) (5.67,2.799) (5.746,2.794) (5.821,2.79) (5.897,2.786) (5.972,2.782) (6.048,2.778) (6.124,2.773)};
\draw[fwd] plot coordinates {(0,3.627) (0.076,3.607) (0.151,3.586) (0.227,3.566) (0.302,3.546) (0.378,3.527) (0.454,3.508) (0.529,3.489) (0.605,3.47) (0.68,3.452) (0.756,3.434) (0.832,3.416) (0.907,3.399) (0.983,3.382) (1.058,3.366) (1.134,3.35) (1.21,3.334) (1.285,3.319) (1.361,3.304) (1.436,3.289) (1.512,3.275) (1.588,3.262) (1.663,3.249) (1.739,3.236) (1.814,3.224) (1.89,3.212) (1.966,3.2) (2.041,3.189) (2.117,3.179) (2.192,3.168) (2.268,3.158) (2.344,3.149) (2.419,3.14) (2.495,3.131) (2.57,3.122) (2.646,3.114) (2.722,3.106) (2.797,3.098) (2.873,3.09) (2.948,3.082) (3.024,3.075) (3.1,3.067) (3.175,3.06) (3.251,3.052) (3.326,3.045) (3.402,3.038) (3.478,3.031) (3.553,3.024) (3.629,3.016) (3.704,3.009) (3.78,3.002) (3.856,2.995) (3.931,2.988) (4.007,2.981) (4.082,2.973) (4.158,2.966) (4.234,2.959) (4.309,2.952) (4.385,2.945) (4.46,2.938) (4.536,2.931) (4.612,2.923) (4.687,2.916) (4.763,2.909) (4.838,2.902) (4.914,2.895) (4.99,2.888) (5.065,2.88) (5.141,2.873) (5.216,2.866) (5.292,2.859) (5.368,2.852) (5.443,2.845) (5.519,2.838) (5.594,2.83) (5.67,2.823) (5.746,2.816) (5.821,2.809) (5.897,2.802) (5.972,2.795) (6.048,2.787) (6.124,2.78)};
\fill[black] (6.3,2.764) circle (1.8pt);
\fill[black] (6.3,1.036) circle (1.8pt);
\node[accent] at (2.406,4.016) {\scriptsize $t^*$};
\end{scope}
\end{tikzpicture}
\caption{Characteristics of the flow-matching field from $N(0,1)$ to the two points $\pm1$ (\cref{prop:pitchfork}). The midpoint is attracting until $t^*=0.382$ and repelling afterwards; the dotted line is the unstable trajectory. No two characteristics cross.}\label{fig:pitchfork}
\end{figure}

\noindent The field satisfies the continuity equation with $p_t$ to $10^{-13}$; the slope at the midpoint changes
sign at $0.381966$ and is $890$ at $t=0.9$ (\cref{fig:pitchfork}). \chk{F3} This is the transport-side picture of
\cref{thm:illposed}(c): mass begins to be committed to one of the two points at a definite time $t^*$, by the bifurcation
that \cite[Thm.~14.7]{PDEHC} describes for a buckling rod, and the field then steepens without bound as the
characteristics focus. Such transition times have been studied for diffusion models in high dimension
\cite{BBDM24}.

\section{Graph networks}\label{sec:graphs}

Let $L$ be the Laplacian of a connected graph with eigenvalues $0=\lambda_1<\lambda_2\le\dots\le\lambda_n$. The
linear part of a message-passing layer is $X\mapsto(I-\tau L)X$, one explicit Euler step of the heat equation on
the graph \cite{CRGBB21}.

\begin{theorem}[Oversmoothing is the spectral gap]\label{thm:oversmooth}
\Scl[\cite{LHW18,OoSu20}] For $0<\tau\le1/\lambda_n$ the non-constant part of $X$ is multiplied by at most
$1-\tau\lambda_2$ per layer, with equality in the limit, and the Dirichlet energy $X^{\T}LX$ by at most
$(1-\tau\lambda_2)^2$. The depth at which features become indistinguishable is of order $1/\tau\lambda_2$.
\end{theorem}

\noindent On a random connected graph with $12$ nodes, $\lambda_2=0.6251$, $\lambda_n=8.2655$ and the measured
factor is $0.92437$, as predicted; $99\%$ of the Dirichlet energy of a random input is gone after $15$ layers,
the bound from $\lambda_2$ alone being $29$. \chk{G1} The depth scale varies by orders of magnitude with the
graph: on $32$ nodes, with $\tau$ at its largest stable value, it is $415$ layers for a path
($\lambda_2=0.00963$), $5$ for the hypercube ($\lambda_2=2$, $\lambda_n=10$) and $1$ for the complete graph,
where a single layer returns the mean exactly (\cref{fig:smooth}). \chk{G2} Graphs that are good for propagating information are
the ones that oversmooth fastest; the two are the same eigenvalue.

\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{scope}[shift={(0,0)}]
\draw[ink2] (0,3.6)--(0,0)--(6.3,0);
\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $0$};
\draw[ink2] (2.1,0)--(2.1,-0.07) node[below,inner sep=1.5pt]{\scriptsize $20$};
\draw[ink2] (4.2,0)--(4.2,-0.07) node[below,inner sep=1.5pt]{\scriptsize $40$};
\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $60$};
\draw[ink2] (0,0)--(-0.07,0) node[left,inner sep=1.5pt]{\scriptsize $10^{-8}$};
\draw[ink2] (0,0.774)--(-0.07,0.774) node[left,inner sep=1.5pt]{\scriptsize $10^{-6}$};
\draw[ink2] (0,1.548)--(-0.07,1.548) node[left,inner sep=1.5pt]{\scriptsize $10^{-4}$};
\draw[ink2] (0,2.322)--(-0.07,2.322) node[left,inner sep=1.5pt]{\scriptsize $10^{-2}$};
\draw[ink2] (0,3.096)--(-0.07,3.096) node[left,inner sep=1.5pt]{\scriptsize $1$};
\node[below] at (3.15,-0.38) {\scriptsize layers};
\draw[thick,accent] plot coordinates {(0,3.096) (0.105,3.318) (0.21,3.323) (0.315,3.125) (0.42,3.116) (0.525,3.297) (0.63,3.315) (0.735,3.189) (0.84,3.118) (0.945,3.285) (1.05,3.305) (1.155,3.206) (1.26,3.149) (1.365,3.277) (1.47,3.293) (1.575,3.222) (1.68,3.161) (1.785,3.264) (1.89,3.302) (1.995,3.215) (2.1,3.168) (2.205,3.264) (2.31,3.293) (2.415,3.222) (2.52,3.201) (2.625,3.253) (2.73,3.265) (2.835,3.248) (2.94,3.234) (3.045,3.237) (3.15,3.237) (3.255,3.249) (3.36,3.263) (3.465,3.249) (3.57,3.206) (3.675,3.219) (3.78,3.298) (3.885,3.255) (3.99,3.183) (4.095,3.206) (4.2,3.291) (4.305,3.289) (4.41,3.172) (4.515,3.166) (4.62,3.298) (4.725,3.307) (4.83,3.122) (4.935,3.166) (5.04,3.311) (5.145,3.303) (5.25,3.103) (5.355,3.16) (5.46,3.322) (5.565,3.302) (5.67,3.089) (5.775,3.138) (5.88,3.326) (5.985,3.311) (6.09,3.088) (6.195,3.134) (6.3,3.311)};
\draw[thick,fwd] plot coordinates {(0,3.096) (0.105,2.805) (0.21,2.707) (0.315,2.632) (0.42,2.569) (0.525,2.514) (0.63,2.466) (0.735,2.424) (0.84,2.388) (0.945,2.356) (1.05,2.329) (1.155,2.305) (1.26,2.284) (1.365,2.266) (1.47,2.249) (1.575,2.234) (1.68,2.22) (1.785,2.206) (1.89,2.194) (1.995,2.182) (2.1,2.17) (2.205,2.159) (2.31,2.148) (2.415,2.137) (2.52,2.127) (2.625,2.116) (2.73,2.106) (2.835,2.096) (2.94,2.087) (3.045,2.077) (3.15,2.067) (3.255,2.058) (3.36,2.049) (3.465,2.04) (3.57,2.031) (3.675,2.022) (3.78,2.014) (3.885,2.005) (3.99,1.997) (4.095,1.988) (4.2,1.98) (4.305,1.972) (4.41,1.964) (4.515,1.956) (4.62,1.949) (4.725,1.941) (4.83,1.934) (4.935,1.926) (5.04,1.919) (5.145,1.912) (5.25,1.905) (5.355,1.898) (5.46,1.891) (5.565,1.884) (5.67,1.878) (5.775,1.871) (5.88,1.865) (5.985,1.858) (6.09,1.852) (6.195,1.846) (6.3,1.84)};
\draw[thick,bwd] plot coordinates {(0,3.096) (0.105,2.849) (0.21,2.662) (0.315,2.52) (0.42,2.407) (0.525,2.31) (0.63,2.223) (0.735,2.141) (0.84,2.062) (0.945,1.984) (1.05,1.908) (1.155,1.832) (1.26,1.757) (1.365,1.682) (1.47,1.606) (1.575,1.531) (1.68,1.456) (1.785,1.381) (1.89,1.306) (1.995,1.231) (2.1,1.156) (2.205,1.081) (2.31,1.006) (2.415,0.931) (2.52,0.856) (2.625,0.781) (2.73,0.706) (2.835,0.631) (2.94,0.556) (3.045,0.481) (3.15,0.406) (3.255,0.331) (3.36,0.256) (3.465,0.181) (3.57,0.106) (3.675,0.031)};
\node[accent] at (4.62,2.74) {\scriptsize wave layer, path};
\node[fwd] at (4.41,2.089) {\scriptsize heat layer, path};
\node[bwd,right] at (2.835,0.959) {\scriptsize heat layer, hypercube};
\end{scope}
\end{tikzpicture}
\caption{Dirichlet energy of a random signal on $32$ nodes against depth, relative to its initial value. Heat-type layers lose it at a rate set by the spectral gap, slowly on a path and quickly on a hypercube; a wave-type layer on the path (\cref{thm:wavegnn}) keeps it.}\label{fig:smooth}
\end{figure}

A layer of second order in depth behaves as a wave equation \cite{RCRMB22}.

\begin{theorem}[Wave layers]\label{thm:wavegnn}
\Srf[\cite{PDEHC}, Thm.~12.3] The recursion $X_{m+1}=2X_m-X_{m-1}-\tau^2LX_m$ conserves
\[ \tau^{-2}\lVert X_{m+1}-X_m\rVert^2+X_{m+1}^{\T}LX_m, \]
which is positive definite on non-constant signals for $\tau^2\lambda_n<4$. For $\tau^2\lambda_n>4$ the top mode
grows per layer by the modulus of the larger root of $\xi^2+(\tau^2\lambda_n-2)\xi+1=0$.
\end{theorem}

\noindent Over $500$ layers at $\tau^2\lambda_n=3.6$ the invariant drifts by $2\times10^{-15}$ and the mean
Dirichlet energy of the last hundred layers is $1.001$ times that of the first hundred: nothing is smoothed. At
$\tau^2\lambda_n=4.4$ the growth factor is $1.8633$. \chk{G3} The bound is the Courant--Friedrichs--Lewy
condition, and the dichotomy between the two theorems is that of \cite[Ch.~4]{PDEHC}: a Markov semigroup
forgets, a symplectic group does not.

Both layers assume that neighbouring nodes should agree. A \term{cellular sheaf} \cite{HaGh19} replaces
agreement by agreement after transport: each edge carries linear maps from the data at its ends, and the sheaf
Laplacian $L_{\mathcal F}$ penalizes the mismatch. Its kernel is the space $H^0$ of global sections. The simplest
case is a sign $\sigma_e=\pm1$ on each edge, with $x^{\T}L_{\mathcal F}x=\sum_{e=uv}(x_u-\sigma_ex_v)^2$.

\begin{theorem}[Sheaf diffusion]\label{thm:sheaf}
\Sli[\cite{HaGh19,BDCLB22}] Diffusion by $L_{\mathcal F}$ converges to the orthogonal projection of the input
onto $H^0$. For a sign sheaf on a connected graph, $H^0$ has dimension one if the signs are a coboundary,
$\sigma_{uv}=\ell_u\ell_v$, in which case it is spanned by $\ell$; otherwise $H^0=0$. The obstruction is the class
of $\sigma$ in $H^1(G;\Z/2)$, a group of order $2^{E-V+1}$.
\end{theorem}

\noindent On a random graph with ten nodes, two classes, and signs $+$ within and $-$ between classes, sheaf
diffusion converges to the class labelling times $\langle\ell,X_0\rangle/n$, to rounding, while ordinary diffusion
converges to a constant and loses the classes. Reversing one edge that lies on a cycle makes the smallest
eigenvalue $0.1606$ and the limit zero; for a cycle of length $7$ with one negative edge it is
$4\sin^2\frac\pi{14}=0.19806$, the antiperiodic Laplacian. \chk{G4} On a graph with $5$ nodes and $7$ edges,
exactly $16=2^{V-1}$ of the $128$ sign assignments have a non-zero section, and they are exactly those whose
Laplacian is singular over the rationals; the $128$ assignments fall into $8=2^{E-V+1}$ classes. \chk{G5} This
is the same first cohomology that classifies flat gauge fields in \cite[Prop.~13.1]{PDEHC}: choosing the
transport maps of a sheaf network is choosing a connection, and heterophily is non-trivial holonomy. A sign
sheaf whose class is non-zero cannot be made consistent by relabelling nodes, and its diffusion has no
non-trivial limit.

\section{Depth as time}\label{sec:depth}

A residual layer $x\mapsto x+hf(x)$ is one step of forward Euler for $\dot x=f(x)$
\cite{E17,HaRu17}. The type of the linearization of $f$ decides which schemes are admissible, as for partial
differential equations \cite[Ch.~12]{PDEHC}.

\begin{theorem}[Layers by type]\label{thm:layers}
\Srf[\cite{HaRu17,CMHRBH18}; \cite{PDEHC}, Thm.~12.2, 12.3]
\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]
\item \emph{Parabolic}, $f=-Ax$ with $A$ positive definite: forward Euler is stable if and only if
$h\le2/\lambda_{\max}(A)$.
\item \emph{Hyperbolic}, $f=Jx$ with $J$ antisymmetric with eigenvalues $\pm i\omega$: forward Euler multiplies
the norm by $\sqrt{1+h^2\omega^2}$ per layer, for every $h$.
\item For $h,\omega>0$, the leapfrog layer $q\mapsto q+hp$ followed by $p\mapsto p-h\omega^2q$, with the new $q$,
has determinant one, conserves $p^2+\omega^2q^2+h\omega^2pq$, and has bounded powers exactly when $h\omega<2$.
\item In this stable range, the Jacobian of $N$ leapfrog layers in coordinates $(\omega q,p)$ has condition number at most
$(1+\frac{h\omega}2)/(1-\frac{h\omega}2)$, for every $N$. The Jacobian of $N$ parabolic layers with
$h\lambda_{\max}<1$ has condition number $\bigl((1-h\lambda_{\min})/(1-h\lambda_{\max})\bigr)^N$.
\end{enumerate}
\end{theorem}
\begin{proof}
(a) and (b) follow from the eigenvalues of $I+hDf$. For (c), direct computation gives the invariant,
which is positive definite when $h\omega<2$. The update matrix has trace $2-(h\omega)^2$ and determinant one.
At $h\omega=2$ it has a nontrivial Jordan block at $-1$, giving linear growth for some data.
For $h\omega>2$ an eigenvalue has modulus greater than one.
(d): a map that preserves a positive definite quadratic form $Q$ is conjugate to an orthogonal one by $Q^{1/2}$,
so its powers have condition number at most that of $Q$.
\end{proof}

\noindent For a random positive definite $A$ of size $6$ the per-layer factor is $0.9814$ at
$h=1.9/\lambda_{\max}$ and $1.1000$ at $2.1/\lambda_{\max}$. \chk{R1} Forward Euler on a rotation grows by
$1.6446$ over time $10$ with $h=0.1$ and by $1.0513$ with $h=0.01$, that is by $e^{h\omega^2T/2}$: the instability
vanishes only in the limit of infinitely many layers. \chk{R2} Leapfrog conserves its invariant to $10^{-14}$ at
$h\omega=1.9$ and grows by $1.8773$ per layer at $2.1$. \chk{R3} With $h\omega=\frac12$ the condition number of the
product of $50$, $200$ and $800$ leapfrog Jacobians is $1.07$, $1.30$ and $1.53$, below the bound $1.667$; for
fifty Euler layers on eigenvalues $1$ and $10$ with $h=0.05$ it is $8.7\times10^{13}$. \chk{R4} Since
back-propagation multiplies by the transposed Jacobians, (d) is a statement about vanishing and exploding
gradients: a symplectic architecture has neither, by the conservation law that a wave equation has and a heat
equation lacks. The price is \cref{thm:wavegnn}: it does not forget either.

\section{Kernels and losses}\label{sec:kernels}

In the kernel regime, gradient descent on the squared loss moves the function by
$f\mapsto f-\eta K(f-y)$ for a positive kernel operator $K$ \cite{JGH18}. For a translation-invariant kernel
whose Fourier coefficients decay like $k^{-2}$, as for the simplest rectifier networks on the circle
\cite{RBADLHBC19}, the model case is $K=(1-\pd_x^2)^{-1}$, the Green's operator of an elliptic equation.

\begin{theorem}[Spectral bias, with aliases]\label{thm:bias}
\Srf[\cite{RBADLHBC19,JGH18}] \emph{(a)} The residual in mode $k$ after $m$ steps is $(1-\eta\mu_k)^m$ times
its initial value, with $\mu_k=(1+k^2)^{-1}$. \emph{(b)} On $N$ equally spaced samples the eigenvalue is
\[ \mu^N_k=\sum_{m\in\Z}\frac1{1+(k+mN)^2}=\frac{\frac\pi N\sinh\frac{2\pi}N}{\cosh\frac{2\pi}N-\cos\frac{2\pi k}N}. \]
\emph{(c)} The interpolant $u=\sum_ic_iK(\cdot-x_i)$ solves $u-u''=\sum_ic_i\delta_{x_i}$.
\end{theorem}

\noindent Statement (b) is the Poisson summation formula of \cite[Thm.~4.6]{PDEHC}: sampling in space is
summation over the dual lattice. On $64$ points $\mu^N_k/\mu_k$ is $1.0016$, $1.0852$ and $1.2346$ at
$k=1,10,16$. \chk{K1} With $\eta=1$ and target $\cos x+\cos8x$, after $200$ steps mode $1$ is fitted to rounding
and $0.0380$ of mode $8$ remains; the steps needed to reach one per cent are $6.6$, $282$ and $956$ for
$k=1,8,16$ (\cref{fig:bias}). \chk{K2} For (c), with five data points the interpolant satisfies the homogeneous equation away from
the data to $10^{-9}$ and its derivative jumps by $-c_i$ at $x_i$: the representer theorem is the statement
that the minimizer is a combination of fundamental solutions. \chk{K5}

\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{scope}[shift={(0,0)}]
\draw[ink2] (0,3.4)--(0,0)--(6.3,0);
\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $0$};
\draw[ink2] (1.575,0)--(1.575,-0.07) node[below,inner sep=1.5pt]{\scriptsize $100$};
\draw[ink2] (3.15,0)--(3.15,-0.07) node[below,inner sep=1.5pt]{\scriptsize $200$};
\draw[ink2] (4.725,0)--(4.725,-0.07) node[below,inner sep=1.5pt]{\scriptsize $300$};
\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $400$};
\draw[ink2] (0,0)--(-0.07,0) node[left,inner sep=1.5pt]{\scriptsize $10^{-3}$};
\draw[ink2] (0,1.092)--(-0.07,1.092) node[left,inner sep=1.5pt]{\scriptsize $10^{-2}$};
\draw[ink2] (0,2.184)--(-0.07,2.184) node[left,inner sep=1.5pt]{\scriptsize $10^{-1}$};
\draw[ink2] (0,3.276)--(-0.07,3.276) node[left,inner sep=1.5pt]{\scriptsize $1$};
\node[below] at (3.15,-0.38) {\scriptsize gradient steps};
\draw[thick,ink2] plot coordinates {(0,3.276) (0.032,2.617) (0.063,1.958) (0.094,1.299) (0.126,0.64)};
\draw[thick,accent] plot coordinates {(0,3.276) (0.032,3.217) (0.063,3.159) (0.094,3.101) (0.126,3.042) (0.158,2.984) (0.189,2.926) (0.221,2.867) (0.252,2.809) (0.283,2.751) (0.315,2.692) (0.346,2.634) (0.378,2.576) (0.409,2.518) (0.441,2.459) (0.472,2.401) (0.504,2.343) (0.535,2.284) (0.567,2.226) (0.599,2.168) (0.63,2.109) (0.661,2.051) (0.693,1.993) (0.725,1.934) (0.756,1.876) (0.787,1.818) (0.819,1.76) (0.85,1.701) (0.882,1.643) (0.913,1.585) (0.945,1.526) (0.976,1.468) (1.008,1.41) (1.04,1.351) (1.071,1.293) (1.103,1.235) (1.134,1.176) (1.165,1.118) (1.197,1.06) (1.228,1.001) (1.26,0.943) (1.292,0.885) (1.323,0.827) (1.354,0.768) (1.386,0.71) (1.417,0.652) (1.449,0.593) (1.48,0.535) (1.512,0.477) (1.543,0.418) (1.575,0.36) (1.607,0.302) (1.638,0.243) (1.669,0.185) (1.701,0.127) (1.732,0.068) (1.764,0.01)};
\draw[thick,fwd] plot coordinates {(0,3.276) (0.032,3.26) (0.063,3.245) (0.094,3.229) (0.126,3.214) (0.158,3.198) (0.189,3.183) (0.221,3.167) (0.252,3.152) (0.283,3.136) (0.315,3.121) (0.346,3.105) (0.378,3.09) (0.409,3.074) (0.441,3.059) (0.472,3.043) (0.504,3.028) (0.535,3.012) (0.567,2.997) (0.599,2.981) (0.63,2.966) (0.661,2.95) (0.693,2.935) (0.725,2.919) (0.756,2.904) (0.787,2.888) (0.819,2.873) (0.85,2.857) (0.882,2.842) (0.913,2.826) (0.945,2.811) (0.976,2.795) (1.008,2.78) (1.04,2.764) (1.071,2.749) (1.103,2.733) (1.134,2.718) (1.165,2.702) (1.197,2.687) (1.228,2.671) (1.26,2.656) (1.292,2.64) (1.323,2.624) (1.354,2.609) (1.386,2.593) (1.417,2.578) (1.449,2.562) (1.48,2.547) (1.512,2.531) (1.543,2.516) (1.575,2.5) (1.607,2.485) (1.638,2.469) (1.669,2.454) (1.701,2.438) (1.732,2.423) (1.764,2.407) (1.795,2.392) (1.827,2.376) (1.859,2.361) (1.89,2.345) (1.921,2.33) (1.953,2.314) (1.984,2.299) (2.016,2.283) (2.047,2.268) (2.079,2.252) (2.111,2.237) (2.142,2.221) (2.173,2.206) (2.205,2.19) (2.236,2.175) (2.268,2.159) (2.3,2.144) (2.331,2.128) (2.362,2.113) (2.394,2.097) (2.425,2.082) (2.457,2.066) (2.489,2.051) (2.52,2.035) (2.551,2.02) (2.583,2.004) (2.615,1.989) (2.646,1.973) (2.678,1.958) (2.709,1.942) (2.74,1.927) (2.772,1.911) (2.803,1.896) (2.835,1.88) (2.866,1.865) (2.898,1.849) (2.929,1.834) (2.961,1.818) (2.993,1.803) (3.024,1.787) (3.056,1.772) (3.087,1.756) (3.118,1.741) (3.15,1.725) (3.181,1.71) (3.213,1.694) (3.244,1.679) (3.276,1.663) (3.308,1.648) (3.339,1.632) (3.371,1.617) (3.402,1.601) (3.433,1.586) (3.465,1.57) (3.496,1.555) (3.528,1.539) (3.559,1.524) (3.591,1.508) (3.623,1.493) (3.654,1.477) (3.686,1.462) (3.717,1.446) (3.748,1.431) (3.78,1.415) (3.811,1.4) (3.843,1.384) (3.874,1.369) (3.906,1.353) (3.938,1.338) (3.969,1.322) (4,1.307) (4.032,1.291) (4.063,1.276) (4.095,1.26) (4.127,1.245) (4.158,1.229) (4.189,1.214) (4.221,1.198) (4.253,1.183) (4.284,1.167) (4.316,1.152) (4.347,1.136) (4.378,1.121) (4.41,1.105) (4.441,1.09) (4.473,1.074) (4.505,1.059) (4.536,1.043) (4.567,1.028) (4.599,1.012) (4.631,0.997) (4.662,0.981) (4.693,0.966) (4.725,0.95) (4.756,0.935) (4.788,0.919) (4.819,0.904) (4.851,0.888) (4.883,0.873) (4.914,0.857) (4.946,0.842) (4.977,0.826) (5.008,0.811) (5.04,0.795) (5.071,0.78) (5.103,0.764) (5.134,0.749) (5.166,0.733) (5.197,0.718) (5.229,0.702) (5.26,0.687) (5.292,0.671) (5.324,0.656) (5.355,0.64) (5.386,0.625) (5.418,0.609) (5.449,0.594) (5.481,0.578) (5.513,0.563) (5.544,0.547) (5.575,0.532) (5.607,0.516) (5.639,0.501) (5.67,0.485) (5.701,0.47) (5.733,0.454) (5.764,0.439) (5.796,0.423) (5.827,0.408) (5.859,0.392) (5.89,0.377) (5.922,0.361) (5.954,0.346) (5.985,0.33) (6.016,0.315) (6.048,0.299) (6.079,0.284) (6.111,0.268) (6.143,0.253) (6.174,0.237) (6.205,0.222) (6.237,0.206) (6.269,0.191) (6.3,0.175)};
\draw[thick,bwd] plot coordinates {(0,3.276) (0.032,3.271) (0.063,3.266) (0.094,3.262) (0.126,3.257) (0.158,3.253) (0.189,3.248) (0.221,3.244) (0.252,3.239) (0.283,3.234) (0.315,3.23) (0.346,3.225) (0.378,3.221) (0.409,3.216) (0.441,3.212) (0.472,3.207) (0.504,3.203) (0.535,3.198) (0.567,3.193) (0.599,3.189) (0.63,3.184) (0.661,3.18) (0.693,3.175) (0.725,3.171) (0.756,3.166) (0.787,3.161) (0.819,3.157) (0.85,3.152) (0.882,3.148) (0.913,3.143) (0.945,3.139) (0.976,3.134) (1.008,3.129) (1.04,3.125) (1.071,3.12) (1.103,3.116) (1.134,3.111) (1.165,3.107) (1.197,3.102) (1.228,3.097) (1.26,3.093) (1.292,3.088) (1.323,3.084) (1.354,3.079) (1.386,3.075) (1.417,3.07) (1.449,3.066) (1.48,3.061) (1.512,3.056) (1.543,3.052) (1.575,3.047) (1.607,3.043) (1.638,3.038) (1.669,3.034) (1.701,3.029) (1.732,3.024) (1.764,3.02) (1.795,3.015) (1.827,3.011) (1.859,3.006) (1.89,3.002) (1.921,2.997) (1.953,2.992) (1.984,2.988) (2.016,2.983) (2.047,2.979) (2.079,2.974) (2.111,2.97) (2.142,2.965) (2.173,2.96) (2.205,2.956) (2.236,2.951) (2.268,2.947) (2.3,2.942) (2.331,2.938) (2.362,2.933) (2.394,2.929) (2.425,2.924) (2.457,2.919) (2.489,2.915) (2.52,2.91) (2.551,2.906) (2.583,2.901) (2.615,2.897) (2.646,2.892) (2.678,2.887) (2.709,2.883) (2.74,2.878) (2.772,2.874) (2.803,2.869) (2.835,2.865) (2.866,2.86) (2.898,2.855) (2.929,2.851) (2.961,2.846) (2.993,2.842) (3.024,2.837) (3.056,2.833) (3.087,2.828) (3.118,2.823) (3.15,2.819) (3.181,2.814) (3.213,2.81) (3.244,2.805) (3.276,2.801) (3.308,2.796) (3.339,2.792) (3.371,2.787) (3.402,2.782) (3.433,2.778) (3.465,2.773) (3.496,2.769) (3.528,2.764) (3.559,2.76) (3.591,2.755) (3.623,2.75) (3.654,2.746) (3.686,2.741) (3.717,2.737) (3.748,2.732) (3.78,2.728) (3.811,2.723) (3.843,2.718) (3.874,2.714) (3.906,2.709) (3.938,2.705) (3.969,2.7) (4,2.696) (4.032,2.691) (4.063,2.686) (4.095,2.682) (4.127,2.677) (4.158,2.673) (4.189,2.668) (4.221,2.664) (4.253,2.659) (4.284,2.655) (4.316,2.65) (4.347,2.645) (4.378,2.641) (4.41,2.636) (4.441,2.632) (4.473,2.627) (4.505,2.623) (4.536,2.618) (4.567,2.613) (4.599,2.609) (4.631,2.604) (4.662,2.6) (4.693,2.595) (4.725,2.591) (4.756,2.586) (4.788,2.581) (4.819,2.577) (4.851,2.572) (4.883,2.568) (4.914,2.563) (4.946,2.559) (4.977,2.554) (5.008,2.549) (5.04,2.545) (5.071,2.54) (5.103,2.536) (5.134,2.531) (5.166,2.527) (5.197,2.522) (5.229,2.518) (5.26,2.513) (5.292,2.508) (5.324,2.504) (5.355,2.499) (5.386,2.495) (5.418,2.49) (5.449,2.486) (5.481,2.481) (5.513,2.476) (5.544,2.472) (5.575,2.467) (5.607,2.463) (5.639,2.458) (5.67,2.454) (5.701,2.449) (5.733,2.444) (5.764,2.44) (5.796,2.435) (5.827,2.431) (5.859,2.426) (5.89,2.422) (5.922,2.417) (5.954,2.412) (5.985,2.408) (6.016,2.403) (6.048,2.399) (6.079,2.394) (6.111,2.39) (6.143,2.385) (6.174,2.381) (6.205,2.376) (6.237,2.371) (6.269,2.367) (6.3,2.362)};
\node[ink2,right] at (0.346,0.521) {\scriptsize $k=1$};
\node[accent,right] at (1.496,1.178) {\scriptsize $4$};
\node[fwd] at (4.567,1.466) {\scriptsize $8$};
\node[bwd] at (5.197,2.75) {\scriptsize $16$};
\end{scope}
\end{tikzpicture}
\caption{Spectral bias in the kernel regime: the remaining fraction of mode $k$ of the target against the number of gradient steps, for the sampled $H^1$ kernel on $64$ points (\cref{thm:bias}).}\label{fig:bias}
\end{figure}

A physics-informed loss \cite{RPK19} for $-u''=f$ is the squared residual $\lVert Au-f\rVert^2$ with
$A=-\pd_x^2$; the energy, or Ritz, loss \cite{EYu18} is $\frac12\langle u,Au\rangle-\langle f,u\rangle$.

\begin{theorem}[Squaring the operator]\label{thm:pinn}
\Scl[cf.\ \cite{KGZKM21,WTP21}] \emph{(a)} The Hessian of the strong-form loss is $A^{\T}A$ and that of the
energy loss is $A$: the condition numbers are $\kappa^2$ and $\kappa$, with $\kappa\sim4N^2/\pi^2$ for $N$
unknowns. \emph{(b)} Composed with the kernel of \cref{thm:bias}, the strong-form loss has mode rates
$\mu_kk^4=k^4/(1+k^2)$, increasing in $k$.
\end{theorem}

\noindent On $31$ unknowns $\kappa=414.3$ and $\kappa^2=171{,}682$; gradient descent with the optimal step needs
$2862$ and $1{,}185{,}936$ iterations to gain six digits. \chk{K3} On $64$ points the regression rates at $k=1$ and
$16$ are $0.5008$ and $0.0048$, a ratio of $1/104$, and the strong-form rates are $0.5000$ and $206.8$, a ratio of
$414$. \chk{K4} The bias is reversed and the spread of rates is four times as large: the low modes, which carry the solution,
are now the slow ones relative to the step that the high modes allow. This
is a linear model of the training difficulties reported for physics-informed networks \cite{KGZKM21,WTP21}, and
it is the reason the weak formulation is the standard one in numerical analysis
\cite[Ch.~12]{PDEHC}.

\section{Noise}\label{sec:noise}

Gradient descent with small isotropic noise on a loss $V$ is modelled by the Langevin equation
$dX=-V'(X)\,dt+\sqrt{2/\beta}\,dW$, whose density solves a Fokker--Planck equation with stationary solution
proportional to $e^{-\beta V}$ \cite{MHB17}.

\begin{proposition}\label{prop:langevin}
\Scl[\cite{Kra40,BaEm85}] \emph{(a)} For $V=\frac12\lambda x^2$ the spectral gap is $\lambda$ for every $\beta$.
\emph{(b)} For a symmetric double well with minima $\pm a$ and barrier $\Delta V$ at $0$, the gap is
$\frac1\pi\sqrt{V''(a)|V''(0)|}\,e^{-\beta\Delta V}(1+O(\beta^{-1}))$. \emph{(c)} Constant-step stochastic gradient
descent on $\frac12\lambda x^2$ with gradient noise of variance $\sigma^2$ has stationary variance
$\eta\sigma^2/\lambda(2-\eta\lambda)$.
\end{proposition}

\noindent A reversible discretization has the Gibbs density as its exact stationary vector, and for $\lambda=1$ its
gap is $0.9997$ at $\beta=1$ and at $\beta=4$. \chk{L1} For $V=(x^2-1)^2$ the gap divided by
$\frac{\sqrt{32}}\pi e^{-\beta}$ is $0.903$, $0.928$, $0.947$, $0.959$ at $\beta=4,6,8,10$, and the gap at
$\beta=10$ is $7.8\times10^{-5}$. \chk{L2} The recursion for the variance converges to $0.0625$ for $\eta=0.1$,
$\lambda=2$, $\sigma=1.5$; the first-order value $\eta\sigma^2/2\lambda$ is $0.05625$, corresponding to the
temperature $\beta^{-1}=\eta\sigma^2/2$. \chk{L3} Within a basin, mixing is as fast as the curvature and does not
depend on the noise; between basins it is exponentially slow in the ratio of barrier height to noise. The
exponent in (b) is where a claim that noise selects wide minima has to be made quantitative.

\section{Control}\label{sec:control}

\begin{proposition}\label{prop:bellman}
\Scl[\cite{CrLi83,BaSo91}] The Bellman operator $TV=\min_a(c_a+\gamma P_aV)$ is monotone and a contraction of
factor $\gamma$ in the supremum norm. A monotone, stable and consistent scheme for a Hamilton--Jacobi equation
converges to its viscosity solution.
\end{proposition}

\noindent For a random problem with $6$ states and $3$ actions, monotonicity and the contraction hold on $300$
random pairs, with largest ratio $0.7072$ at $\gamma=0.9$, and value iteration obeys the geometric bound.
\chk{Q1} For $|u'|=1$ on the unit interval with zero boundary values the scheme
$u_i=\min(u_{i-1},u_{i+1})+h$ converges to the distance to the boundary exactly, and has residual $2h$ on a
sawtooth and on minus the distance, which satisfy the equation almost everywhere. \chk{Q2} Monotonicity is the
discrete maximum principle: the Bellman operator is to control what a Markov kernel is to diffusion, and the
selection of the viscosity solution is the selection of the entropy solution of a conservation law
\cite[Thm.~14.3]{PDEHC}.

\section{How many numbers an operator needs}\label{sec:rank}

Operator learning approximates a solution map by a finite-rank one, composed with nonlinear maps
\cite{LJPZK21,KLLABSA23}. For linear problems the best rank-$r$ error is the $(r+1)$-st singular value.

\begin{theorem}[Rank by type]\label{thm:rank}
\Scl On the unit interval with Dirichlet conditions:
\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]
\item \emph{Elliptic.} $(-\pd_x^2)^{-1}$ has singular values $(\pi k)^{-2}$: relative accuracy $\epsilon$ needs
$\epsilon^{-1/2}$ modes.
\item \emph{Parabolic.} $e^{t\pd_x^2}$ has singular values $e^{-\pi^2k^2t}$: accuracy $\epsilon$ needs
$\sqrt{1+\log(1/\epsilon)/\pi^2t}$ modes.
\item \emph{Hyperbolic.} The wave propagator is an isometry of the energy: every mode is needed.
\item Translates of a step function have covariance eigenvalues $((k-\frac12)\pi)^{-2}$, so the best
$r$-dimensional linear approximation of the family has relative error $\sim(\pi^2r/2)^{-1/2}$.
\end{enumerate}
\end{theorem}

\noindent At $\epsilon=10^{-6}$ this is $1000$ modes for the elliptic inverse and $11$ for the heat flow at
$t=0.01$ ($12$ on a grid of $63$ points, whose stencil underestimates the higher eigenvalues as in
\cref{thm:alias}(b)); the discrete wave propagator preserves the energy form to $10^{-15}$. \chk{O1} For the
steps, the first three eigenvalues are $0.40531$, $0.04505$, $0.01623$ on $60$ points, and the relative error
is $0.142$ with ten modes and $0.045$ with a hundred (\cref{fig:rank}). \chk{O2} A linear-reduction architecture is therefore
well matched to parabolic problems, adequate for elliptic ones, and badly matched to transport, where the
natural coordinates are the characteristics of \cref{sec:flows} and not a fixed basis. This is the Kolmogorov
barrier of reduced-order modelling \cite{OhRa16}, read off the type of the equation.

\begin{figure}[t]
\centering
\begin{tikzpicture}
\begin{scope}[shift={(0,0)}]
\draw[ink2] (0,3.6)--(0,0)--(6.3,0);
\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $1$};
\draw[ink2] (1.454,0)--(1.454,-0.07) node[below,inner sep=1.5pt]{\scriptsize $10$};
\draw[ink2] (3.069,0)--(3.069,-0.07) node[below,inner sep=1.5pt]{\scriptsize $20$};
\draw[ink2] (4.685,0)--(4.685,-0.07) node[below,inner sep=1.5pt]{\scriptsize $30$};
\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $40$};
\draw[ink2] (0,0)--(-0.07,0) node[left,inner sep=1.5pt]{\scriptsize $10^{-6}$};
\draw[ink2] (0,1.112)--(-0.07,1.112) node[left,inner sep=1.5pt]{\scriptsize $10^{-4}$};
\draw[ink2] (0,2.223)--(-0.07,2.223) node[left,inner sep=1.5pt]{\scriptsize $10^{-2}$};
\draw[ink2] (0,3.335)--(-0.07,3.335) node[left,inner sep=1.5pt]{\scriptsize $1$};
\node[below] at (3.15,-0.38) {\scriptsize mode $k$};
\draw[thick,accent] plot coordinates {(0,3.335) (0.162,3.335) (0.323,3.335) (0.485,3.335) (0.646,3.335) (0.808,3.335) (0.969,3.335) (1.131,3.335) (1.292,3.335) (1.454,3.335) (1.615,3.335) (1.777,3.335) (1.938,3.335) (2.1,3.335) (2.262,3.335) (2.423,3.335) (2.585,3.335) (2.746,3.335) (2.908,3.335) (3.069,3.335) (3.231,3.335) (3.392,3.335) (3.554,3.335) (3.715,3.335) (3.877,3.335) (4.038,3.335) (4.2,3.335) (4.362,3.335) (4.523,3.335) (4.685,3.335) (4.846,3.335) (5.008,3.335) (5.169,3.335) (5.331,3.335) (5.492,3.335) (5.654,3.335) (5.815,3.335) (5.977,3.335) (6.138,3.335) (6.3,3.335)};
\draw[thick,ink2,densely dashed] plot coordinates {(0,3.335) (0.162,3.07) (0.323,2.946) (0.485,2.865) (0.646,2.804) (0.808,2.756) (0.969,2.716) (1.131,2.681) (1.292,2.651) (1.454,2.624) (1.615,2.6) (1.777,2.578) (1.938,2.558) (2.1,2.539) (2.262,2.522) (2.423,2.506) (2.585,2.491) (2.746,2.477) (2.908,2.463) (3.069,2.45) (3.231,2.438) (3.392,2.427) (3.554,2.416) (3.715,2.405) (3.877,2.395) (4.038,2.386) (4.2,2.376) (4.362,2.368) (4.523,2.359) (4.685,2.351) (4.846,2.343) (5.008,2.335) (5.169,2.327) (5.331,2.32) (5.492,2.313) (5.654,2.306) (5.815,2.299) (5.977,2.293) (6.138,2.286) (6.3,2.28)};
\draw[thick,fwd] plot coordinates {(0,3.335) (0.162,3) (0.323,2.804) (0.485,2.666) (0.646,2.558) (0.808,2.47) (0.969,2.395) (1.131,2.331) (1.292,2.274) (1.454,2.223) (1.615,2.177) (1.777,2.135) (1.938,2.097) (2.1,2.061) (2.262,2.027) (2.423,1.996) (2.585,1.967) (2.746,1.939) (2.908,1.913) (3.069,1.889) (3.231,1.865) (3.392,1.843) (3.554,1.821) (3.715,1.801) (3.877,1.781) (4.038,1.762) (4.2,1.744) (4.362,1.726) (4.523,1.709) (4.685,1.693) (4.846,1.677) (5.008,1.662) (5.169,1.647) (5.331,1.632) (5.492,1.618) (5.654,1.605) (5.815,1.592) (5.977,1.579) (6.138,1.566) (6.3,1.554)};
\draw[thick,bwd] plot coordinates {(0,3.335) (0.162,3.263) (0.323,3.144) (0.485,2.977) (0.646,2.763) (0.808,2.501) (0.969,2.191) (1.131,1.834) (1.292,1.429) (1.454,0.976) (1.615,0.476)};
\node[accent] at (4.685,3.502) {\scriptsize wave propagator};
\node[ink2] at (4.685,2.656) {\scriptsize translated steps};
\node[fwd] at (4.846,1.342) {\scriptsize elliptic inverse};
\node[bwd,right] at (2.1,0.723) {\scriptsize heat flow};
\end{scope}
\end{tikzpicture}
\caption{Singular values of solution operators on the unit interval, relative to the first, by type (\cref{thm:rank}): none decay for the wave equation, they decay algebraically for the elliptic inverse and for translates of a step, and faster than exponentially for the heat flow at $t=0.01$.}\label{fig:rank}
\end{figure}

\section{Summary of results}\label{sec:summary}

\begin{table*}[t]
\centering\small
\begin{tabular}{@{}>{\raggedright}p{0.15\textwidth}>{\raggedright}p{0.31\textwidth}>{\raggedright}p{0.36\textwidth}l@{}}
\toprule
Topic & Qualitative statement & Quantitative statement & Checks\\
\midrule
Padding & reflect $=$ Neumann by images, iff kernels symmetric; zero $=$ Dirichlet only for radius one & defects $1.18$, $0.100$, $0.160$; mass $0.532$ left after $200$ zero-padded layers & N1--N6\\
Resolution & multipliers natural; stencils to second order; polynomials iff $N>(d+1)K$; rectifiers never & symbol $0.717$, $0.980$ of $k^2$; rectifier error $-\pi/3N^2$ & A1--A4\\
Diffusion models & reversal is ill-posed for densities, well-posed for samples given the score & $5.2\times10^{21}$ against $0.264$; score slope $-1/t+a^2/t^2$ & S3--S5\\
Score matching, entropy & denoising $=$ explicit $+$ constant; de Bruijn; contraction & constant $1/t-J=0.924$; rate $2$, limit $\frac12(\E X_0)^2$ & S2, S6--S8\\
Flow matching & in one dimension, the monotone map for any coupling; commitment by a pitchfork & error $10^{-13}$; $t^*=0.382$; slope $890$ at $t=0.9$ & F1--F3\\
Message passing & oversmoothing is the spectral gap & factor $0.92437$; depth scales $415$, $5$, $1$ & G1, G2\\
Wave layers & conserve an energy; CFL bound & drift $2\times10^{-15}$; growth $1.8633$ beyond & G3\\
Sheaf diffusion & limit is the projection to $H^0$; obstruction in $H^1(G;\Z/2)$ & $16$ of $128$; gaps $0.1606$, $0.19806$ & G4, G5\\
Residual layers & Euler unstable on antisymmetric layers; leapfrog bounded gradients & $1.6446$, $1.0513$; condition $\le1.667$ against $8.7\times10^{13}$ & R1--R4\\
Kernel regression & low modes first; sampled spectrum is a Poisson sum & ratios $1.0016$, $1.0852$, $1.2346$; steps $6.6$, $282$, $956$ & K1, K2, K5\\
Physics-informed loss & squares the condition number and reverses the bias & $414$ against $171{,}682$; rate ratios $1/104$ and $414$ & K3, K4\\
Noise & gap $=$ curvature in a basin, Kramers between basins & $0.9997$; ratios $0.903,\dots,0.959$ & L1--L3\\
Control & Bellman operator monotone contraction; monotone scheme selects the viscosity solution & ratio $\le0.7072$; residual $2h$ on the wrong solutions & Q1, Q2\\
Operator rank & algebraic, exponential, none, by type & $1000$, $11$, all; steps: $0.142$ with ten modes & O1, O2\\
\bottomrule
\end{tabular}
\caption{Qualitative statements with their quantitative counterparts. Every number is recomputed by the companion script.}\label{tab:summary}
\end{table*}

\Cref{tab:summary} pairs the statements. Read by column of the title, the entries on padding, resolution, flow
matching and sheaves are about naturality; those on diffusion models, residual layers, physics-informed losses
and control are about well-posedness; and those on message passing, kernels, noise and rank are about rates. The
divisions are not sharp, because in each case the three questions are answered by the same object: a group
acting on a domain and the spectrum of an operator that commutes with it.

\section{Limits}\label{sec:limits}

\emph{Linearity.} Most results concern the linear part of a layer or the kernel regime of training. Nonlinear
layers enter only where they can be treated exactly: pointwise maps commute with even extension
(\cref{thm:reflect}), and their aliasing is computed (\cref{thm:alias}). Nothing is proved about the training
dynamics of a finite network.

\emph{Dimension.} The examples are one-dimensional or on small graphs. \Cref{thm:reflect} extends to boxes and
to the other chambers of reflection groups \cite[Rem.~3.6]{PDEHC}; \cref{thm:fm} does not extend, and says so.

\emph{The score is assumed.} \Cref{thm:illposed} takes the score as given with a uniform error. How well a
network estimates it, and in which norm, is the statistical question that the theorem isolates and does not
answer.

\emph{Models of noise.} \Cref{prop:langevin} models stochastic gradient descent by isotropic Langevin dynamics,
which is known to be a rough approximation: real gradient noise is anisotropic and state-dependent.

\emph{Checks.} The script verifies instances with a fixed seed. A check that passes on a random instance
supports a statement and does not prove it; the proofs are those given or cited.

\emph{Novelty.} We know of no result here that a specialist in the relevant area would find new. The statements
tagged as observations (\cref{prop:zero}, \cref{thm:fm}, \cref{prop:pitchfork}) are elementary and may well be
in the literature in other words.

\emph{What is higher.} As in \cite{PDEHC}, the categorical content is modest: naturality squares, a group
action with its characters, and cohomology in degree one for sheaves on graphs. The one place where a higher
structure is forced, gauge fields forming a groupoid \cite[Rem.~13.2]{PDEHC}, has a counterpart here in
\cref{thm:sheaf}, where relabelling nodes is a gauge transformation; we have not developed it.

\appendix
\section{The companion script}\label{app:checks}

\texttt{verify-pde-learning.py} runs in about a second and prints one line per check. The checks are:
\textbf{N1--N6} padding rules, their eigenbases, the image constructions and their failures, mass and the
maximum principle;
\textbf{A1--A4} resampling of multipliers, stencils, squares and rectifiers;
\textbf{S1--S8} the heat equation for a Gaussian mixture, Tweedie's formula, the probability flow, sensitivity
to the score, the two-point score, de Bruijn's identity, contraction of relative entropy, and the
score-matching constant;
\textbf{F1--F3} change of variables along characteristics, flow matching between Gaussians, and the two-point
field with its bifurcation time;
\textbf{G1--G5} oversmoothing rates, depth scales on three graphs, the wave layer, sheaf diffusion, and the
count of balanced sign sheaves;
\textbf{R1--R4} stability of Euler and leapfrog layers and the conditioning of their Jacobians;
\textbf{K1--K5} the sampled kernel spectrum, kernel gradient descent, the two condition numbers, the reversed
rates, and the interpolant as a sum of fundamental solutions;
\textbf{L1--L3} Fokker--Planck gaps and the variance of stochastic gradient descent;
\textbf{Q1--Q2} the Bellman operator and the monotone scheme for the eikonal equation;
\textbf{O1--O2} ranks of solution operators and the spectrum of translated steps.
Random inputs are drawn with seed $20261117$. Tolerances are stated in the script next to each check. The curves in the figures were computed from the formulas of the text by a separate plotting script and are embedded in the source as coordinates; the points of \cref{fig:alias} for the random input are the values printed by check A4, and the signal of \cref{fig:smooth} is a random one drawn for the figure.

\begin{thebibliography}{RBADLHBC19}\footnotesize\setlength{\itemsep}{0.5pt}
\newcommand{\bi}[1]{\bibitem[#1]{#1}}
\bi{And82} B.~D.~O.~Anderson, Reverse-time diffusion equation models, \emph{Stochastic Process.\ Appl.} 12 (1982), 313--326.
\bi{BaEm85} D.~Bakry, M.~\'Emery, Diffusions hypercontractives, in \emph{S\'eminaire de Probabilit\'es XIX}, Lecture Notes in Math.\ 1123, Springer, 1985, 177--206.
\bi{BaSo91} G.~Barles, P.~E.~Souganidis, Convergence of approximation schemes for fully nonlinear second order equations, \emph{Asymptotic Anal.} 4 (1991), 271--283.
\bi{BBCV21} M.~M.~Bronstein, J.~Bruna, T.~Cohen, P.~Veli\v{c}kovi\'c, Geometric deep learning: grids, groups, graphs, geodesics, and gauges, arXiv:2104.13478, 2021.
\bi{BBDM24} G.~Biroli, T.~Bonnaire, V.~de Bortoli, M.~M\'ezard, Dynamical regimes of diffusion models, \emph{Nature Commun.} 15 (2024).
\bi{BBRMMA23} F.~Bartolucci, E.~de B\'ezenac, B.~Raoni\'c, R.~Molinaro, S.~Mishra, R.~Alaifari, Representation equivalent neural operators: a framework for alias-free operator learning, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 36 (2023); arXiv:2305.19913. Checked.
\bi{BDCLB22} C.~Bodnar, F.~Di Giovanni, B.~P.~Chamberlain, P.~Li\`o, M.~M.~Bronstein, Neural sheaf diffusion: a topological perspective on heterophily and oversmoothing in GNNs, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 35 (2022); arXiv:2202.04579. Checked.
\bi{CMHRBH18} B.~Chang, L.~Meng, E.~Haber, L.~Ruthotto, D.~Begert, E.~Holtham, Reversible architectures for arbitrarily deep residual neural networks, \emph{Proc.\ AAAI} 32 (2018).
\bi{CoWe16} T.~Cohen, M.~Welling, Group equivariant convolutional networks, \emph{Proc.\ ICML} 2016, 2990--2999.
\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 31 (2018).
\bi{CRGBB21} B.~P.~Chamberlain, J.~Rowbottom, M.~Gorinova, S.~Webb, E.~Rossi, M.~M.~Bronstein, GRAND: graph neural diffusion, \emph{Proc.\ ICML} 2021, 1407--1418.
\bi{CrLi83} M.~G.~Crandall, P.-L.~Lions, Viscosity solutions of Hamilton--Jacobi equations, \emph{Trans.\ Amer.\ Math.\ Soc.} 277 (1983), 1--42.
\bi{DCWJXKY26} Dai, Chen, Wang, Jia, Xie, Kumar, Yu, Learning PDE solvers with physics and data: a unifying view of physics-informed neural networks and neural operators, arXiv:2601.14517, 2026. Checked; surnames only were recorded.
\bi{E17} W.~E, A proposal on machine learning via dynamical systems, \emph{Commun.\ Math.\ Stat.} 5 (2017), 1--11.
\bi{Efr11} B.~Efron, Tweedie's formula and selection bias, \emph{J.~Amer.\ Statist.\ Assoc.} 106 (2011), 1602--1614.
\bi{EYu18} W.~E, B.~Yu, The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems, \emph{Commun.\ Math.\ Stat.} 6 (2018), 1--12.
\bi{HaGh19} J.~Hansen, R.~Ghrist, Toward a spectral theory of cellular sheaves, \emph{J.~Appl.\ Comput.\ Topol.} 3 (2019), 315--358.
\bi{HaRu17} E.~Haber, L.~Ruthotto, Stable architectures for deep neural networks, \emph{Inverse Problems} 34 (2017), 014004.
\bi{HJA20} J.~Ho, A.~Jain, P.~Abbeel, Denoising diffusion probabilistic models, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 33 (2020).
\bi{Hyv05} A.~Hyv\"arinen, Estimation of non-normalized statistical models by score matching, \emph{J.~Mach.\ Learn.\ Res.} 6 (2005), 695--709.
\bi{ILB20} M.~A.~Islam, S.~Jia, N.~D.~B.~Bruce, How much position information do convolutional neural networks encode?, \emph{Proc.\ ICLR} 2020.
\bi{JGH18} A.~Jacot, F.~Gabriel, C.~Hongler, Neural tangent kernel: convergence and generalization in neural networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 31 (2018).
\bi{KALHHLA21} T.~Karras, M.~Aittala, S.~Laine, E.~H\"ark\"onen, J.~Hellsten, J.~Lehtinen, T.~Aila, Alias-free generative adversarial networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 34 (2021).
\bi{KGZKM21} A.~S.~Krishnapriyan, A.~Gholami, S.~Zhe, R.~M.~Kirby, M.~W.~Mahoney, Characterizing possible failure modes in physics-informed neural networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 34 (2021).
\bi{KLLABSA23} N.~Kovachki, Z.~Li, B.~Liu, K.~Azizzadenesheli, K.~Bhattacharya, A.~Stuart, A.~Anandkumar, Neural operator: learning maps between function spaces with applications to PDEs, \emph{J.~Mach.\ Learn.\ Res.} 24 (2023), no.~89.
\bi{Kra40} H.~A.~Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, \emph{Physica} 7 (1940), 284--304.
\bi{LCBNL23} Y.~Lipman, R.~T.~Q.~Chen, H.~Ben-Hamu, M.~Nickel, M.~Le, Flow matching for generative modeling, \emph{Proc.\ ICLR} 2023.
\bi{LGL23} X.~Liu, C.~Gong, Q.~Liu, Flow straight and fast: learning to generate and transfer data with rectified flow, \emph{Proc.\ ICLR} 2023.
\bi{LHW18} Q.~Li, Z.~Han, X.-M.~Wu, Deeper insights into graph convolutional networks for semi-supervised learning, \emph{Proc.\ AAAI} 32 (2018).
\bi{LJPZK21} L.~Lu, P.~Jin, G.~Pang, Z.~Zhang, G.~E.~Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, \emph{Nature Mach.\ Intell.} 3 (2021), 218--229.
\bi{LKALBSA21} Z.~Li, N.~Kovachki, K.~Azizzadenesheli, B.~Liu, K.~Bhattacharya, A.~Stuart, A.~Anandkumar, Fourier neural operator for parametric partial differential equations, \emph{Proc.\ ICLR} 2021.
\bi{MHB17} S.~Mandt, M.~D.~Hoffman, D.~M.~Blei, Stochastic gradient descent as approximate Bayesian inference, \emph{J.~Mach.\ Learn.\ Res.} 18 (2017), no.~134.
\bi{OhRa16} M.~Ohlberger, S.~Rave, Reduced basis methods: success, limitations and future challenges, \emph{Proc.\ ALGORITMY} 2016, 1--12.
\bi{OoSu20} K.~Oono, T.~Suzuki, Graph neural networks exponentially lose expressive power for node classification, \emph{Proc.\ ICLR} 2020.
\bi{Ors71} S.~A.~Orszag, On the elimination of aliasing in finite-difference schemes by filtering high-wavenumber components, \emph{J.~Atmos.\ Sci.} 28 (1971), 1074.
\bi{PDEHC} Anonymous, \emph{Partial Differential Equations in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.
\bi{RBADLHBC19} N.~Rahaman, A.~Baratin, D.~Arpit, F.~Draxler, M.~Lin, F.~Hamprecht, Y.~Bengio, A.~Courville, On the spectral bias of neural networks, \emph{Proc.\ ICML} 2019, 5301--5310.
\bi{RCRMB22} T.~K.~Rusch, B.~P.~Chamberlain, J.~Rowbottom, S.~Mishra, M.~M.~Bronstein, Graph-coupled oscillator networks, \emph{Proc.\ ICML} 2022, 18888--18909.
\bi{RPK19} M.~Raissi, P.~Perdikaris, G.~E.~Karniadakis, Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, \emph{J.~Comput.\ Phys.} 378 (2019), 686--707.
\bi{SSKKEP21} Y.~Song, J.~Sohl-Dickstein, D.~P.~Kingma, A.~Kumar, S.~Ermon, B.~Poole, Score-based generative modeling through stochastic differential equations, \emph{Proc.\ ICLR} 2021.
\bi{Sta59} A.~J.~Stam, Some inequalities satisfied by the quantities of information of Fisher and Shannon, \emph{Inform.\ and Control} 2 (1959), 101--112.
\bi{Str08} W.~A.~Strauss, \emph{Partial Differential Equations: An Introduction}, 2nd ed., Wiley, 2008.
\bi{Str99} G.~Strang, The discrete cosine transform, \emph{SIAM Rev.} 41 (1999), 135--147.
\bi{SWMG15} J.~Sohl-Dickstein, E.~Weiss, N.~Maheswaranathan, S.~Ganguli, Deep unsupervised learning using nonequilibrium thermodynamics, \emph{Proc.\ ICML} 2015, 2256--2265.
\bi{Vin11} P.~Vincent, A connection between score matching and denoising autoencoders, \emph{Neural Comput.} 23 (2011), 1661--1674.
\bi{WTP21} S.~Wang, Y.~Teng, P.~Perdikaris, Understanding and mitigating gradient flow pathologies in physics-informed neural networks, \emph{SIAM J.~Sci.\ Comput.} 43 (2021), A3055--A3081.
\end{thebibliography}
\end{document}

End of manuscript · Back to top ↑