Indexed text attachment
partial-differential-equations-in-machine-learning-and-ai.tex
Source for Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates
View paper · Download original file
Lines 965–1064 of 1085. Source text is displayed literally.
965Wave layers & conserve an energy; CFL bound & drift $2\times10^{-15}$; growth $1.8633$ beyond & G3\\966Sheaf diffusion & limit is the projection to $H^0$; obstruction in $H^1(G;\Z/2)$ & $16$ of $128$; gaps $0.1606$, $0.19806$ & G4, G5\\967Residual layers & Euler unstable on antisymmetric layers; leapfrog bounded gradients & $1.6446$, $1.0513$; condition $\le1.667$ against $8.7\times10^{13}$ & R1--R4\\968Kernel regression & low modes first; sampled spectrum is a Poisson sum & ratios $1.0016$, $1.0852$, $1.2346$; steps $6.6$, $282$, $956$ & K1, K2, K5\\969Physics-informed loss & squares the condition number and reverses the bias & $414$ against $171{,}682$; rate ratios $1/104$ and $414$ & K3, K4\\970Noise & gap $=$ curvature in a basin, Kramers between basins & $0.9997$; ratios $0.903,\dots,0.959$ & L1--L3\\971Control & Bellman operator monotone contraction; monotone scheme selects the viscosity solution & ratio $\le0.7072$; residual $2h$ on the wrong solutions & Q1, Q2\\972Operator rank & algebraic, exponential, none, by type & $1000$, $11$, all; steps: $0.142$ with ten modes & O1, O2\\973\bottomrule974\end{tabular}975\caption{Qualitative statements with their quantitative counterparts. Every number is recomputed by the companion script.}\label{tab:summary}976\end{table*}977 978\Cref{tab:summary} pairs the statements. Read by column of the title, the entries on padding, resolution, flow979matching and sheaves are about naturality; those on diffusion models, residual layers, physics-informed losses980and control are about well-posedness; and those on message passing, kernels, noise and rank are about rates. The981divisions are not sharp, because in each case the three questions are answered by the same object: a group982acting on a domain and the spectrum of an operator that commutes with it.983 984\section{Limits}\label{sec:limits}985 986\emph{Linearity.} Most results concern the linear part of a layer or the kernel regime of training. Nonlinear987layers enter only where they can be treated exactly: pointwise maps commute with even extension988(\cref{thm:reflect}), and their aliasing is computed (\cref{thm:alias}). Nothing is proved about the training989dynamics of a finite network.990 991\emph{Dimension.} The examples are one-dimensional or on small graphs. \Cref{thm:reflect} extends to boxes and992to the other chambers of reflection groups \cite[Rem.~3.6]{PDEHC}; \cref{thm:fm} does not extend, and says so.993 994\emph{The score is assumed.} \Cref{thm:illposed} takes the score as given with a uniform error. How well a995network estimates it, and in which norm, is the statistical question that the theorem isolates and does not996answer.997 998\emph{Models of noise.} \Cref{prop:langevin} models stochastic gradient descent by isotropic Langevin dynamics,999which is known to be a rough approximation: real gradient noise is anisotropic and state-dependent.1000 1001\emph{Checks.} The script verifies instances with a fixed seed. A check that passes on a random instance1002supports a statement and does not prove it; the proofs are those given or cited.1003 1004\emph{Novelty.} We know of no result here that a specialist in the relevant area would find new. The statements1005tagged as observations (\cref{prop:zero}, \cref{thm:fm}, \cref{prop:pitchfork}) are elementary and may well be1006in the literature in other words.1007 1008\emph{What is higher.} As in \cite{PDEHC}, the categorical content is modest: naturality squares, a group1009action with its characters, and cohomology in degree one for sheaves on graphs. The one place where a higher1010structure is forced, gauge fields forming a groupoid \cite[Rem.~13.2]{PDEHC}, has a counterpart here in1011\cref{thm:sheaf}, where relabelling nodes is a gauge transformation; we have not developed it.1012 1013\appendix1014\section{The companion script}\label{app:checks}1015 1016\texttt{verify-pde-learning.py} runs in about a second and prints one line per check. The checks are:1017\textbf{N1--N6} padding rules, their eigenbases, the image constructions and their failures, mass and the1018maximum principle;1019\textbf{A1--A4} resampling of multipliers, stencils, squares and rectifiers;1020\textbf{S1--S8} the heat equation for a Gaussian mixture, Tweedie's formula, the probability flow, sensitivity1021to the score, the two-point score, de Bruijn's identity, contraction of relative entropy, and the1022score-matching constant;1023\textbf{F1--F3} change of variables along characteristics, flow matching between Gaussians, and the two-point1024field with its bifurcation time;1025\textbf{G1--G5} oversmoothing rates, depth scales on three graphs, the wave layer, sheaf diffusion, and the1026count of balanced sign sheaves;1027\textbf{R1--R4} stability of Euler and leapfrog layers and the conditioning of their Jacobians;1028\textbf{K1--K5} the sampled kernel spectrum, kernel gradient descent, the two condition numbers, the reversed1029rates, and the interpolant as a sum of fundamental solutions;1030\textbf{L1--L3} Fokker--Planck gaps and the variance of stochastic gradient descent;1031\textbf{Q1--Q2} the Bellman operator and the monotone scheme for the eikonal equation;1032\textbf{O1--O2} ranks of solution operators and the spectrum of translated steps.1033Random inputs are drawn with seed $20261117$. Tolerances are stated in the script next to each check. The curves in the figures were computed from the formulas of the text by a separate plotting script and are embedded in the source as coordinates; the points of \cref{fig:alias} for the random input are the values printed by check A4, and the signal of \cref{fig:smooth} is a random one drawn for the figure.1034 1035\begin{thebibliography}{RBADLHBC19}\footnotesize\setlength{\itemsep}{0.5pt}1036\newcommand{\bi}[1]{\bibitem[#1]{#1}}1037\bi{And82} B.~D.~O.~Anderson, Reverse-time diffusion equation models, \emph{Stochastic Process.\ Appl.} 12 (1982), 313--326.1038\bi{BaEm85} D.~Bakry, M.~\'Emery, Diffusions hypercontractives, in \emph{S\'eminaire de Probabilit\'es XIX}, Lecture Notes in Math.\ 1123, Springer, 1985, 177--206.1039\bi{BaSo91} G.~Barles, P.~E.~Souganidis, Convergence of approximation schemes for fully nonlinear second order equations, \emph{Asymptotic Anal.} 4 (1991), 271--283.1040\bi{BBCV21} M.~M.~Bronstein, J.~Bruna, T.~Cohen, P.~Veli\v{c}kovi\'c, Geometric deep learning: grids, groups, graphs, geodesics, and gauges, arXiv:2104.13478, 2021.1041\bi{BBDM24} G.~Biroli, T.~Bonnaire, V.~de Bortoli, M.~M\'ezard, Dynamical regimes of diffusion models, \emph{Nature Commun.} 15 (2024).1042\bi{BBRMMA23} F.~Bartolucci, E.~de B\'ezenac, B.~Raoni\'c, R.~Molinaro, S.~Mishra, R.~Alaifari, Representation equivalent neural operators: a framework for alias-free operator learning, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 36 (2023); arXiv:2305.19913. Checked.1043\bi{BDCLB22} C.~Bodnar, F.~Di Giovanni, B.~P.~Chamberlain, P.~Li\`o, M.~M.~Bronstein, Neural sheaf diffusion: a topological perspective on heterophily and oversmoothing in GNNs, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 35 (2022); arXiv:2202.04579. Checked.1044\bi{CMHRBH18} B.~Chang, L.~Meng, E.~Haber, L.~Ruthotto, D.~Begert, E.~Holtham, Reversible architectures for arbitrarily deep residual neural networks, \emph{Proc.\ AAAI} 32 (2018).1045\bi{CoWe16} T.~Cohen, M.~Welling, Group equivariant convolutional networks, \emph{Proc.\ ICML} 2016, 2990--2999.1046\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 31 (2018).1047\bi{CRGBB21} B.~P.~Chamberlain, J.~Rowbottom, M.~Gorinova, S.~Webb, E.~Rossi, M.~M.~Bronstein, GRAND: graph neural diffusion, \emph{Proc.\ ICML} 2021, 1407--1418.1048\bi{CrLi83} M.~G.~Crandall, P.-L.~Lions, Viscosity solutions of Hamilton--Jacobi equations, \emph{Trans.\ Amer.\ Math.\ Soc.} 277 (1983), 1--42.1049\bi{DCWJXKY26} Dai, Chen, Wang, Jia, Xie, Kumar, Yu, Learning PDE solvers with physics and data: a unifying view of physics-informed neural networks and neural operators, arXiv:2601.14517, 2026. Checked; surnames only were recorded.1050\bi{E17} W.~E, A proposal on machine learning via dynamical systems, \emph{Commun.\ Math.\ Stat.} 5 (2017), 1--11.1051\bi{Efr11} B.~Efron, Tweedie's formula and selection bias, \emph{J.~Amer.\ Statist.\ Assoc.} 106 (2011), 1602--1614.1052\bi{EYu18} W.~E, B.~Yu, The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems, \emph{Commun.\ Math.\ Stat.} 6 (2018), 1--12.1053\bi{HaGh19} J.~Hansen, R.~Ghrist, Toward a spectral theory of cellular sheaves, \emph{J.~Appl.\ Comput.\ Topol.} 3 (2019), 315--358.1054\bi{HaRu17} E.~Haber, L.~Ruthotto, Stable architectures for deep neural networks, \emph{Inverse Problems} 34 (2017), 014004.1055\bi{HJA20} J.~Ho, A.~Jain, P.~Abbeel, Denoising diffusion probabilistic models, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 33 (2020).1056\bi{Hyv05} A.~Hyv\"arinen, Estimation of non-normalized statistical models by score matching, \emph{J.~Mach.\ Learn.\ Res.} 6 (2005), 695--709.1057\bi{ILB20} M.~A.~Islam, S.~Jia, N.~D.~B.~Bruce, How much position information do convolutional neural networks encode?, \emph{Proc.\ ICLR} 2020.1058\bi{JGH18} A.~Jacot, F.~Gabriel, C.~Hongler, Neural tangent kernel: convergence and generalization in neural networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 31 (2018).1059\bi{KALHHLA21} T.~Karras, M.~Aittala, S.~Laine, E.~H\"ark\"onen, J.~Hellsten, J.~Lehtinen, T.~Aila, Alias-free generative adversarial networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 34 (2021).1060\bi{KGZKM21} A.~S.~Krishnapriyan, A.~Gholami, S.~Zhe, R.~M.~Kirby, M.~W.~Mahoney, Characterizing possible failure modes in physics-informed neural networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 34 (2021).1061\bi{KLLABSA23} N.~Kovachki, Z.~Li, B.~Liu, K.~Azizzadenesheli, K.~Bhattacharya, A.~Stuart, A.~Anandkumar, Neural operator: learning maps between function spaces with applications to PDEs, \emph{J.~Mach.\ Learn.\ Res.} 24 (2023), no.~89.1062\bi{Kra40} H.~A.~Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, \emph{Physica} 7 (1940), 284--304.1063\bi{LCBNL23} Y.~Lipman, R.~T.~Q.~Chen, H.~Ben-Hamu, M.~Nickel, M.~Le, Flow matching for generative modeling, \emph{Proc.\ ICLR} 2023.1064\bi{LGL23} X.~Liu, C.~Gong, Q.~Liu, Flow straight and fast: learning to generate and transfer data with rectified flow, \emph{Proc.\ ICLR} 2023.