Indexed text attachment
partial-differential-equations-in-machine-learning-and-ai.tex
Source for Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates
View paper · Download original file
Lines 1001–1085 of 1085. Source text is displayed literally.
1001\emph{Checks.} The script verifies instances with a fixed seed. A check that passes on a random instance1002supports a statement and does not prove it; the proofs are those given or cited.1003 1004\emph{Novelty.} We know of no result here that a specialist in the relevant area would find new. The statements1005tagged as observations (\cref{prop:zero}, \cref{thm:fm}, \cref{prop:pitchfork}) are elementary and may well be1006in the literature in other words.1007 1008\emph{What is higher.} As in \cite{PDEHC}, the categorical content is modest: naturality squares, a group1009action with its characters, and cohomology in degree one for sheaves on graphs. The one place where a higher1010structure is forced, gauge fields forming a groupoid \cite[Rem.~13.2]{PDEHC}, has a counterpart here in1011\cref{thm:sheaf}, where relabelling nodes is a gauge transformation; we have not developed it.1012 1013\appendix1014\section{The companion script}\label{app:checks}1015 1016\texttt{verify-pde-learning.py} runs in about a second and prints one line per check. The checks are:1017\textbf{N1--N6} padding rules, their eigenbases, the image constructions and their failures, mass and the1018maximum principle;1019\textbf{A1--A4} resampling of multipliers, stencils, squares and rectifiers;1020\textbf{S1--S8} the heat equation for a Gaussian mixture, Tweedie's formula, the probability flow, sensitivity1021to the score, the two-point score, de Bruijn's identity, contraction of relative entropy, and the1022score-matching constant;1023\textbf{F1--F3} change of variables along characteristics, flow matching between Gaussians, and the two-point1024field with its bifurcation time;1025\textbf{G1--G5} oversmoothing rates, depth scales on three graphs, the wave layer, sheaf diffusion, and the1026count of balanced sign sheaves;1027\textbf{R1--R4} stability of Euler and leapfrog layers and the conditioning of their Jacobians;1028\textbf{K1--K5} the sampled kernel spectrum, kernel gradient descent, the two condition numbers, the reversed1029rates, and the interpolant as a sum of fundamental solutions;1030\textbf{L1--L3} Fokker--Planck gaps and the variance of stochastic gradient descent;1031\textbf{Q1--Q2} the Bellman operator and the monotone scheme for the eikonal equation;1032\textbf{O1--O2} ranks of solution operators and the spectrum of translated steps.1033Random inputs are drawn with seed $20261117$. Tolerances are stated in the script next to each check. The curves in the figures were computed from the formulas of the text by a separate plotting script and are embedded in the source as coordinates; the points of \cref{fig:alias} for the random input are the values printed by check A4, and the signal of \cref{fig:smooth} is a random one drawn for the figure.1034 1035\begin{thebibliography}{RBADLHBC19}\footnotesize\setlength{\itemsep}{0.5pt}1036\newcommand{\bi}[1]{\bibitem[#1]{#1}}1037\bi{And82} B.~D.~O.~Anderson, Reverse-time diffusion equation models, \emph{Stochastic Process.\ Appl.} 12 (1982), 313--326.1038\bi{BaEm85} D.~Bakry, M.~\'Emery, Diffusions hypercontractives, in \emph{S\'eminaire de Probabilit\'es XIX}, Lecture Notes in Math.\ 1123, Springer, 1985, 177--206.1039\bi{BaSo91} G.~Barles, P.~E.~Souganidis, Convergence of approximation schemes for fully nonlinear second order equations, \emph{Asymptotic Anal.} 4 (1991), 271--283.1040\bi{BBCV21} M.~M.~Bronstein, J.~Bruna, T.~Cohen, P.~Veli\v{c}kovi\'c, Geometric deep learning: grids, groups, graphs, geodesics, and gauges, arXiv:2104.13478, 2021.1041\bi{BBDM24} G.~Biroli, T.~Bonnaire, V.~de Bortoli, M.~M\'ezard, Dynamical regimes of diffusion models, \emph{Nature Commun.} 15 (2024).1042\bi{BBRMMA23} F.~Bartolucci, E.~de B\'ezenac, B.~Raoni\'c, R.~Molinaro, S.~Mishra, R.~Alaifari, Representation equivalent neural operators: a framework for alias-free operator learning, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 36 (2023); arXiv:2305.19913. Checked.1043\bi{BDCLB22} C.~Bodnar, F.~Di Giovanni, B.~P.~Chamberlain, P.~Li\`o, M.~M.~Bronstein, Neural sheaf diffusion: a topological perspective on heterophily and oversmoothing in GNNs, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 35 (2022); arXiv:2202.04579. Checked.1044\bi{CMHRBH18} B.~Chang, L.~Meng, E.~Haber, L.~Ruthotto, D.~Begert, E.~Holtham, Reversible architectures for arbitrarily deep residual neural networks, \emph{Proc.\ AAAI} 32 (2018).1045\bi{CoWe16} T.~Cohen, M.~Welling, Group equivariant convolutional networks, \emph{Proc.\ ICML} 2016, 2990--2999.1046\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 31 (2018).1047\bi{CRGBB21} B.~P.~Chamberlain, J.~Rowbottom, M.~Gorinova, S.~Webb, E.~Rossi, M.~M.~Bronstein, GRAND: graph neural diffusion, \emph{Proc.\ ICML} 2021, 1407--1418.1048\bi{CrLi83} M.~G.~Crandall, P.-L.~Lions, Viscosity solutions of Hamilton--Jacobi equations, \emph{Trans.\ Amer.\ Math.\ Soc.} 277 (1983), 1--42.1049\bi{DCWJXKY26} Dai, Chen, Wang, Jia, Xie, Kumar, Yu, Learning PDE solvers with physics and data: a unifying view of physics-informed neural networks and neural operators, arXiv:2601.14517, 2026. Checked; surnames only were recorded.1050\bi{E17} W.~E, A proposal on machine learning via dynamical systems, \emph{Commun.\ Math.\ Stat.} 5 (2017), 1--11.1051\bi{Efr11} B.~Efron, Tweedie's formula and selection bias, \emph{J.~Amer.\ Statist.\ Assoc.} 106 (2011), 1602--1614.1052\bi{EYu18} W.~E, B.~Yu, The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems, \emph{Commun.\ Math.\ Stat.} 6 (2018), 1--12.1053\bi{HaGh19} J.~Hansen, R.~Ghrist, Toward a spectral theory of cellular sheaves, \emph{J.~Appl.\ Comput.\ Topol.} 3 (2019), 315--358.1054\bi{HaRu17} E.~Haber, L.~Ruthotto, Stable architectures for deep neural networks, \emph{Inverse Problems} 34 (2017), 014004.1055\bi{HJA20} J.~Ho, A.~Jain, P.~Abbeel, Denoising diffusion probabilistic models, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 33 (2020).1056\bi{Hyv05} A.~Hyv\"arinen, Estimation of non-normalized statistical models by score matching, \emph{J.~Mach.\ Learn.\ Res.} 6 (2005), 695--709.1057\bi{ILB20} M.~A.~Islam, S.~Jia, N.~D.~B.~Bruce, How much position information do convolutional neural networks encode?, \emph{Proc.\ ICLR} 2020.1058\bi{JGH18} A.~Jacot, F.~Gabriel, C.~Hongler, Neural tangent kernel: convergence and generalization in neural networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 31 (2018).1059\bi{KALHHLA21} T.~Karras, M.~Aittala, S.~Laine, E.~H\"ark\"onen, J.~Hellsten, J.~Lehtinen, T.~Aila, Alias-free generative adversarial networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 34 (2021).1060\bi{KGZKM21} A.~S.~Krishnapriyan, A.~Gholami, S.~Zhe, R.~M.~Kirby, M.~W.~Mahoney, Characterizing possible failure modes in physics-informed neural networks, \emph{Adv.\ Neural Inf.\ Process.\ Syst.} 34 (2021).1061\bi{KLLABSA23} N.~Kovachki, Z.~Li, B.~Liu, K.~Azizzadenesheli, K.~Bhattacharya, A.~Stuart, A.~Anandkumar, Neural operator: learning maps between function spaces with applications to PDEs, \emph{J.~Mach.\ Learn.\ Res.} 24 (2023), no.~89.1062\bi{Kra40} H.~A.~Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, \emph{Physica} 7 (1940), 284--304.1063\bi{LCBNL23} Y.~Lipman, R.~T.~Q.~Chen, H.~Ben-Hamu, M.~Nickel, M.~Le, Flow matching for generative modeling, \emph{Proc.\ ICLR} 2023.1064\bi{LGL23} X.~Liu, C.~Gong, Q.~Liu, Flow straight and fast: learning to generate and transfer data with rectified flow, \emph{Proc.\ ICLR} 2023.1065\bi{LHW18} Q.~Li, Z.~Han, X.-M.~Wu, Deeper insights into graph convolutional networks for semi-supervised learning, \emph{Proc.\ AAAI} 32 (2018).1066\bi{LJPZK21} L.~Lu, P.~Jin, G.~Pang, Z.~Zhang, G.~E.~Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, \emph{Nature Mach.\ Intell.} 3 (2021), 218--229.1067\bi{LKALBSA21} Z.~Li, N.~Kovachki, K.~Azizzadenesheli, B.~Liu, K.~Bhattacharya, A.~Stuart, A.~Anandkumar, Fourier neural operator for parametric partial differential equations, \emph{Proc.\ ICLR} 2021.1068\bi{MHB17} S.~Mandt, M.~D.~Hoffman, D.~M.~Blei, Stochastic gradient descent as approximate Bayesian inference, \emph{J.~Mach.\ Learn.\ Res.} 18 (2017), no.~134.1069\bi{OhRa16} M.~Ohlberger, S.~Rave, Reduced basis methods: success, limitations and future challenges, \emph{Proc.\ ALGORITMY} 2016, 1--12.1070\bi{OoSu20} K.~Oono, T.~Suzuki, Graph neural networks exponentially lose expressive power for node classification, \emph{Proc.\ ICLR} 2020.1071\bi{Ors71} S.~A.~Orszag, On the elimination of aliasing in finite-difference schemes by filtering high-wavenumber components, \emph{J.~Atmos.\ Sci.} 28 (1971), 1074.1072\bi{PDEHC} Anonymous, \emph{Partial Differential Equations in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.1073\bi{RBADLHBC19} N.~Rahaman, A.~Baratin, D.~Arpit, F.~Draxler, M.~Lin, F.~Hamprecht, Y.~Bengio, A.~Courville, On the spectral bias of neural networks, \emph{Proc.\ ICML} 2019, 5301--5310.1074\bi{RCRMB22} T.~K.~Rusch, B.~P.~Chamberlain, J.~Rowbottom, S.~Mishra, M.~M.~Bronstein, Graph-coupled oscillator networks, \emph{Proc.\ ICML} 2022, 18888--18909.1075\bi{RPK19} M.~Raissi, P.~Perdikaris, G.~E.~Karniadakis, Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, \emph{J.~Comput.\ Phys.} 378 (2019), 686--707.1076\bi{SSKKEP21} Y.~Song, J.~Sohl-Dickstein, D.~P.~Kingma, A.~Kumar, S.~Ermon, B.~Poole, Score-based generative modeling through stochastic differential equations, \emph{Proc.\ ICLR} 2021.1077\bi{Sta59} A.~J.~Stam, Some inequalities satisfied by the quantities of information of Fisher and Shannon, \emph{Inform.\ and Control} 2 (1959), 101--112.1078\bi{Str08} W.~A.~Strauss, \emph{Partial Differential Equations: An Introduction}, 2nd ed., Wiley, 2008.1079\bi{Str99} G.~Strang, The discrete cosine transform, \emph{SIAM Rev.} 41 (1999), 135--147.1080\bi{SWMG15} J.~Sohl-Dickstein, E.~Weiss, N.~Maheswaranathan, S.~Ganguli, Deep unsupervised learning using nonequilibrium thermodynamics, \emph{Proc.\ ICML} 2015, 2256--2265.1081\bi{Vin11} P.~Vincent, A connection between score matching and denoising autoencoders, \emph{Neural Comput.} 23 (2011), 1661--1674.1082\bi{WTP21} S.~Wang, Y.~Teng, P.~Perdikaris, Understanding and mitigating gradient flow pathologies in physics-informed neural networks, \emph{SIAM J.~Sci.\ Comput.} 43 (2021), A3055--A3081.1083\end{thebibliography}1084\end{document}1085