Independent scholarship · Open accessThe monthly digest

Indexed text attachment

partial-differential-equations-in-machine-learning-and-ai.tex

Source for Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates

View paper · Download original file

Lines 925–1024 of 1085. Source text is displayed literally.

925\begin{tikzpicture}926\begin{scope}[shift={(0,0)}]927\draw[ink2] (0,3.6)--(0,0)--(6.3,0);928\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $1$};929\draw[ink2] (1.454,0)--(1.454,-0.07) node[below,inner sep=1.5pt]{\scriptsize $10$};930\draw[ink2] (3.069,0)--(3.069,-0.07) node[below,inner sep=1.5pt]{\scriptsize $20$};931\draw[ink2] (4.685,0)--(4.685,-0.07) node[below,inner sep=1.5pt]{\scriptsize $30$};932\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $40$};933\draw[ink2] (0,0)--(-0.07,0) node[left,inner sep=1.5pt]{\scriptsize $10^{-6}$};934\draw[ink2] (0,1.112)--(-0.07,1.112) node[left,inner sep=1.5pt]{\scriptsize $10^{-4}$};935\draw[ink2] (0,2.223)--(-0.07,2.223) node[left,inner sep=1.5pt]{\scriptsize $10^{-2}$};936\draw[ink2] (0,3.335)--(-0.07,3.335) node[left,inner sep=1.5pt]{\scriptsize $1$};937\node[below] at (3.15,-0.38) {\scriptsize mode $k$};938\draw[thick,accent] plot coordinates {(0,3.335) (0.162,3.335) (0.323,3.335) (0.485,3.335) (0.646,3.335) (0.808,3.335) (0.969,3.335) (1.131,3.335) (1.292,3.335) (1.454,3.335) (1.615,3.335) (1.777,3.335) (1.938,3.335) (2.1,3.335) (2.262,3.335) (2.423,3.335) (2.585,3.335) (2.746,3.335) (2.908,3.335) (3.069,3.335) (3.231,3.335) (3.392,3.335) (3.554,3.335) (3.715,3.335) (3.877,3.335) (4.038,3.335) (4.2,3.335) (4.362,3.335) (4.523,3.335) (4.685,3.335) (4.846,3.335) (5.008,3.335) (5.169,3.335) (5.331,3.335) (5.492,3.335) (5.654,3.335) (5.815,3.335) (5.977,3.335) (6.138,3.335) (6.3,3.335)};939\draw[thick,ink2,densely dashed] plot coordinates {(0,3.335) (0.162,3.07) (0.323,2.946) (0.485,2.865) (0.646,2.804) (0.808,2.756) (0.969,2.716) (1.131,2.681) (1.292,2.651) (1.454,2.624) (1.615,2.6) (1.777,2.578) (1.938,2.558) (2.1,2.539) (2.262,2.522) (2.423,2.506) (2.585,2.491) (2.746,2.477) (2.908,2.463) (3.069,2.45) (3.231,2.438) (3.392,2.427) (3.554,2.416) (3.715,2.405) (3.877,2.395) (4.038,2.386) (4.2,2.376) (4.362,2.368) (4.523,2.359) (4.685,2.351) (4.846,2.343) (5.008,2.335) (5.169,2.327) (5.331,2.32) (5.492,2.313) (5.654,2.306) (5.815,2.299) (5.977,2.293) (6.138,2.286) (6.3,2.28)};940\draw[thick,fwd] plot coordinates {(0,3.335) (0.162,3) (0.323,2.804) (0.485,2.666) (0.646,2.558) (0.808,2.47) (0.969,2.395) (1.131,2.331) (1.292,2.274) (1.454,2.223) (1.615,2.177) (1.777,2.135) (1.938,2.097) (2.1,2.061) (2.262,2.027) (2.423,1.996) (2.585,1.967) (2.746,1.939) (2.908,1.913) (3.069,1.889) (3.231,1.865) (3.392,1.843) (3.554,1.821) (3.715,1.801) (3.877,1.781) (4.038,1.762) (4.2,1.744) (4.362,1.726) (4.523,1.709) (4.685,1.693) (4.846,1.677) (5.008,1.662) (5.169,1.647) (5.331,1.632) (5.492,1.618) (5.654,1.605) (5.815,1.592) (5.977,1.579) (6.138,1.566) (6.3,1.554)};941\draw[thick,bwd] plot coordinates {(0,3.335) (0.162,3.263) (0.323,3.144) (0.485,2.977) (0.646,2.763) (0.808,2.501) (0.969,2.191) (1.131,1.834) (1.292,1.429) (1.454,0.976) (1.615,0.476)};942\node[accent] at (4.685,3.502) {\scriptsize wave propagator};943\node[ink2] at (4.685,2.656) {\scriptsize translated steps};944\node[fwd] at (4.846,1.342) {\scriptsize elliptic inverse};945\node[bwd,right] at (2.1,0.723) {\scriptsize heat flow};946\end{scope}947\end{tikzpicture}948\caption{Singular values of solution operators on the unit interval, relative to the first, by type (\cref{thm:rank}): none decay for the wave equation, they decay algebraically for the elliptic inverse and for translates of a step, and faster than exponentially for the heat flow at $t=0.01$.}\label{fig:rank}949\end{figure}950 951\section{Summary of results}\label{sec:summary}952 953\begin{table*}[t]954\centering\small955\begin{tabular}{@{}>{\raggedright}p{0.15\textwidth}>{\raggedright}p{0.31\textwidth}>{\raggedright}p{0.36\textwidth}l@{}}956\toprule957Topic & Qualitative statement & Quantitative statement & Checks\\958\midrule959Padding & reflect $=$ Neumann by images, iff kernels symmetric; zero $=$ Dirichlet only for radius one & defects $1.18$, $0.100$, $0.160$; mass $0.532$ left after $200$ zero-padded layers & N1--N6\\960Resolution & multipliers natural; stencils to second order; polynomials iff $N>(d+1)K$; rectifiers never & symbol $0.717$, $0.980$ of $k^2$; rectifier error $-\pi/3N^2$ & A1--A4\\961Diffusion models & reversal is ill-posed for densities, well-posed for samples given the score & $5.2\times10^{21}$ against $0.264$; score slope $-1/t+a^2/t^2$ & S3--S5\\962Score matching, entropy & denoising $=$ explicit $+$ constant; de Bruijn; contraction & constant $1/t-J=0.924$; rate $2$, limit $\frac12(\E X_0)^2$ & S2, S6--S8\\963Flow matching & in one dimension, the monotone map for any coupling; commitment by a pitchfork & error $10^{-13}$; $t^*=0.382$; slope $890$ at $t=0.9$ & F1--F3\\964Message passing & oversmoothing is the spectral gap & factor $0.92437$; depth scales $415$, $5$, $1$ & G1, G2\\965Wave layers & conserve an energy; CFL bound & drift $2\times10^{-15}$; growth $1.8633$ beyond & G3\\966Sheaf diffusion & limit is the projection to $H^0$; obstruction in $H^1(G;\Z/2)$ & $16$ of $128$; gaps $0.1606$, $0.19806$ & G4, G5\\967Residual layers & Euler unstable on antisymmetric layers; leapfrog bounded gradients & $1.6446$, $1.0513$; condition $\le1.667$ against $8.7\times10^{13}$ & R1--R4\\968Kernel regression & low modes first; sampled spectrum is a Poisson sum & ratios $1.0016$, $1.0852$, $1.2346$; steps $6.6$, $282$, $956$ & K1, K2, K5\\969Physics-informed loss & squares the condition number and reverses the bias & $414$ against $171{,}682$; rate ratios $1/104$ and $414$ & K3, K4\\970Noise & gap $=$ curvature in a basin, Kramers between basins & $0.9997$; ratios $0.903,\dots,0.959$ & L1--L3\\971Control & Bellman operator monotone contraction; monotone scheme selects the viscosity solution & ratio $\le0.7072$; residual $2h$ on the wrong solutions & Q1, Q2\\972Operator rank & algebraic, exponential, none, by type & $1000$, $11$, all; steps: $0.142$ with ten modes & O1, O2\\973\bottomrule974\end{tabular}975\caption{Qualitative statements with their quantitative counterparts. Every number is recomputed by the companion script.}\label{tab:summary}976\end{table*}977 978\Cref{tab:summary} pairs the statements. Read by column of the title, the entries on padding, resolution, flow979matching and sheaves are about naturality; those on diffusion models, residual layers, physics-informed losses980and control are about well-posedness; and those on message passing, kernels, noise and rank are about rates. The981divisions are not sharp, because in each case the three questions are answered by the same object: a group982acting on a domain and the spectrum of an operator that commutes with it.983 984\section{Limits}\label{sec:limits}985 986\emph{Linearity.} Most results concern the linear part of a layer or the kernel regime of training. Nonlinear987layers enter only where they can be treated exactly: pointwise maps commute with even extension988(\cref{thm:reflect}), and their aliasing is computed (\cref{thm:alias}). Nothing is proved about the training989dynamics of a finite network.990 991\emph{Dimension.} The examples are one-dimensional or on small graphs. \Cref{thm:reflect} extends to boxes and992to the other chambers of reflection groups \cite[Rem.~3.6]{PDEHC}; \cref{thm:fm} does not extend, and says so.993 994\emph{The score is assumed.} \Cref{thm:illposed} takes the score as given with a uniform error. How well a995network estimates it, and in which norm, is the statistical question that the theorem isolates and does not996answer.997 998\emph{Models of noise.} \Cref{prop:langevin} models stochastic gradient descent by isotropic Langevin dynamics,999which is known to be a rough approximation: real gradient noise is anisotropic and state-dependent.1000 1001\emph{Checks.} The script verifies instances with a fixed seed. A check that passes on a random instance1002supports a statement and does not prove it; the proofs are those given or cited.1003 1004\emph{Novelty.} We know of no result here that a specialist in the relevant area would find new. The statements1005tagged as observations (\cref{prop:zero}, \cref{thm:fm}, \cref{prop:pitchfork}) are elementary and may well be1006in the literature in other words.1007 1008\emph{What is higher.} As in \cite{PDEHC}, the categorical content is modest: naturality squares, a group1009action with its characters, and cohomology in degree one for sheaves on graphs. The one place where a higher1010structure is forced, gauge fields forming a groupoid \cite[Rem.~13.2]{PDEHC}, has a counterpart here in1011\cref{thm:sheaf}, where relabelling nodes is a gauge transformation; we have not developed it.1012 1013\appendix1014\section{The companion script}\label{app:checks}1015 1016\texttt{verify-pde-learning.py} runs in about a second and prints one line per check. The checks are:1017\textbf{N1--N6} padding rules, their eigenbases, the image constructions and their failures, mass and the1018maximum principle;1019\textbf{A1--A4} resampling of multipliers, stencils, squares and rectifiers;1020\textbf{S1--S8} the heat equation for a Gaussian mixture, Tweedie's formula, the probability flow, sensitivity1021to the score, the two-point score, de Bruijn's identity, contraction of relative entropy, and the1022score-matching constant;1023\textbf{F1--F3} change of variables along characteristics, flow matching between Gaussians, and the two-point1024field with its bifurcation time;