Independent scholarship · Open accessThe monthly digest

Indexed text attachment

partial-differential-equations-in-machine-learning-and-ai.tex

Source for Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates

View paper · Download original file

Lines 441–540 of 1085. Source text is displayed literally.

441 442\begin{figure}[t]443\centering444\begin{tikzpicture}445\begin{scope}[shift={(0,0)}]446\draw[ink2] (0,3.5)--(0,0)--(6.3,0);447\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $8$};448\draw[ink2] (1.26,0)--(1.26,-0.07) node[below,inner sep=1.5pt]{\scriptsize $16$};449\draw[ink2] (2.52,0)--(2.52,-0.07) node[below,inner sep=1.5pt]{\scriptsize $32$};450\draw[ink2] (3.78,0)--(3.78,-0.07) node[below,inner sep=1.5pt]{\scriptsize $64$};451\draw[ink2] (5.04,0)--(5.04,-0.07) node[below,inner sep=1.5pt]{\scriptsize $128$};452\draw[ink2] (6.3,0)--(6.3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $256$};453\draw[ink2] (0,0)--(-0.07,0) node[left,inner sep=1.5pt]{\scriptsize $10^{-5}$};454\draw[ink2] (0,0.814)--(-0.07,0.814) node[left,inner sep=1.5pt]{\scriptsize $10^{-4}$};455\draw[ink2] (0,1.628)--(-0.07,1.628) node[left,inner sep=1.5pt]{\scriptsize $10^{-3}$};456\draw[ink2] (0,2.441)--(-0.07,2.441) node[left,inner sep=1.5pt]{\scriptsize $10^{-2}$};457\draw[ink2] (0,3.255)--(-0.07,3.255) node[left,inner sep=1.5pt]{\scriptsize $10^{-1}$};458\node[below] at (3.15,-0.38) {\scriptsize grid points $N$};459\draw[ink2,densely dashed] plot coordinates {(0,3.319) (6.3,0.87)};460\draw[thick,fwd] plot coordinates {(0,2.619) (1.26,2.126) (2.52,1.636) (3.78,1.146) (5.04,0.656) (6.3,0.166)};461\draw[thick,bwd] plot coordinates {(1.26,3.149) (2.52,2.434) (3.78,1.89) (5.04,1.321) (6.3,0.933)};462\fill[fwd] (0,2.619) circle (1.6pt);463\fill[fwd] (1.26,2.126) circle (1.6pt);464\fill[fwd] (2.52,1.636) circle (1.6pt);465\fill[fwd] (3.78,1.146) circle (1.6pt);466\fill[fwd] (5.04,0.656) circle (1.6pt);467\fill[fwd] (6.3,0.166) circle (1.6pt);468\fill[bwd] (1.26,3.149) circle (1.6pt);469\fill[bwd] (2.52,2.434) circle (1.6pt);470\fill[bwd] (3.78,1.89) circle (1.6pt);471\fill[bwd] (5.04,1.321) circle (1.6pt);472\fill[bwd] (6.3,0.933) circle (1.6pt);473\node[fwd] at (1.666,1.092) {\scriptsize mean of $\relu(\sin x)$};474\node[bwd] at (4.923,2.261) {\scriptsize random input, band $4$};475\node[ink2] at (2.926,2.751) {\scriptsize slope $-2$};476\end{scope}477\end{tikzpicture}478\caption{Aliasing error of a rectifier against the number of grid points, on logarithmic scales: the closed-form case of \cref{thm:alias}(d) and the largest in-band coefficient error for a random input of band $4$. Both follow the slope $-2$.}\label{fig:alias}479\end{figure}480 481\noindent For a random input of band $5$, the multiplier $(1+k^2)^{-1}$ computed on $16$ and on $64$ points agrees482with the exact result to $10^{-14}$; the stencil is in relative error by $0.221$ and $0.0153$, its symbol at $k=5$483being $0.7173$ and $0.9801$ of the true one. \chk{A1} For band $4$ the in-band coefficients of the square are484exact on $13$ points and wrong by $1.4$ on $12$: the ``two-thirds rule'' of spectral methods \cite{Ors71}.485\chk{A2} For the rectifier one case is closed-form: the mean of $\relu(\sin x)$ sampled on $N$ points is486$\frac1N\cot\frac\pi N$ instead of $\frac1\pi$, an error of $-\pi/3N^2$ to leading order; $N^2$ times the error is487$-1.058$, $-1.050$, $-1.048$, $-1.047$ for $N=8,\dots,64$. \chk{A3} For a random input of band $4$ the largest488in-band error of the rectified signal is $7.4\times10^{-2}$, $9.8\times10^{-3}$, $2.1\times10^{-3}$,489$4.2\times10^{-4}$, $1.4\times10^{-4}$ on $16,\dots,256$ points (\cref{fig:alias}). \chk{A4} This is the quantitative content of the490practice of applying nonlinearities on an oversampled grid and filtering \cite{KALHHLA21}: oversampling by a491factor $M$ buys $M^{2}$, and no finite factor buys exactness. An architecture built from (a) alone is492resolution-independent \cite{LKALBSA21}; one with stencils is so to the order of the stencil; one with493rectifiers is so only approximately, at every resolution.494 495\section{Diffusion models}\label{sec:diffusion}496 497Let data $X_0\sim p_0$ on the line be noised by $X_t=X_0+W_t$, so that the density solves the heat equation498$\pd_tp=\frac12\pd_x^2p$, and write $s=\pd_x\log p_t$ for the \term{score}. Our example is a mixture of two499Gaussians, for which everything is closed-form. \chk{S1}500 501\begin{proposition}\label{prop:tweedie}502\Scl[\cite{Efr11,Hyv05,Vin11}] \emph{(a)} $\E[X_0\mid X_t=x]=x+t\,s(x,t)$. \emph{(b)} For every candidate $\sigma$,503\begin{multline*}504\E\bigl|\sigma(X_t)+\tfrac{X_t-X_0}t\bigr|^2\\=\E|\sigma(X_t)-s(X_t,t)|^2+\tfrac1t-J(p_t),505\end{multline*}506where $J$ is the Fisher information: denoising and explicit score matching differ by a constant.507\end{proposition}508 509\noindent Both hold to rounding in the example; for two-point data at $t=\frac12$ the constant is $0.924073$ for510each of two test functions. \chk{S2, S8} Since $\frac12\pd_x^2p=-\pd_x\bigl(p\cdot(-\frac12s)\bigr)$, the heat equation511is also a continuity equation, and its characteristics are the \term{probability flow}512$\dot x=-\frac12s(x,t)$ \cite{SSKKEP21}.513 514\begin{theorem}[Where the ill-posedness goes]\label{thm:illposed}515\Srf[\cite{PDEHC}, Ex.~2.5; \cite{And82,SSKKEP21}]516\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]517\item The backward heat equation multiplies the $k$-th Fourier mode of $p_T$ by $e^{k^2T/2}$: recovering $p_0$518from $p_T$ is ill-posed in every Sobolev norm.519\item Given the score on $[t_0,T]$ with $t_0>0$, the probability flow is an ordinary differential equation with520a Lipschitz field; it transports $p_T$ to $p_{t_0}$, and an error $\epsilon$ in the score, in the supremum norm,521moves each sample by at most $C(t_0)\epsilon$.522\item For data on the two points $\pm a$, $\pd_xs(0,t)=-1/t+a^2/t^2$. The Lipschitz constant of the field is not523integrable at $t=0$, and $C(t_0)\to\infty$ as $t_0\to0$.524\end{enumerate}525\end{theorem}526\begin{proof}527(a) is Fourier analysis. (b) is the uniqueness theorem for characteristics together with Gr\"onwall's528inequality. (c) Here $s=-x/t+(a/t)\tanh(ax/t)$.529\end{proof}530 531\begin{figure}[t]532\centering533\begin{tikzpicture}534\begin{scope}[shift={(0,0)}]535\draw[ink2] (0,3.4)--(0,0)--(6,0);536\draw[ink2] (0,0)--(0,-0.07) node[below,inner sep=1.5pt]{\scriptsize $-2$};537\draw[ink2] (1.5,0)--(1.5,-0.07) node[below,inner sep=1.5pt]{\scriptsize $-1$};538\draw[ink2] (3,0)--(3,-0.07) node[below,inner sep=1.5pt]{\scriptsize $0$};539\draw[ink2] (4.5,0)--(4.5,-0.07) node[below,inner sep=1.5pt]{\scriptsize $1$};540\draw[ink2] (6,0)--(6,-0.07) node[below,inner sep=1.5pt]{\scriptsize $2$};