Indexed text attachment
partial-differential-equations-in-machine-learning-and-ai.tex
Source for Partial Differential Equations in Machine Learning and AI: Naturality, Well-Posedness, and Spectral Rates
View paper · Download original file
Lines 301–400 of 1085. Source text is displayed literally.
301\fill[bwd] (4,-4) circle (1.3pt);302\draw[thick,bwd] (4.4,-4)--(4.4,-4);303\fill[bwd] (4.4,-4) circle (1.3pt);304\draw[thick,bwd] (4.8,-4)--(4.8,-4);305\fill[bwd] (4.8,-4) circle (1.3pt);306\node[anchor=west] at (-3.04,-2.98) {\scriptsize zero};307\fill[fwd!8] (-0.2,-6.72) rectangle (2.2,-5.22);308\draw[ink2] (-3.04,-6)--(5.04,-6);309\draw[accent,densely dashed] (-0.2,-6.72)--(-0.2,-5.22);310\draw[accent,densely dashed] (2.2,-6.72)--(2.2,-5.22);311\draw[thick,bwd] (-2.8,-6)--(-2.8,-5.34);312\fill[bwd] (-2.8,-5.34) circle (1.3pt);313\draw[thick,bwd] (-2.4,-6)--(-2.4,-5.67);314\fill[bwd] (-2.4,-5.67) circle (1.3pt);315\draw[thick,bwd] (-2,-6)--(-2,-5.406);316\fill[bwd] (-2,-5.406) circle (1.3pt);317\draw[thick,bwd] (-1.6,-6)--(-1.6,-5.538);318\fill[bwd] (-1.6,-5.538) circle (1.3pt);319\draw[thick,bwd] (-1.2,-6)--(-1.2,-5.802);320\fill[bwd] (-1.2,-5.802) circle (1.3pt);321\draw[thick,bwd] (-0.8,-6)--(-0.8,-5.637);322\fill[bwd] (-0.8,-5.637) circle (1.3pt);323\draw[thick,bwd] (-0.4,-6)--(-0.4,-5.34);324\fill[bwd] (-0.4,-5.34) circle (1.3pt);325\draw[thick,fwd] (0,-6)--(0,-5.67);326\fill[fwd] (0,-5.67) circle (1.3pt);327\draw[thick,fwd] (0.4,-6)--(0.4,-5.406);328\fill[fwd] (0.4,-5.406) circle (1.3pt);329\draw[thick,fwd] (0.8,-6)--(0.8,-5.538);330\fill[fwd] (0.8,-5.538) circle (1.3pt);331\draw[thick,fwd] (1.2,-6)--(1.2,-5.802);332\fill[fwd] (1.2,-5.802) circle (1.3pt);333\draw[thick,fwd] (1.6,-6)--(1.6,-5.637);334\fill[fwd] (1.6,-5.637) circle (1.3pt);335\draw[thick,fwd] (2,-6)--(2,-5.34);336\fill[fwd] (2,-5.34) circle (1.3pt);337\draw[thick,bwd] (2.4,-6)--(2.4,-5.67);338\fill[bwd] (2.4,-5.67) circle (1.3pt);339\draw[thick,bwd] (2.8,-6)--(2.8,-5.406);340\fill[bwd] (2.8,-5.406) circle (1.3pt);341\draw[thick,bwd] (3.2,-6)--(3.2,-5.538);342\fill[bwd] (3.2,-5.538) circle (1.3pt);343\draw[thick,bwd] (3.6,-6)--(3.6,-5.802);344\fill[bwd] (3.6,-5.802) circle (1.3pt);345\draw[thick,bwd] (4,-6)--(4,-5.637);346\fill[bwd] (4,-5.637) circle (1.3pt);347\draw[thick,bwd] (4.4,-6)--(4.4,-5.34);348\fill[bwd] (4.4,-5.34) circle (1.3pt);349\draw[thick,bwd] (4.8,-6)--(4.8,-5.67);350\fill[bwd] (4.8,-5.67) circle (1.3pt);351\node[anchor=west] at (-3.04,-4.98) {\scriptsize circular};352\end{tikzpicture}353\caption{Four ways of continuing a signal of six samples (shaded) beyond its ends. Reflect padding is the even extension about the half-sample points (dashed), which is the Neumann condition. The odd extension about the neighbouring grid points is the Dirichlet condition; zero padding agrees with it only on the first sample outside. Circular padding is the periodic extension.}\label{fig:pad}354\end{figure}355 356The method of images \cite[Thm.~3.5]{PDEHC} explains this construction. In \cref{fig:pad}, the padding reflects357about half-sample points, repeating the end samples. Let $E\colon\R^n\to\R^{2n}$ be the even extension358$Ex=(x_0,\dots,x_{n-1},x_{n-1},\dots,x_0)$, regarded as a signal on the cycle $\Z/2n$, and let $R$ be the359restriction to the first $n$ entries. The image of $E$ is the space of signals invariant under the reflection360$\rho\colon i\mapsto2n-1-i$.361 362\begin{theorem}[Reflect padding is the Neumann condition]\label{thm:reflect}363\Srf[\cite{Str99}; \cite{PDEHC}, Thm.~3.5] Write $C^{\mathrm{refl}}_w$ and $C^{\mathrm{circ}}_w$ for the layer with364reflect padding on $\R^n$ and circular padding on $\R^{2n}$.365\begin{enumerate}[label=(\alph*),leftmargin=1.6em,itemsep=1pt,topsep=2pt]366\item $C^{\mathrm{refl}}_w=R\,C^{\mathrm{circ}}_wE$ for every kernel.367\item $\rho\,C^{\mathrm{circ}}_w\rho=C^{\mathrm{circ}}_{\check w}$ with $\check w_j=w_{-j}$. Hence if $w$ is symmetric,368$C^{\mathrm{circ}}_wE=E\,C^{\mathrm{refl}}_w$, and a network of symmetric kernels, reflect padding and pointwise369nonlinearities is the restriction to invariant signals of a circular network on the doubled cycle.370\item The circular layer preserves the reflected subspace exactly when the kernel, reduced modulo $2n$ by371summing coefficients with the same residue, is symmetric. An asymmetric periodized kernel sends some372even input outside that subspace.373\end{enumerate}374\end{theorem}375\begin{proof}376(a) The reflect rule is the definition of $E$ read at indices outside $0,\dots,n-1$. (b) Substituting377$i\mapsto2n-1-i$ in the sum reverses the kernel. Pointwise maps commute with $E$ and $R$.378For (c), work with the periodized kernel and set $a=w-\check w$. If the circular layer preserves the379reflected subspace, then $C^{\mathrm{circ}}_a$ annihilates the even input $\delta_0+\delta_{-1}$,380by (b). Its output at $i$ is $a_i+a_{i+1}=0$. Since $a_0=0$, all $a_i$ vanish.381\end{proof}382 383\noindent For a random input and the kernels $(0.2,-0.5,1,-0.5,0.2)$ and $(0.2,-0.5,1,0.7,-0.3)$: identity (a)384holds to rounding for both; the output on the double is invariant to rounding for the first and has an385anti-invariant part of relative size $1.18$ for the second; and two layers on the double differ from two386reflect-padded layers by $0$ when the symmetric kernel comes first and by $0.100$ when the other does.387\chk{N3} The circular model with symmetric kernels is equivariant under the dihedral group of the doubled388cycle. The half-sample construction restricts it to signals fixed by the chosen reflection. The Dirichlet389construction uses the odd extension about the neighbouring grid points.390 391\begin{proposition}[Zero padding is Dirichlet only for short kernels]\label{prop:zero}392\Sob Let $O\colon\R^n\to\R^{2n+2}$ be the odd extension $(0,x,0,-x^{\mathrm{rev}})$. For a symmetric kernel of radius393one, zero padding equals circular convolution on $Ox$ restricted to the original positions. For radius two or394more it does not: the image construction supplies $-x_0$ beyond the boundary where zero padding supplies $0$.395Pointwise maps commute with $O$ only if they are odd.396\end{proposition}397 398\noindent The relative difference is $0$ for $(0.3,1,0.3)$ and $0.160$ for the five-point symmetric kernel above.399\chk{N4} The Dirichlet image construction can differ from zero padding for wider kernels. A rectifier can400also destroy the odd symmetry required by the construction.