Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1801–1900 of 2141. Source text is displayed literally.

1801permutation, sign, scaling symmetry & group of invertible $2$-cells & \cref{prop:syms}\\1802redundant hidden unit & reducible parameter; positive-dimensional fibre & \cref{prop:fibres}\\1803space of networks & quotient manifold, or stack & \cref{prop:moduli}\\1804decision boundary & preimage of a regular value & \cref{thm:boundary}\\1805multi-class boundaries & preimage of a stratification & \cref{prop:multiclass}\\1806robust topology of a classifier & chamber of the discriminant complement & \cref{thm:chambers}\\1807change of boundary topology & surgery; critical point of the trace & \cref{thm:surgery}\\1808training run, seen in input space & embedded cobordism with a Morse function & \cref{thm:trace}\\1809two runs with the same endpoints & Cerf moves & \cref{prop:moves}\\1810interpolating parameters & fibre of the evaluation map over a regular value & \cref{thm:interp}\\1811sharpness at a minimum & normal Hessian $J^{\mathsf T}J$ & \cref{prop:hess}\\1812loss barrier, mode connectivity & death level in the degree-zero barcode & \cref{prop:barrier}\\1813saturation, diverging weights & failure of properness & \cref{prop:notproper}\\1814hidden representation & immersion or embedding of the data manifold & \cref{prop:firstlayer}\\1815minimal width & invariance of domain & \cref{thm:obstruction}\\1816neural ODE & diffeotopy & \cref{prop:node}\\1817augmented neural ODE & isotopy of embeddings in the stable range & \cref{thm:augment}\\1818residual block with small branch & diffeomorphism isotopic to the identity & \cref{prop:iresnet}\\1819normalizing flow & diffeomorphism; pushforward of measures & \cref{prop:mismatch}\\1820equilibrium layer & transverse fixed point; Lefschetz count & \cref{thm:deq}\\1821\bottomrule1822\end{longtable}1823}1824%% ---- end ch/A-dictionary.tex ----1825%% ---- begin ch/B-computations.tex ----1826\chapter{Computations}\label{app:computations}1827 1828The script \texttt{verify-parametrized-networks.py} uses only the Python standard library. It implements small feedforward1829networks (widths $(2,3,1)$, $(2,4,4,1)$ and a few others) with forward evaluation, forward-mode1830differentiation by dual numbers and reverse-mode differentiation by backpropagation, a Runge--Kutta1831integrator, a Jacobi eigenvalue routine and a flood-fill component counter. It performs 49 checks in about1832three seconds, each printing \texttt{PASS} or \texttt{FAIL}, and exits with status zero if and only if all1833pass. The checks recompute the examples in the text and test numerical instances of the structural1834statements on random parameters with a fixed seed. They are not proofs, and a statement about almost every1835parameter is tested only at the parameters sampled.1836 1837{\small1838\begin{longtable}{@{}p{0.13\textwidth}p{0.79\textwidth}@{}}1839\toprule1840Chapter & What is checked\\1841\midrule1842\endhead1843\ref{ch:families} & parameter counts (\cref{exer:count}); associativity of composition in $\Para$; the1844derivative of a weight-tied network (\cref{exer:sharing}); the family $\theta+x^3$ (\cref{exer:notrich})\\1845\ref{ch:backprop} & backpropagation against forward mode and finite differences for two architectures, and1846for the input covector (\cref{thm:backprop}); the tangent kernel is positive semidefinite of rank $\rk J$;1847the output dynamics $\dot u=-K(u-y)$ (\cref{prop:ntk}); the loss gradient; the Hessian as Gauss--Newton plus1848curvature term (\cref{exer:hess})\\1849\ref{ch:symmetry} & all $48$ symmetries of the $\tanh$ network $(2,3,1)$ and the size of an orbit1850(\cref{prop:syms}); the scaling symmetry for $\relu$; rank $P=13$ at a random parameter and ranks $10$ and1851$9$ at reducible ones (\cref{thm:ident}, \cref{exer:rankdrop}); for $\relu$, the generators $\xi_j$ in1852$\ker J$ and $\rk J\le P-H$ (\cref{prop:relu}); a path of irreducible networks exchanging two hidden units1853(\cref{prop:moduli})\\1854\ref{ch:decision} & $\partial F/\partial c=1$; degenerate zeros of a one-dimensional network occur only at1855the critical values (\cref{thm:boundary}(a)); the two chambers of $x_1^2-x_2^2+c$ (\cref{exer:crossex});1856three sectors at a triple point (\cref{exer:triple}); the submersion of \cref{prop:walls}\\1857\ref{ch:paths} & component counts, critical points and indices in \cref{ex:twoblobs}; the Euler1858characteristic formula of \cref{prop:chi} for $d\le6$; the Hessian of the height function on the trace1859(\cref{exer:hessgraph}); births and deaths of boundary points on a circle (\cref{exer:d1path})\\1860\ref{ch:loss} & $\rk J=N$ and the dimension $P-N$ of the solution set; the Hessian $J^{\mathsf T}J$ with $4$1861positive and $9$ zero eigenvalues (\cref{thm:interp}); the inequality of \cref{prop:PL}; sublevel1862components, the barrier and the Hessians of \cref{ex:hyperbola}; \cref{prop:notproper}; \cref{exer:deeplin};1863invariance of the loss under the symmetries\\1864\ref{ch:isotopy} & random first layers of width $3$ embed a circle in $\R^5$ (\cref{prop:firstlayer}), and a1865coordinate projection that does not (\cref{exer:proj}); the convex hull property of \cref{thm:obstruction}1866for random injective networks; order preservation on the line; positivity of the Jacobian determinant and1867Liouville's formula for a planar neural ODE (\cref{prop:node}); the swap of \cref{exer:swap}; invariance of1868the linking number under a neural ODE in $\R^3$ (\cref{cor:nodeobst}); separation of nested circles after1869augmentation (\cref{cor:separate})\\1870\ref{ch:degree} & convergence of the fixed-point inversion and the determinant bound for a residual block1871(\cref{prop:iresnet}, \cref{exer:det}); a non-proper local diffeomorphism (\cref{thm:hadamard}); odd numbers1872of equilibria with signed count $1$ in one and two dimensions (\cref{thm:deq})\\1873\bottomrule1874\end{longtable}1875}1876%% ---- end ch/B-computations.tex ----1877%% ---- begin ch/C-solutions.tex ----1878\chapter{Solutions to the Exercises}\label{app:solutions}1879 1880Solutions to all exercises follow, by chapter. Where a solution relies on a computation, the computation is1881also carried out by \texttt{verify-parametrized-networks.py}.1882%% ---- end ch/C-solutions.tex ----1883%% ---- begin sol/01.tex ----1884\section*{Chapter 1}1885 1886\sol{1.1} $(2,3,1)$: $P=3\cdot3+1\cdot4=13$, $H=3$. $(2,4,4,1)$: $P=4\cdot3+4\cdot5+1\cdot5=37$, $H=8$.1887$(d,k,1)$: $P=k(d+1)+(k+1)=k(d+2)+1$, $H=k$.1888 1889\sol{1.2} Let $(\Theta,F)\colon X\to Y$, $(\Theta',G)\colon Y\to Z$, $(\Theta'',K)\colon Z\to V$. Both1890bracketings have underlying function $(\theta,\theta',\theta'',x)\mapsto K(\theta'',G(\theta',F(\theta,x)))$,1891on the parameter spaces $(\Theta\times\Theta')\times\Theta''$ and $\Theta\times(\Theta'\times\Theta'')$1892respectively. The reassociation diffeomorphism $r$ satisfies $F_{\text{left}}=F_{\text{right}}\circ(r\times1893\id_X)$, so $r$ and $r^{-1}$ are $2$-cells, mutually inverse.1894 1895\sol{1.3} The diagonal $\delta\colon\Theta\to\Theta\times\Theta$ gives a $2$-cell1896$(\Theta,(F\circ F)\circ(\delta\times\id))\Rightarrow(\Theta\times\Theta,F\circ F)$, and its source is1897$(\theta,x)\mapsto F(\theta,F(\theta,x))$, the weight-tied network. By the chain rule, the derivative of1898$\theta\mapsto G(\delta(\theta))$ is $\partial_1G+\partial_2G$ evaluated on the diagonal, for1899$G(\theta_1,\theta_2)=F(\theta_2,F(\theta_1,x))$.1900