Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1621–1720 of 2141. Source text is displayed literally.

1621\begin{enumerate}[label=(\alph*)]1622\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an1623eigenvalue of $Dg$ there. The fixed points are then finite in number.1624\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed1625points is odd, and at least one exists.1626\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with1627$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.1628\end{enumerate}1629\end{theorem}1630 1631\begin{proof}1632(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,1633which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of1634$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed1635points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For1636$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so1637$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to1638(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.1639At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.1640\end{proof}1641 1642In one dimension, $z=\tanh(az+u)$ has one solution for $a\le1$, and for $a>1$ and small $|u|$ three, with1643signs $+,-,+$: two stable equilibria and an unstable one between them. The script finds the fixed points for1644several $(a,u)$ and for a two-dimensional example and checks that the signed count is $1$.1645 1646Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the1647transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The1648exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set $\Delta$ of1649\cref{ch:decision}; crossing it creates or destroys a pair of equilibria of opposite sign, as in the1650birth--death family of \DT{Thm.~8.12}, and it is there that the output of an equilibrium layer can jump.1651 1652\section*{Notes}1653 1654The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and1655Mazur, Palais). Invertible residual networks are from \cite{BGC19}. The observation that the topology of1656the support obstructs normalizing flows, with a quantitative form, is in \cite{CCDD20}. Deep equilibrium1657models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone1658operator parametrizations) are a separate line of work, and \cref{thm:deq} describes what happens without1659them.1660 1661\section*{Exercises}1662 1663\begin{exercise}\label{exer:banach}1664Prove \cref{prop:iresnet}(a).1665\end{exercise}1666 1667\begin{exercise}\label{exer:det}1668Let $g$ be smooth with $\|Dg\|\le L<1$ everywhere. Show $\det(1+Dg)\ge(1-L)^d$.1669\end{exercise}1670 1671\begin{exercise}\label{exer:deq1}1672For $k=1$ and $u=0$, find the fixed points of $z=\tanh(az)$ as a function of $a$ and verify1673\cref{thm:deq}(b). What happens at $a=1$?1674\end{exercise}1675 1676\begin{exercise}\label{exer:supp}1677Prove \cref{prop:mismatch}.1678\end{exercise}1679%% ---- end ch/08-degree.tex ----1680\part{Assessment}1681%% ---- begin ch/09-coda.tex ----1682\chapter{What Was Used, and What Is Missing}\label{ch:coda}1683 1684\section{The inputs}1685 1686\Cref{tab:used} lists the main statements of this book with the theorems of the companion volume from which1687they follow. Every entry in the right-hand column was known by the mid-1970s.1688 1689\begin{table}[ht]\centering\small1690\begin{tabular}{@{}p{0.47\textwidth}p{0.43\textwidth}@{}}1691\toprule1692statement here & from \cite{DTHC}\\1693\midrule1694backpropagation composes (\cref{thm:backprop}) & reverse chain rule, Prop.~2.11\\1695generic decision boundaries are hypersurfaces (\cref{thm:boundary}) & Sard, Thm.~4.1; regular values, Thm.~1.12\\1696chambers and walls (\cref{thm:chambers,prop:walls}) & fibrewise lemma 4.4; Ehresmann, Thm.~8.5; isotopy extension, Thm.~10.3\\1697every hypersurface is realized (\cref{thm:realize}) & tubular neighbourhoods, Thm.~6.2; openness, Thm.~3.2\\1698surgery along paths (\cref{thm:trace,thm:surgery}) & neat preimages, Thm.~1.17; handles, Thm.~8.6\\1699moves between paths (\cref{prop:moves}) & Cerf, Thm.~8.12\\1700interpolation manifold (\cref{thm:interp}) & regular values, Thm.~1.12; Sard, Thm.~4.1\\1701barriers (\cref{prop:barrier}) & sublevel functor, Cor.~8.7, Prop.~8.10\\1702first layer embeds (\cref{prop:firstlayer}) & parametric transversality, Thm.~4.5\\1703neural ODEs are diffeotopies (\cref{prop:node}) & flows, Thm.~10.2\\1704augmentation (\cref{thm:augment}) & isotopy in the stable range, Thm.~10.9; Thm.~10.3\\1705odd number of equilibria (\cref{thm:deq}) & degree, Thm.~7.2; Thm.~4.5\\1706\bottomrule1707\end{tabular}1708\caption{What is used.}\label{tab:used}1709\end{table}1710 1711\section{What is and is not new}1712 1713Nothing in this book is a new theorem of differential topology, and most of its statements about networks1714have appeared in the machine learning literature in some special case, with the references given in the1715text: generic smoothness of decision boundaries, the manifold of global minima, the obstruction to neural1716ODEs and its removal by augmentation, the mismatch of supports for flows. What the book adds is the uniform1717derivation, with the hypothesis that makes each statement true stated as a transversality property of the1718architecture (\cref{prin:rich}), and a small number of statements tagged as observations that we have not1719found in this form: the description of the moduli of irreducible one-hidden-layer networks as a1720configuration space, with its monodromy (\cref{prop:moduli}); the wall structure of the discriminant