Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1601–1700 of 2141. Source text is displayed literally.
1601$\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu)$. Hence if the supports of $\mu$ and $\nu$ are not1602homeomorphic, no homeomorphism satisfies $h_*\mu=\nu$.1603\end{proposition}1604 1605A Gaussian has support $\R^d$. A target supported on two disjoint balls, on a sphere, or on any proper1606closed subset cannot be represented exactly, and Cornish et al.\ show that a sequence of flows approximating1607such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature1608are the ones differential topology suggests: a target supported on a submanifold $M^m\subset\R^d$ is modelled1609by a flow on $\R^m$ followed by an embedding $\R^m\to\R^d$, and when $M$ is not diffeomorphic to an open1610subset of $\R^m$, by several such charts with a discrete index, which is an atlas in the sense of1611\DT{Ch.~1}. The minimal number of charts is an invariant of $M$; for a sphere it is two.1612 1613\section{Equilibrium layers}1614 1615A \term{deep equilibrium layer} outputs a solution $z$ of $z=g_\theta(z,x)$ for a network $g_\theta$. The1616existence and number of solutions is a fixed-point problem, and \DT{Prop.~7.11} describes its three aspects:1617a generic signed count, a derived fibre product, and a trace.1618 1619\begin{theorem}[Equilibria of a bounded layer]\label{thm:deq}\Srf[\DT{Thm.~4.5, Thm.~7.2, Thm.~7.9}]1620Let $g(z)=\tanh(Az+u)$ on $\R^k$, with $A\in\R^{k\times k}$ and $u\in\R^k$.1621\begin{enumerate}[label=(\alph*)]1622\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an1623eigenvalue of $Dg$ there. The fixed points are then finite in number.1624\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed1625points is odd, and at least one exists.1626\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with1627$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.1628\end{enumerate}1629\end{theorem}1630 1631\begin{proof}1632(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,1633which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of1634$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed1635points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For1636$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so1637$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to1638(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.1639At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.1640\end{proof}1641 1642In one dimension, $z=\tanh(az+u)$ has one solution for $a\le1$, and for $a>1$ and small $|u|$ three, with1643signs $+,-,+$: two stable equilibria and an unstable one between them. The script finds the fixed points for1644several $(a,u)$ and for a two-dimensional example and checks that the signed count is $1$.1645 1646Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the1647transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The1648exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set $\Delta$ of1649\cref{ch:decision}; crossing it creates or destroys a pair of equilibria of opposite sign, as in the1650birth--death family of \DT{Thm.~8.12}, and it is there that the output of an equilibrium layer can jump.1651 1652\section*{Notes}1653 1654The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and1655Mazur, Palais). Invertible residual networks are from \cite{BGC19}. The observation that the topology of1656the support obstructs normalizing flows, with a quantitative form, is in \cite{CCDD20}. Deep equilibrium1657models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone1658operator parametrizations) are a separate line of work, and \cref{thm:deq} describes what happens without1659them.1660 1661\section*{Exercises}1662 1663\begin{exercise}\label{exer:banach}1664Prove \cref{prop:iresnet}(a).1665\end{exercise}1666 1667\begin{exercise}\label{exer:det}1668Let $g$ be smooth with $\|Dg\|\le L<1$ everywhere. Show $\det(1+Dg)\ge(1-L)^d$.1669\end{exercise}1670 1671\begin{exercise}\label{exer:deq1}1672For $k=1$ and $u=0$, find the fixed points of $z=\tanh(az)$ as a function of $a$ and verify1673\cref{thm:deq}(b). What happens at $a=1$?1674\end{exercise}1675 1676\begin{exercise}\label{exer:supp}1677Prove \cref{prop:mismatch}.1678\end{exercise}1679%% ---- end ch/08-degree.tex ----1680\part{Assessment}1681%% ---- begin ch/09-coda.tex ----1682\chapter{What Was Used, and What Is Missing}\label{ch:coda}1683 1684\section{The inputs}1685 1686\Cref{tab:used} lists the main statements of this book with the theorems of the companion volume from which1687they follow. Every entry in the right-hand column was known by the mid-1970s.1688 1689\begin{table}[ht]\centering\small1690\begin{tabular}{@{}p{0.47\textwidth}p{0.43\textwidth}@{}}1691\toprule1692statement here & from \cite{DTHC}\\1693\midrule1694backpropagation composes (\cref{thm:backprop}) & reverse chain rule, Prop.~2.11\\1695generic decision boundaries are hypersurfaces (\cref{thm:boundary}) & Sard, Thm.~4.1; regular values, Thm.~1.12\\1696chambers and walls (\cref{thm:chambers,prop:walls}) & fibrewise lemma 4.4; Ehresmann, Thm.~8.5; isotopy extension, Thm.~10.3\\1697every hypersurface is realized (\cref{thm:realize}) & tubular neighbourhoods, Thm.~6.2; openness, Thm.~3.2\\1698surgery along paths (\cref{thm:trace,thm:surgery}) & neat preimages, Thm.~1.17; handles, Thm.~8.6\\1699moves between paths (\cref{prop:moves}) & Cerf, Thm.~8.12\\1700interpolation manifold (\cref{thm:interp}) & regular values, Thm.~1.12; Sard, Thm.~4.1\\