Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1581–1680 of 2141. Source text is displayed literally.

1581\end{proposition}1582 1583\begin{proof}1584(a) $x\mapsto y-g(x)$ is a contraction; and $|f(x)-f(x')|\ge(1-L)|x-x'|$. (b), (c): the eigenvalues of $Dg$1585have modulus at most $L$, so $1+sDg$ is invertible for $s\in[0,1]$, and its determinant is continuous,1586non-zero, and equal to $1$ at $s=0$.1587\end{proof}1588 1589Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning1590networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common1591use all realize diffeomorphisms in the identity component of $\Diff(\R^d)$, and by \DT{Thm.~10.2} every1592such diffeomorphism with compact support is conversely the time-one map of a flow.1593 1594\section{Topological mismatch}1595 1596A normalizing flow models a probability measure $\nu$ on $\R^d$ as the image $h_*\mu$ of a simple measure1597$\mu$, usually a Gaussian, under a diffeomorphism $h$.1598 1599\begin{proposition}\label{prop:mismatch}\Scl[cf.\ Cornish et al.\ \cite{CCDD20}]1600For a homeomorphism $h$ of $\R^d$ and a Borel probability measure $\mu$,1601$\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu)$. Hence if the supports of $\mu$ and $\nu$ are not1602homeomorphic, no homeomorphism satisfies $h_*\mu=\nu$.1603\end{proposition}1604 1605A Gaussian has support $\R^d$. A target supported on two disjoint balls, on a sphere, or on any proper1606closed subset cannot be represented exactly, and Cornish et al.\ show that a sequence of flows approximating1607such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature1608are the ones differential topology suggests: a target supported on a submanifold $M^m\subset\R^d$ is modelled1609by a flow on $\R^m$ followed by an embedding $\R^m\to\R^d$, and when $M$ is not diffeomorphic to an open1610subset of $\R^m$, by several such charts with a discrete index, which is an atlas in the sense of1611\DT{Ch.~1}. The minimal number of charts is an invariant of $M$; for a sphere it is two.1612 1613\section{Equilibrium layers}1614 1615A \term{deep equilibrium layer} outputs a solution $z$ of $z=g_\theta(z,x)$ for a network $g_\theta$. The1616existence and number of solutions is a fixed-point problem, and \DT{Prop.~7.11} describes its three aspects:1617a generic signed count, a derived fibre product, and a trace.1618 1619\begin{theorem}[Equilibria of a bounded layer]\label{thm:deq}\Srf[\DT{Thm.~4.5, Thm.~7.2, Thm.~7.9}]1620Let $g(z)=\tanh(Az+u)$ on $\R^k$, with $A\in\R^{k\times k}$ and $u\in\R^k$.1621\begin{enumerate}[label=(\alph*)]1622\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an1623eigenvalue of $Dg$ there. The fixed points are then finite in number.1624\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed1625points is odd, and at least one exists.1626\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with1627$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.1628\end{enumerate}1629\end{theorem}1630 1631\begin{proof}1632(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,1633which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of1634$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed1635points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For1636$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so1637$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to1638(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.1639At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.1640\end{proof}1641 1642In one dimension, $z=\tanh(az+u)$ has one solution for $a\le1$, and for $a>1$ and small $|u|$ three, with1643signs $+,-,+$: two stable equilibria and an unstable one between them. The script finds the fixed points for1644several $(a,u)$ and for a two-dimensional example and checks that the signed count is $1$.1645 1646Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the1647transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The1648exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set $\Delta$ of1649\cref{ch:decision}; crossing it creates or destroys a pair of equilibria of opposite sign, as in the1650birth--death family of \DT{Thm.~8.12}, and it is there that the output of an equilibrium layer can jump.1651 1652\section*{Notes}1653 1654The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and1655Mazur, Palais). Invertible residual networks are from \cite{BGC19}. The observation that the topology of1656the support obstructs normalizing flows, with a quantitative form, is in \cite{CCDD20}. Deep equilibrium1657models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone1658operator parametrizations) are a separate line of work, and \cref{thm:deq} describes what happens without1659them.1660 1661\section*{Exercises}1662 1663\begin{exercise}\label{exer:banach}1664Prove \cref{prop:iresnet}(a).1665\end{exercise}1666 1667\begin{exercise}\label{exer:det}1668Let $g$ be smooth with $\|Dg\|\le L<1$ everywhere. Show $\det(1+Dg)\ge(1-L)^d$.1669\end{exercise}1670 1671\begin{exercise}\label{exer:deq1}1672For $k=1$ and $u=0$, find the fixed points of $z=\tanh(az)$ as a function of $a$ and verify1673\cref{thm:deq}(b). What happens at $a=1$?1674\end{exercise}1675 1676\begin{exercise}\label{exer:supp}1677Prove \cref{prop:mismatch}.1678\end{exercise}1679%% ---- end ch/08-degree.tex ----1680\part{Assessment}