Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1561–1660 of 2141. Source text is displayed literally.

1561A proper local diffeomorphism is a covering map, and $\R^d$ is simply connected. In terms of degree: every1562point is a regular value, the sign of $\det Df$ is constant, and so the number of preimages of a point is1563the absolute value of the degree, which is constant; it is $1$ because a covering of a simply connected1564space is trivial.1565\end{proof}1566 1567A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the1568only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded1569one.1570 1571\begin{proposition}[Invertible residual blocks]\label{prop:iresnet}1572\Scl[Banach; Behrmann et al.\ \cite{BGC19}]1573Let $g\colon\R^d\to\R^d$ have Lipschitz constant $L<1$ and let $f=\id+g$.1574\begin{enumerate}[label=(\alph*)]1575\item $f$ is a bijection, its inverse is Lipschitz with constant $(1-L)^{-1}$, and1576$x_{j+1}=y-g(x_j)$ converges to $f^{-1}(y)$ with $|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|$.1577\item If $g$ is smooth then $f$ is a diffeomorphism with $\det Df>0$.1578\item $f_s=\id+sg$, $0\le s\le1$, is a diffeotopy from the identity to $f$. In particular $f$ is proper of1579degree $1$.1580\end{enumerate}1581\end{proposition}1582 1583\begin{proof}1584(a) $x\mapsto y-g(x)$ is a contraction; and $|f(x)-f(x')|\ge(1-L)|x-x'|$. (b), (c): the eigenvalues of $Dg$1585have modulus at most $L$, so $1+sDg$ is invertible for $s\in[0,1]$, and its determinant is continuous,1586non-zero, and equal to $1$ at $s=0$.1587\end{proof}1588 1589Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning1590networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common1591use all realize diffeomorphisms in the identity component of $\Diff(\R^d)$, and by \DT{Thm.~10.2} every1592such diffeomorphism with compact support is conversely the time-one map of a flow.1593 1594\section{Topological mismatch}1595 1596A normalizing flow models a probability measure $\nu$ on $\R^d$ as the image $h_*\mu$ of a simple measure1597$\mu$, usually a Gaussian, under a diffeomorphism $h$.1598 1599\begin{proposition}\label{prop:mismatch}\Scl[cf.\ Cornish et al.\ \cite{CCDD20}]1600For a homeomorphism $h$ of $\R^d$ and a Borel probability measure $\mu$,1601$\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu)$. Hence if the supports of $\mu$ and $\nu$ are not1602homeomorphic, no homeomorphism satisfies $h_*\mu=\nu$.1603\end{proposition}1604 1605A Gaussian has support $\R^d$. A target supported on two disjoint balls, on a sphere, or on any proper1606closed subset cannot be represented exactly, and Cornish et al.\ show that a sequence of flows approximating1607such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature1608are the ones differential topology suggests: a target supported on a submanifold $M^m\subset\R^d$ is modelled1609by a flow on $\R^m$ followed by an embedding $\R^m\to\R^d$, and when $M$ is not diffeomorphic to an open1610subset of $\R^m$, by several such charts with a discrete index, which is an atlas in the sense of1611\DT{Ch.~1}. The minimal number of charts is an invariant of $M$; for a sphere it is two.1612 1613\section{Equilibrium layers}1614 1615A \term{deep equilibrium layer} outputs a solution $z$ of $z=g_\theta(z,x)$ for a network $g_\theta$. The1616existence and number of solutions is a fixed-point problem, and \DT{Prop.~7.11} describes its three aspects:1617a generic signed count, a derived fibre product, and a trace.1618 1619\begin{theorem}[Equilibria of a bounded layer]\label{thm:deq}\Srf[\DT{Thm.~4.5, Thm.~7.2, Thm.~7.9}]1620Let $g(z)=\tanh(Az+u)$ on $\R^k$, with $A\in\R^{k\times k}$ and $u\in\R^k$.1621\begin{enumerate}[label=(\alph*)]1622\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an1623eigenvalue of $Dg$ there. The fixed points are then finite in number.1624\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed1625points is odd, and at least one exists.1626\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with1627$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.1628\end{enumerate}1629\end{theorem}1630 1631\begin{proof}1632(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,1633which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of1634$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed1635points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For1636$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so1637$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to1638(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.1639At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.1640\end{proof}1641 1642In one dimension, $z=\tanh(az+u)$ has one solution for $a\le1$, and for $a>1$ and small $|u|$ three, with1643signs $+,-,+$: two stable equilibria and an unstable one between them. The script finds the fixed points for1644several $(a,u)$ and for a two-dimensional example and checks that the signed count is $1$.1645 1646Part (c) is the formula used to differentiate through an equilibrium layer. It is valid exactly at the1647transverse fixed points, and (a) says that for almost every bias all fixed points are transverse. The1648exceptional parameters form the discriminant of the fixed-point problem, the analogue of the set $\Delta$ of1649\cref{ch:decision}; crossing it creates or destroys a pair of equilibria of opposite sign, as in the1650birth--death family of \DT{Thm.~8.12}, and it is there that the output of an equilibrium layer can jump.1651 1652\section*{Notes}1653 1654The global inverse function theorem is due to Hadamard and has many refinements (Caccioppoli, Banach and1655Mazur, Palais). Invertible residual networks are from \cite{BGC19}. The observation that the topology of1656the support obstructs normalizing flows, with a quantitative form, is in \cite{CCDD20}. Deep equilibrium1657models were introduced by Bai, Kolter and Koltun; conditions for uniqueness of the equilibrium (monotone1658operator parametrizations) are a separate line of work, and \cref{thm:deq} describes what happens without1659them.1660