Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1541–1640 of 2141. Source text is displayed literally.
1541\begin{exercise}\label{exer:proj}1542Let $K$ be the curve $t\mapsto(\cos t,\sin t,\cos2t,\sin2t,\cos3t)$ in $\R^5$. Show that the projection to1543the first two coordinates is an embedding of $K$, and that the projection to the third and fourth is not.1544\end{exercise}1545%% ---- end ch/07-isotopy.tex ----1546%% ---- begin ch/08-degree.tex ----1547\chapter{Degree and Invertibility}\label{ch:degree}1548 1549Some architectures are designed so that every member is invertible: normalizing flows, invertible residual1550networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both1551cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from1552\DT{Ch.~7}.1553 1554\section{Global invertibility}1555 1556\begin{theorem}[Hadamard]\label{thm:hadamard}\Scl1557A local diffeomorphism $f\colon\R^d\to\R^d$ is a diffeomorphism if and only if it is proper.1558\end{theorem}1559 1560\begin{proof}1561A proper local diffeomorphism is a covering map, and $\R^d$ is simply connected. In terms of degree: every1562point is a regular value, the sign of $\det Df$ is constant, and so the number of preimages of a point is1563the absolute value of the degree, which is constant; it is $1$ because a covering of a simply connected1564space is trivial.1565\end{proof}1566 1567A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the1568only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded1569one.1570 1571\begin{proposition}[Invertible residual blocks]\label{prop:iresnet}1572\Scl[Banach; Behrmann et al.\ \cite{BGC19}]1573Let $g\colon\R^d\to\R^d$ have Lipschitz constant $L<1$ and let $f=\id+g$.1574\begin{enumerate}[label=(\alph*)]1575\item $f$ is a bijection, its inverse is Lipschitz with constant $(1-L)^{-1}$, and1576$x_{j+1}=y-g(x_j)$ converges to $f^{-1}(y)$ with $|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|$.1577\item If $g$ is smooth then $f$ is a diffeomorphism with $\det Df>0$.1578\item $f_s=\id+sg$, $0\le s\le1$, is a diffeotopy from the identity to $f$. In particular $f$ is proper of1579degree $1$.1580\end{enumerate}1581\end{proposition}1582 1583\begin{proof}1584(a) $x\mapsto y-g(x)$ is a contraction; and $|f(x)-f(x')|\ge(1-L)|x-x'|$. (b), (c): the eigenvalues of $Dg$1585have modulus at most $L$, so $1+sDg$ is invertible for $s\in[0,1]$, and its determinant is continuous,1586non-zero, and equal to $1$ at $s=0$.1587\end{proof}1588 1589Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning1590networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common1591use all realize diffeomorphisms in the identity component of $\Diff(\R^d)$, and by \DT{Thm.~10.2} every1592such diffeomorphism with compact support is conversely the time-one map of a flow.1593 1594\section{Topological mismatch}1595 1596A normalizing flow models a probability measure $\nu$ on $\R^d$ as the image $h_*\mu$ of a simple measure1597$\mu$, usually a Gaussian, under a diffeomorphism $h$.1598 1599\begin{proposition}\label{prop:mismatch}\Scl[cf.\ Cornish et al.\ \cite{CCDD20}]1600For a homeomorphism $h$ of $\R^d$ and a Borel probability measure $\mu$,1601$\operatorname{supp}(h_*\mu)=h(\operatorname{supp}\mu)$. Hence if the supports of $\mu$ and $\nu$ are not1602homeomorphic, no homeomorphism satisfies $h_*\mu=\nu$.1603\end{proposition}1604 1605A Gaussian has support $\R^d$. A target supported on two disjoint balls, on a sphere, or on any proper1606closed subset cannot be represented exactly, and Cornish et al.\ show that a sequence of flows approximating1607such a target must have bi-Lipschitz constants tending to infinity. The remedies proposed in the literature1608are the ones differential topology suggests: a target supported on a submanifold $M^m\subset\R^d$ is modelled1609by a flow on $\R^m$ followed by an embedding $\R^m\to\R^d$, and when $M$ is not diffeomorphic to an open1610subset of $\R^m$, by several such charts with a discrete index, which is an atlas in the sense of1611\DT{Ch.~1}. The minimal number of charts is an invariant of $M$; for a sphere it is two.1612 1613\section{Equilibrium layers}1614 1615A \term{deep equilibrium layer} outputs a solution $z$ of $z=g_\theta(z,x)$ for a network $g_\theta$. The1616existence and number of solutions is a fixed-point problem, and \DT{Prop.~7.11} describes its three aspects:1617a generic signed count, a derived fibre product, and a trace.1618 1619\begin{theorem}[Equilibria of a bounded layer]\label{thm:deq}\Srf[\DT{Thm.~4.5, Thm.~7.2, Thm.~7.9}]1620Let $g(z)=\tanh(Az+u)$ on $\R^k$, with $A\in\R^{k\times k}$ and $u\in\R^k$.1621\begin{enumerate}[label=(\alph*)]1622\item For every $A$ and almost every $u$, every fixed point of $g$ is non-degenerate: $1$ is not an1623eigenvalue of $Dg$ there. The fixed points are then finite in number.1624\item In that case $\sum_{g(z)=z}\operatorname{sign}\det(1-Dg(z))=1$. In particular the number of fixed1625points is odd, and at least one exists.1626\item Each non-degenerate fixed point is locally a smooth function of $(A,u)$, with1627$dz=(1-D_zg)^{-1}\,d_{(A,u)}g$.1628\end{enumerate}1629\end{theorem}1630 1631\begin{proof}1632(a) The map $\Psi(u,z)=z-\tanh(Az+u)$ has $D_u\Psi=-\operatorname{diag}(\operatorname{sech}^2(Az+u))$,1633which is invertible; so $\Psi$ is a submersion and by \DT{Thm.~4.5} zero is a regular value of1634$\Psi(u,\cdot)$ for almost every $u$. Regularity at a zero $z$ means that $1-Dg(z)$ is invertible. All fixed1635points lie in the open cube $(-1,1)^k$; being isolated in a compact set they are finite. (b) For1636$0\le s\le1$ the map $g_s(z)=\tanh(s(Az+u))$ sends the closed cube $Q=[-1,1]^k$ into its interior, so1637$\id-g_s$ has no zero on $\partial Q$, and the degree at $0$ of $\id-g_s\colon(Q,\partial Q)\to1638(\R^k,\R^k\setminus0)$ is independent of $s$ by \DT{Thm.~7.2}. At $s=0$ it is the degree of the identity.1639At $s=1$, zero being a regular value, it is the stated sum. (c) is the implicit function theorem.1640\end{proof}