Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1501–1600 of 2141. Source text is displayed literally.
1501than one is a diffeomorphism isotopic to the identity (\cref{prop:iresnet}), and a composite of such blocks1502is a discretized isotopy. Depth, for such architectures, is time.1503 1504\section{The space of representations}1505 1506\begin{problem}\label{prob:calculus}\Spg1507For a compact data manifold $K$ of dimension $m$ and width $k\ge m+3$, the space $\Emb(K,\R^k)$ of1508representations that preserve the topology of the data is described by manifold calculus1509\DT{Thm.~10.10}: it is the limit of a tower whose first stage is the space of immersions and whose higher1510stages are built from configuration spaces of $K$. Determine which components and which homotopy classes of1511this space are reached by the realization map of a given architecture, as a function of width and depth.1512\end{problem}1513 1514For $k\ge2m+2$ the space is connected and the question is about higher homotopy; in the range1515$m+3\le k\le2m+1$ already the set of components is interesting, and \cref{cor:nodeobst}(c) is the first1516example outside that range.1517 1518\section*{Notes}1519 1520The picture of a network as progressively untangling data manifolds goes back at least to expository1521writing of Olah in 2014, where \cref{thm:obstruction} for $d=2$ and the role of an extra dimension are1522described. Minimal width for universal approximation is treated in \cite{HS17,Joh19}. Neural ODEs are from1523\cite{CRBD18}, their limitations and augmentation from \cite{DDT19}, and approximation properties from1524\cite{ZGUA20}.1525 1526\section*{Exercises}1527 1528\begin{exercise}\label{exer:monotone}1529Show that the time-one map of a bounded smooth vector field on $\R$ is increasing.1530\end{exercise}1531 1532\begin{exercise}\label{exer:swap}1533Write down a compactly supported vector field on $\R^2$ whose time-one map exchanges the points $(\pm1,0)$.1534\end{exercise}1535 1536\begin{exercise}\label{exer:link}1537Show that the linking number of two disjoint closed curves in $\R^3$ is unchanged when both are moved by a1538diffeomorphism isotopic to the identity.1539\end{exercise}1540 1541\begin{exercise}\label{exer:proj}1542Let $K$ be the curve $t\mapsto(\cos t,\sin t,\cos2t,\sin2t,\cos3t)$ in $\R^5$. Show that the projection to1543the first two coordinates is an embedding of $K$, and that the projection to the third and fourth is not.1544\end{exercise}1545%% ---- end ch/07-isotopy.tex ----1546%% ---- begin ch/08-degree.tex ----1547\chapter{Degree and Invertibility}\label{ch:degree}1548 1549Some architectures are designed so that every member is invertible: normalizing flows, invertible residual1550networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both1551cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from1552\DT{Ch.~7}.1553 1554\section{Global invertibility}1555 1556\begin{theorem}[Hadamard]\label{thm:hadamard}\Scl1557A local diffeomorphism $f\colon\R^d\to\R^d$ is a diffeomorphism if and only if it is proper.1558\end{theorem}1559 1560\begin{proof}1561A proper local diffeomorphism is a covering map, and $\R^d$ is simply connected. In terms of degree: every1562point is a regular value, the sign of $\det Df$ is constant, and so the number of preimages of a point is1563the absolute value of the degree, which is constant; it is $1$ because a covering of a simply connected1564space is trivial.1565\end{proof}1566 1567A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the1568only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded1569one.1570 1571\begin{proposition}[Invertible residual blocks]\label{prop:iresnet}1572\Scl[Banach; Behrmann et al.\ \cite{BGC19}]1573Let $g\colon\R^d\to\R^d$ have Lipschitz constant $L<1$ and let $f=\id+g$.1574\begin{enumerate}[label=(\alph*)]1575\item $f$ is a bijection, its inverse is Lipschitz with constant $(1-L)^{-1}$, and1576$x_{j+1}=y-g(x_j)$ converges to $f^{-1}(y)$ with $|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|$.1577\item If $g$ is smooth then $f$ is a diffeomorphism with $\det Df>0$.1578\item $f_s=\id+sg$, $0\le s\le1$, is a diffeotopy from the identity to $f$. In particular $f$ is proper of1579degree $1$.1580\end{enumerate}1581\end{proposition}1582 1583\begin{proof}1584(a) $x\mapsto y-g(x)$ is a contraction; and $|f(x)-f(x')|\ge(1-L)|x-x'|$. (b), (c): the eigenvalues of $Dg$1585have modulus at most $L$, so $1+sDg$ is invertible for $s\in[0,1]$, and its determinant is continuous,1586non-zero, and equal to $1$ at $s=0$.1587\end{proof}1588 1589Coupling layers and autoregressive layers are also isotopic to the identity, by scaling their conditioning1590networks to zero. So, apart from fixed permutations and reflections, the invertible architectures in common1591use all realize diffeomorphisms in the identity component of $\Diff(\R^d)$, and by \DT{Thm.~10.2} every1592such diffeomorphism with compact support is conversely the time-one map of a flow.1593 1594\section{Topological mismatch}1595 1596A normalizing flow models a probability measure $\nu$ on $\R^d$ as the image $h_*\mu$ of a simple measure1597$\mu$, usually a Gaussian, under a diffeomorphism $h$.1598 1599\begin{proposition}\label{prop:mismatch}\Scl[cf.\ Cornish et al.\ \cite{CCDD20}]1600For a homeomorphism $h$ of $\R^d$ and a Borel probability measure $\mu$,