Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1481–1580 of 2141. Source text is displayed literally.
1481\end{corollary}1482 1483\begin{proof}1484Take $e_0$ the inclusion of $K$ in $\R^d\times\{0\}\subset\R^N$ and $e_1=e_0\pm(0,\dots,0,1)$ on $K_\pm$, and1485apply \cref{thm:augment} to get a field $v$ with flow $\varphi$. For the last statement, approximate $v$ in1486$C^1$ on a compact set containing the support, by Hornik's theorem \cite{Hor91}; by Gr\"onwall's inequality1487the time-one maps are then close in $C^1$ on compact sets, and the conclusion, being an open condition on1488the images of the compact sets $K_\pm$, persists.1489\end{proof}1490 1491So the obstructions of \cref{cor:nodeobst} disappear once the state space has dimension at least $2m+2$,1492where $m$ is the dimension of the data and not of the ambient space. For finite data sets $m=0$ and two1493dimensions suffice, in agreement with (a): points on a line cannot be reordered by a flow, points in the1494plane can. For curves the bound is $4$, and it is sharp, since circles link in $\R^3$. For connected data1495of dimension $m\ge2$ the bound can be lowered to $2m+1$ by the second part of \DT{Thm.~10.9}. Zhang et al.\1496\cite{ZGUA20} prove a related statement about maps: every homeomorphism of $\R^d$ can be embedded in a flow on1497$\R^{2d}$, and hence approximated by a neural ODE or an invertible residual network acting there. The dimension count of \cref{thm:augment}1498depends on the data alone.1499 1500Residual networks fit the same description. A block $x\mapsto x+g(x)$ with $g$ of Lipschitz constant less1501than one is a diffeomorphism isotopic to the identity (\cref{prop:iresnet}), and a composite of such blocks1502is a discretized isotopy. Depth, for such architectures, is time.1503 1504\section{The space of representations}1505 1506\begin{problem}\label{prob:calculus}\Spg1507For a compact data manifold $K$ of dimension $m$ and width $k\ge m+3$, the space $\Emb(K,\R^k)$ of1508representations that preserve the topology of the data is described by manifold calculus1509\DT{Thm.~10.10}: it is the limit of a tower whose first stage is the space of immersions and whose higher1510stages are built from configuration spaces of $K$. Determine which components and which homotopy classes of1511this space are reached by the realization map of a given architecture, as a function of width and depth.1512\end{problem}1513 1514For $k\ge2m+2$ the space is connected and the question is about higher homotopy; in the range1515$m+3\le k\le2m+1$ already the set of components is interesting, and \cref{cor:nodeobst}(c) is the first1516example outside that range.1517 1518\section*{Notes}1519 1520The picture of a network as progressively untangling data manifolds goes back at least to expository1521writing of Olah in 2014, where \cref{thm:obstruction} for $d=2$ and the role of an extra dimension are1522described. Minimal width for universal approximation is treated in \cite{HS17,Joh19}. Neural ODEs are from1523\cite{CRBD18}, their limitations and augmentation from \cite{DDT19}, and approximation properties from1524\cite{ZGUA20}.1525 1526\section*{Exercises}1527 1528\begin{exercise}\label{exer:monotone}1529Show that the time-one map of a bounded smooth vector field on $\R$ is increasing.1530\end{exercise}1531 1532\begin{exercise}\label{exer:swap}1533Write down a compactly supported vector field on $\R^2$ whose time-one map exchanges the points $(\pm1,0)$.1534\end{exercise}1535 1536\begin{exercise}\label{exer:link}1537Show that the linking number of two disjoint closed curves in $\R^3$ is unchanged when both are moved by a1538diffeomorphism isotopic to the identity.1539\end{exercise}1540 1541\begin{exercise}\label{exer:proj}1542Let $K$ be the curve $t\mapsto(\cos t,\sin t,\cos2t,\sin2t,\cos3t)$ in $\R^5$. Show that the projection to1543the first two coordinates is an embedding of $K$, and that the projection to the third and fourth is not.1544\end{exercise}1545%% ---- end ch/07-isotopy.tex ----1546%% ---- begin ch/08-degree.tex ----1547\chapter{Degree and Invertibility}\label{ch:degree}1548 1549Some architectures are designed so that every member is invertible: normalizing flows, invertible residual1550networks, the neural ODEs of the last chapter. Others are designed around a fixed-point equation. In both1551cases the relevant invariants are the degree of a map and the signed count of solutions of an equation, from1552\DT{Ch.~7}.1553 1554\section{Global invertibility}1555 1556\begin{theorem}[Hadamard]\label{thm:hadamard}\Scl1557A local diffeomorphism $f\colon\R^d\to\R^d$ is a diffeomorphism if and only if it is proper.1558\end{theorem}1559 1560\begin{proof}1561A proper local diffeomorphism is a covering map, and $\R^d$ is simply connected. In terms of degree: every1562point is a regular value, the sign of $\det Df$ is constant, and so the number of preimages of a point is1563the absolute value of the degree, which is constant; it is $1$ because a covering of a simply connected1564space is trivial.1565\end{proof}1566 1567A network all of whose layer Jacobians are invertible is a local diffeomorphism. The theorem says that the1568only way it can fail to be globally invertible is by sending a sequence tending to infinity to a bounded1569one.1570 1571\begin{proposition}[Invertible residual blocks]\label{prop:iresnet}1572\Scl[Banach; Behrmann et al.\ \cite{BGC19}]1573Let $g\colon\R^d\to\R^d$ have Lipschitz constant $L<1$ and let $f=\id+g$.1574\begin{enumerate}[label=(\alph*)]1575\item $f$ is a bijection, its inverse is Lipschitz with constant $(1-L)^{-1}$, and1576$x_{j+1}=y-g(x_j)$ converges to $f^{-1}(y)$ with $|x_j-f^{-1}(y)|\le L^j|x_0-f^{-1}(y)|$.1577\item If $g$ is smooth then $f$ is a diffeomorphism with $\det Df>0$.1578\item $f_s=\id+sg$, $0\le s\le1$, is a diffeotopy from the identity to $f$. In particular $f$ is proper of1579degree $1$.1580\end{enumerate}