Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 2021–2120 of 2141. Source text is displayed literally.
2021$M_y=\theta^*+\ker\Phi$ is an affine subspace of dimension $P-n$, where $\theta^*$ is any solution, for2022instance the minimum-norm one $\Phi^{\mathsf T}(\Phi\Phi^{\mathsf T})^{-1}y$. The Hessian is the constant2023matrix $\Phi^{\mathsf T}\Phi$, of rank $n$, with kernel $\ker\Phi$.2024 2025\sol{6.2} $\nabla\loss=(ab-1)(b,a)$, which vanishes when $ab=1$ or $(a,b)=0$. The Hessian is2026$\bigl(\begin{smallmatrix}b^2&2ab-1\\2ab-1&a^2\end{smallmatrix}\bigr)$. On $ab=1$ it is2027$\bigl(\begin{smallmatrix}b^2&1\\1&a^2\end{smallmatrix}\bigr)$, of determinant $a^2b^2-1=0$ and trace2028$a^2+b^2>0$: rank $1$, with kernel spanned by $(a,-b)$, the tangent to the hyperbola. At the origin it is2029$\bigl(\begin{smallmatrix}0&-1\\-1&0\end{smallmatrix}\bigr)$, with eigenvalues $\pm1$: a saddle of index2030$1$ with value $\frac12$. The sublevel set is $\{1-\sqrt{2c}\le ab\le1+\sqrt{2c}\}$. For $c<\frac12$ the2031lower bound is positive, and the set lies in the two quadrants $ab>0$, one component in each. For2032$c\ge\frac12$ it contains the axes and is connected. Any path from $(1,1)$ to $(-1,-1)$ must cross the set2033$ab=0$, where $\loss=\frac12$.2034 2035\sol{6.3} The flow line $\gamma$ has length at most $\ell=\sqrt{2\loss(\theta_0)/\mu}$ while it stays in $U$.2036If the closed ball of radius $\ell$ about $\theta_0$ lies in $U$, the flow line cannot leave $U$: at the2037first exit time it would be at distance at most its length, which is at most $\ell$, from $\theta_0$,2038hence still inside. So the flow line is defined for all time, has finite length, and therefore converges; the2039limit $\theta_\infty$ satisfies $|\theta_\infty-\theta_0|\le\ell$ and $\loss(\theta_\infty)=\lim2040\loss(\gamma(t))=0$.2041 2042\sol{6.4} $\loss=\frac12(\prod a_i-1)^2$ and $M_y=\{\prod a_i=1\}$, the preimage of the regular value $1$ of2043the product map (its gradient at a point with all $a_i\ne0$ is non-zero); so $\dim M_y=k-1$. On $M_y$ every2044$a_i\ne0$ and the number of negative $a_i$ is even; each admissible sign pattern gives one component (a2045graph over the positive orthant in the first $k-1$ coordinates), and there are $2^{k-1}$ sign patterns with2046an even number of minus signs. The gradient is $(\prod a_i-1)\bigl(\prod_{j\ne i}a_j\bigr)_i$, which vanishes2047at the origin for $k\ge2$; the Hessian at the origin has entries2048$-\partial_i\partial_j\prod a_l$, which are zero for $k\ge3$. So the origin is a critical point with zero2049Hessian.2050%% ---- end sol/06.tex ----2051%% ---- begin sol/07.tex ----2052\section*{Chapter 7}2053 2054\sol{7.1} Let $\varphi^t$ be the flow. If $x<y$ then $\varphi^t(x)\ne\varphi^t(y)$ for all $t$ by uniqueness2055of solutions, so $\varphi^t(y)-\varphi^t(x)$ is continuous in $t$, never zero, and positive at $t=0$; hence2056positive at $t=1$.2057 2058\sol{7.2} Let $\rho\colon[0,\infty)\to[0,1]$ be smooth with $\rho=1$ on $[0,2]$ and $\rho=0$ on $[3,\infty)$,2059and let $v(x)=\pi\rho(|x|)\,(-x_2,x_1)$. The flow preserves $|x|$, since $v$ is tangent to circles, and on2060$\{|x|\le2\}$ it is rotation with angular speed $\pi$. At time one the points $(\pm1,0)$ have been rotated2061by the angle $\pi$ and are exchanged.2062 2063\sol{7.3} Let $\varphi_t$ be the diffeotopy and $K_t=\varphi_t(K)$, $L_t=\varphi_t(L)$; they are disjoint for2064every $t$ because $\varphi_t$ is injective. The Gauss maps2065$G_t\colon K\times L\to S^2$, $G_t(x,y)=(\varphi_t(y)-\varphi_t(x))/|\varphi_t(y)-\varphi_t(x)|$, form a2066homotopy, and the degree is a homotopy invariant by \DT{Thm.~7.2}.2067 2068\sol{7.4} The first projection is $t\mapsto(\cos t,\sin t)$, the standard embedding of the circle. The second2069is $t\mapsto(\cos2t,\sin2t)$, which takes the same value at $t$ and $t+\pi$; it is an immersion but not2070injective, a double cover of the circle.2071%% ---- end sol/07.tex ----2072%% ---- begin sol/08.tex ----2073\section*{Chapter 8}2074 2075\sol{8.1} For fixed $y$ the map $T(x)=y-g(x)$ satisfies $|T(x)-T(x')|\le L|x-x'|$, so it has a unique fixed2076point $x^*$, which is the unique solution of $x+g(x)=y$, and $|x_j-x^*|=|T^j(x_0)-T^j(x^*)|\le L^j|x_0-x^*|$.2077So $f$ is a bijection. And $|f(x)-f(x')|\ge|x-x'|-|g(x)-g(x')|\ge(1-L)|x-x'|$, which says that $f^{-1}$ is2078Lipschitz with constant $(1-L)^{-1}$.2079 2080\sol{8.2} The singular values of $1+Dg$ are at least $1-L$, since $|(1+Dg)v|\ge|v|-|Dg\,v|\ge(1-L)|v|$. The2081absolute value of the determinant is the product of the singular values, so it is at least $(1-L)^d$; and2082the determinant is positive by \cref{prop:iresnet}(b).2083 2084\sol{8.3} $z=0$ is always a fixed point, with $1-g'(0)=1-a$. For $a\le1$ it is the only one: $\tanh(az)$ is2085strictly concave for $z>0$ with slope $a\le1$ at $0$, so $\tanh(az)<z$ for $z>0$, and symmetrically for2086$z<0$. For $a>1$, $\tanh(az)>z$ for small $z>0$ and $\tanh(az)<1<z$ for $z>1$, so there is a fixed point2087$z^*\in(0,1)$, unique by concavity, and $-z^*$ by symmetry. At $z^*$ the graph of $\tanh(az)$ crosses the2088diagonal from above, so $g'(z^*)<1$ and the sign is $+$; at $0$ the sign of $1-a$ is $-$. The signed count is2089$1-1+1=1$. For $a<1$ it is the single $+1$. At $a=1$ the fixed point $0$ is degenerate ($1-g'(0)=0$): this2090is the exceptional parameter at which the pair $\pm z^*$ is born, a pitchfork, which is not a generic2091birth--death because of the symmetry $z\mapsto-z$ present when $u=0$.2092 2093\sol{8.4} A point $y$ lies in $\operatorname{supp}(h_*\mu)$ if and only if every open neighbourhood $V$ of $y$2094has $\mu(h^{-1}(V))>0$. As $h$ is a homeomorphism, the sets $h^{-1}(V)$ are exactly the open neighbourhoods of2095$h^{-1}(y)$; so the condition is $h^{-1}(y)\in\operatorname{supp}\mu$. If $h_*\mu=\nu$ then2096$h$ restricts to a homeomorphism from $\operatorname{supp}\mu$ to $\operatorname{supp}\nu$.2097%% ---- end sol/08.tex ----2098\backmatter2099%% ---- begin bib.tex ----2100\chapter{References}2101\begingroup\small2102\renewcommand{\chapter}[2]{}2103\begin{thebibliography}{CGGWZ22}\setlength{\itemsep}{1pt}2104\newcommand{\bi}[1]{\bibitem[#1]{#1}}2105\bi{BGC19} J.~Behrmann, W.~Grathwohl, R.~T.~Q.~Chen, D.~Duvenaud, J.-H.~Jacobsen, Invertible residual networks, in \emph{ICML 2019}, PMLR 97, 573--582.2106\bi{BS14} M.~Bianchini, F.~Scarselli, On the complexity of neural network classifiers: a comparison between shallow and deep architectures, \emph{IEEE Trans.\ Neural Netw.\ Learn.\ Syst.} 25 (2014), 1553--1565.2107\bi{CCDD20} R.~Cornish, A.~Caterini, G.~Deligiannidis, A.~Doucet, Relaxing bijectivity constraints with continuously indexed normalising flows, in \emph{ICML 2020}, PMLR 119.2108\bi{CCG20} R.~Cockett, G.~Cruttwell, J.~Gallagher, J.-S.~P.~Lemay, B.~MacAdam, G.~Plotkin, D.~Pronk, Reverse derivative categories, in \emph{Computer Science Logic 2020}, LIPIcs 152, 18:1--18:16.2109\bi{CGGWZ22} G.~S.~H.~Cruttwell, B.~Gavranovi\'c, N.~Ghani, P.~Wilson, F.~Zanasi, Categorical foundations of gradient-based learning, in \emph{ESOP 2022}, Lecture Notes in Computer Science 13240, Springer, 1--28.2110\bi{Coo21} Y.~Cooper, Global minima of overparameterized neural networks, \emph{SIAM J.~Math.\ Data Sci.} 3 (2021), 676--691.2111\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, in \emph{NeurIPS 2018}.2112\bi{DDT19} E.~Dupont, A.~Doucet, Y.~W.~Teh, Augmented neural ODEs, in \emph{NeurIPS 2019}.2113\bi{DTHC} Anonymous, \emph{Differential Topology in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.2114\bi{DVSH18} F.~Draxler, K.~Veschgini, M.~Salmhofer, F.~Hamprecht, Essentially no barriers in neural network energy landscape, in \emph{ICML 2018}, PMLR 80.2115\bi{EKSN22} R.~Entezari, H.~Sedghi, O.~Saukh, B.~Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, in \emph{ICLR 2022}.2116\bi{FA00} K.~Fukumizu, S.~Amari, Local minima and plateaus in hierarchical structures of multilayer perceptrons, \emph{Neural Networks} 13 (2000), 317--327.2117\bi{Fef94} C.~Fefferman, Reconstructing a neural net from its output, \emph{Rev.\ Mat.\ Iberoamericana} 10 (1994), 507--555.2118\bi{FMN16} C.~Fefferman, S.~Mitter, H.~Narayanan, Testing the manifold hypothesis, \emph{J.~Amer.\ Math.\ Soc.} 29 (2016), 983--1049.2119\bi{FST19} B.~Fong, D.~I.~Spivak, R.~Tuy\'eras, Backprop as functor: a compositional perspective on supervised learning, in \emph{LICS 2019}, IEEE, 2019.2120\bi{Fuk96} K.~Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network, \emph{Neural Networks} 9 (1996), 871--879.