Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 2081–2141 of 2141. Source text is displayed literally.

2081absolute value of the determinant is the product of the singular values, so it is at least $(1-L)^d$; and2082the determinant is positive by \cref{prop:iresnet}(b).2083 2084\sol{8.3} $z=0$ is always a fixed point, with $1-g'(0)=1-a$. For $a\le1$ it is the only one: $\tanh(az)$ is2085strictly concave for $z>0$ with slope $a\le1$ at $0$, so $\tanh(az)<z$ for $z>0$, and symmetrically for2086$z<0$. For $a>1$, $\tanh(az)>z$ for small $z>0$ and $\tanh(az)<1<z$ for $z>1$, so there is a fixed point2087$z^*\in(0,1)$, unique by concavity, and $-z^*$ by symmetry. At $z^*$ the graph of $\tanh(az)$ crosses the2088diagonal from above, so $g'(z^*)<1$ and the sign is $+$; at $0$ the sign of $1-a$ is $-$. The signed count is2089$1-1+1=1$. For $a<1$ it is the single $+1$. At $a=1$ the fixed point $0$ is degenerate ($1-g'(0)=0$): this2090is the exceptional parameter at which the pair $\pm z^*$ is born, a pitchfork, which is not a generic2091birth--death because of the symmetry $z\mapsto-z$ present when $u=0$.2092 2093\sol{8.4} A point $y$ lies in $\operatorname{supp}(h_*\mu)$ if and only if every open neighbourhood $V$ of $y$2094has $\mu(h^{-1}(V))>0$. As $h$ is a homeomorphism, the sets $h^{-1}(V)$ are exactly the open neighbourhoods of2095$h^{-1}(y)$; so the condition is $h^{-1}(y)\in\operatorname{supp}\mu$. If $h_*\mu=\nu$ then2096$h$ restricts to a homeomorphism from $\operatorname{supp}\mu$ to $\operatorname{supp}\nu$.2097%% ---- end sol/08.tex ----2098\backmatter2099%% ---- begin bib.tex ----2100\chapter{References}2101\begingroup\small2102\renewcommand{\chapter}[2]{}2103\begin{thebibliography}{CGGWZ22}\setlength{\itemsep}{1pt}2104\newcommand{\bi}[1]{\bibitem[#1]{#1}}2105\bi{BGC19} J.~Behrmann, W.~Grathwohl, R.~T.~Q.~Chen, D.~Duvenaud, J.-H.~Jacobsen, Invertible residual networks, in \emph{ICML 2019}, PMLR 97, 573--582.2106\bi{BS14} M.~Bianchini, F.~Scarselli, On the complexity of neural network classifiers: a comparison between shallow and deep architectures, \emph{IEEE Trans.\ Neural Netw.\ Learn.\ Syst.} 25 (2014), 1553--1565.2107\bi{CCDD20} R.~Cornish, A.~Caterini, G.~Deligiannidis, A.~Doucet, Relaxing bijectivity constraints with continuously indexed normalising flows, in \emph{ICML 2020}, PMLR 119.2108\bi{CCG20} R.~Cockett, G.~Cruttwell, J.~Gallagher, J.-S.~P.~Lemay, B.~MacAdam, G.~Plotkin, D.~Pronk, Reverse derivative categories, in \emph{Computer Science Logic 2020}, LIPIcs 152, 18:1--18:16.2109\bi{CGGWZ22} G.~S.~H.~Cruttwell, B.~Gavranovi\'c, N.~Ghani, P.~Wilson, F.~Zanasi, Categorical foundations of gradient-based learning, in \emph{ESOP 2022}, Lecture Notes in Computer Science 13240, Springer, 1--28.2110\bi{Coo21} Y.~Cooper, Global minima of overparameterized neural networks, \emph{SIAM J.~Math.\ Data Sci.} 3 (2021), 676--691.2111\bi{CRBD18} R.~T.~Q.~Chen, Y.~Rubanova, J.~Bettencourt, D.~Duvenaud, Neural ordinary differential equations, in \emph{NeurIPS 2018}.2112\bi{DDT19} E.~Dupont, A.~Doucet, Y.~W.~Teh, Augmented neural ODEs, in \emph{NeurIPS 2019}.2113\bi{DTHC} Anonymous, \emph{Differential Topology in Higher Categories}, Higher Categorical Semantics, 2026. Companion volume.2114\bi{DVSH18} F.~Draxler, K.~Veschgini, M.~Salmhofer, F.~Hamprecht, Essentially no barriers in neural network energy landscape, in \emph{ICML 2018}, PMLR 80.2115\bi{EKSN22} R.~Entezari, H.~Sedghi, O.~Saukh, B.~Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks, in \emph{ICLR 2022}.2116\bi{FA00} K.~Fukumizu, S.~Amari, Local minima and plateaus in hierarchical structures of multilayer perceptrons, \emph{Neural Networks} 13 (2000), 317--327.2117\bi{Fef94} C.~Fefferman, Reconstructing a neural net from its output, \emph{Rev.\ Mat.\ Iberoamericana} 10 (1994), 507--555.2118\bi{FMN16} C.~Fefferman, S.~Mitter, H.~Narayanan, Testing the manifold hypothesis, \emph{J.~Amer.\ Math.\ Soc.} 29 (2016), 983--1049.2119\bi{FST19} B.~Fong, D.~I.~Spivak, R.~Tuy\'eras, Backprop as functor: a compositional perspective on supervised learning, in \emph{LICS 2019}, IEEE, 2019.2120\bi{Fuk96} K.~Fukumizu, A regularity condition of the information matrix of a multilayer perceptron network, \emph{Neural Networks} 9 (1996), 871--879.2121\bi{GIP18} T.~Garipov, P.~Izmailov, D.~Podoprikhin, D.~Vetrov, A.~G.~Wilson, Loss surfaces, mode connectivity, and fast ensembling of DNNs, in \emph{NeurIPS 2018}.2122\bi{GL22} J.~E.~Grigsby, K.~Lindsey, On transversality of bent hyperplane arrangements and the topological expressiveness of ReLU neural networks, \emph{SIAM J.~Appl.\ Algebra Geom.} 6 (2022), 216--242.2123\bi{GLMW22} J.~E.~Grigsby, K.~Lindsey, R.~Meyerhoff, C.~Wu, Functional dimension of feedforward ReLU neural networks, preprint, arXiv:2209.04036.2124\bi{Hir76} M.~W.~Hirsch, \emph{Differential Topology}, Graduate Texts in Mathematics 33, Springer, 1976.2125\bi{Hor91} K.~Hornik, Approximation capabilities of multilayer feedforward networks, \emph{Neural Networks} 4 (1991), 251--257.2126\bi{HS17} B.~Hanin, M.~Sellke, Approximating continuous functions by ReLU nets of minimal width, preprint, arXiv:1710.11278.2127\bi{JGH18} A.~Jacot, F.~Gabriel, C.~Hongler, Neural tangent kernel: convergence and generalization in neural networks, in \emph{NeurIPS 2018}.2128\bi{Joh19} J.~Johnson, Deep, skinny neural networks are not universal approximators, in \emph{ICLR 2019}.2129\bi{LZB22} C.~Liu, L.~Zhu, M.~Belkin, Loss landscapes and optimization in over-parameterized non-linear systems and neural networks, \emph{Appl.\ Comput.\ Harmon.\ Anal.} 59 (2022), 85--116.2130\bi{Ngu19} Q.~Nguyen, On connected sublevel sets in deep learning, in \emph{ICML 2019}, PMLR 97.2131\bi{NZL20} G.~Naitzat, A.~Zhitnikov, L.-H.~Lim, Topology of deep neural networks, \emph{J.~Mach.\ Learn.\ Res.} 21 (2020), no.~184, 1--40.2132\bi{PL20} M.~Phuong, C.~H.~Lampert, Functional vs.\ parametric equivalence of ReLU networks, in \emph{ICLR 2020}.2133\bi{SGJ21} B.~\c{S}im\c{s}ek, F.~Ged, A.~Jacot, F.~Spadaro, C.~Hongler, W.~Gerstner, J.~Brea, Geometry of the loss landscape in overparameterized neural networks: symmetries and invariances, in \emph{ICML 2021}, PMLR 139.2134\bi{Sus92} H.~J.~Sussmann, Uniqueness of the weights for minimal feedforward nets with a given input-output map, \emph{Neural Networks} 5 (1992), 589--593.2135\bi{Wat09} S.~Watanabe, \emph{Algebraic Geometry and Statistical Learning Theory}, Cambridge University Press, 2009.2136\bi{ZGUA20} H.~Zhang, X.~Gao, J.~Unterman, T.~Arodz, Approximation capabilities of neural ODEs and invertible residual networks, in \emph{ICML 2020}, PMLR 119.2137\end{thebibliography}2138\endgroup2139%% ---- end bib.tex ----2140\end{document}2141