Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1941–2040 of 2141. Source text is displayed literally.

1941sum. (ii) If $\sigma$ is odd, $(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1942(iii) If $\sigma(\lambda s)=\lambda\sigma(s)$ for $\lambda>0$, then1943$(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1944 1945\sol{3.2} The columns of $J$ belonging to $(w_j,b_j)$ are1946$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i$ and1947$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i$; if $a_j=0$ these $d+1$ columns vanish, so1948$\rk J\le P-(d+1)$. If $(w_j,b_j)=(w_k,b_k)$, the columns for $a_j$ and $a_k$ are both1949$\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i$, so two columns coincide and $\rk J\le P-1$. (In the second1950case the columns for $(w_j,b_j)$ and $(w_k,b_k)$ are also proportional, with ratio $a_j/a_k$, so the rank1951drops by $d+2$ in all.)1952 1953\sol{3.3} $\theta=(w,b,a,c)$ and $\xi=(w,b,-a,0)$. Where $wx+b>0$, $F=a(wx+b)+c$ and1954$D_\theta F=(ax,\ a,\ wx+b,\ 1)$, so $D_\theta F\cdot\xi=awx+ab-a(wx+b)=0$. Where $wx+b<0$, $F=c$ and1955$D_\theta F=(0,0,0,1)$, again orthogonal to $\xi$. Here $P=4$, $H=1$, so the functional dimension is at most1956$3$; it equals $3$ when $a\ne0$ and $w\ne0$, since the function $x\mapsto a\,\relu(wx+b)+c$ is determined by1957its breakpoint $-b/w$, the slope $aw$ on the active side, and $c$ (and the side, which is discrete).1958 1959\sol{3.4} $(2,3,1)$: one hidden layer of $3$ units, $|G|=2^3\cdot3!=48$. $(2,4,4,1)$: two hidden layers of1960$4$ units, $|G|=(2^4\cdot4!)^2=384^2=147456$.1961%% ---- end sol/03.tex ----1962%% ---- begin sol/04.tex ----1963\section*{Chapter 4}1964 1965\sol{4.1} By \cref{thm:boundary}(c), for almost every $\theta$ the set $B_\theta$ is a compact neat1966$0$-dimensional submanifold of $[-R,R]$: a finite set of interior points at which $f_\theta'\ne0$, with1967$f_\theta(\pm R)\ne0$. At each such point $f_\theta$ changes sign. So the number of zeros is even if1968$f_\theta(-R)$ and $f_\theta(R)$ have the same sign and odd otherwise.1969 1970\sol{4.2} A compact neat $1$-dimensional submanifold of the disc: a finite disjoint union of circles in the1971interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such1972configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints1973by non-crossing arcs.1974 1975\sol{4.3} $d_xF=(2x_1,-2x_2)$ vanishes only at $x=0$, where $F=c$. So $\Sigma=\{(0,0)\}$ and $\Delta=\{0\}$.1976There are two chambers, $c>0$ and $c<0$. In each the boundary is a hyperbola; for $c<0$ its branches open1977along the $x_1$-axis ($x_1^2=x_2^2-c\ge-c$) and for $c>0$ along the $x_2$-axis. The region $\{f<0\}$ is connected for1978$c<0$ (it contains the origin) and has two components for $c>0$. (The plane is not compact; the statement about chambers holds here1979because the family is a product near infinity up to scaling. Restricted to a disc the same description1980holds.)1981 1982\sol{4.4} At a triple point $p$, $f_\theta$ is transverse to $A_{\{1,2,3\}}$, the line $y_1=y_2=y_3$ in $\R^3$,1983of codimension $2$. So the map $x\mapsto(f_1-f_2,f_2-f_3)$ from the plane to $\R^2$ is a local1984diffeomorphism at $p$, and in the coordinates $(u,v)=(f_1-f_2,f_2-f_3)$ the decision regions are1985$\{u>0,\ u+v>0\}$, $\{u<0,\ v>0\}$ and $\{v<0,\ u+v<0\}$: three sectors bounded by the rays1986$\{u=0,v>0\}$, $\{v=0,u>0\}$ and $\{u+v=0,u<0\}$.1987%% ---- end sol/04.tex ----1988%% ---- begin sol/05.tex ----1989\section*{Chapter 5}1990 1991\sol{5.1} $g=(x_1^2-1)^2+x_2^2$ has $\nabla g=(4x_1(x_1^2-1),2x_2)$, vanishing at $(0,0)$ and $(\pm1,0)$. The1992Hessian is $\operatorname{diag}(12x_1^2-4,\,2)$: at $(\pm1,0)$ it is $\operatorname{diag}(8,2)$, index $0$,1993value $0$; at the origin $\operatorname{diag}(-4,2)$, index $1$, value $1$. The sublevel set $\{g\le t\}$ is1994empty for $t<0$, two discs around $(\pm1,0)$ for $0<t<1$ (on the line $x_2=0$ it is1995$\{|x_1^2-1|\le\sqrt t\}$, two intervals when $t<1$), and one disc for $t>1$. Its Euler characteristic is1996$0,2,1$, consistent with attaching two $0$-cells and then one $1$-cell.1997 1998\sol{5.2} Here $B_t$ is a finite set and $W_\gamma$ is a compact $1$-manifold with boundary $B_0\sqcup B_1$.1999A critical point of $\tau$ on a $1$-manifold is a local minimum or maximum of $t$ along a curve: two points2000of $B_t$ appear together or disappear together (surgery with $\lambda=0$ or $1$ on a $0$-manifold: add or2001remove an $S^0$). Between critical times the points move continuously. If $X$ is a circle, $f_t$ changes2002sign at each point of $B_t$ and returns to its original sign after one circuit, so $\#B_t$ is even for every2003regular $t$.2004 2005\sol{5.3} Two. By \cref{prop:chi} with $d=3$, the Euler characteristic of the region changes by2006$\pm(-1)^k$ at each surgery; the solid torus has $\chi=0$ and needs at least one $0$-handle to be non-empty,2007so at least one more handle of odd index is needed. A $0$-handle (a ball appears, $k=0$) followed by a2008$1$-handle attached to it (the ball becomes a solid torus, $k=1$) suffices. The boundary goes from empty to2009a sphere to a torus.2010 2011\sol{5.4} Write $\Phi=\Phi_\gamma$ and let $t(x)$ be defined by $\Phi(t(x),x)=0$ near $x_i$, with2012$t(x_i)=t_i$. Differentiating once, $\Phi_t\,\partial_jt+\partial_j\Phi=0$, so $\partial_jt(x_i)=0$ because2013$d_x\Phi=0$ there. Differentiating again and evaluating at $x_i$, where the terms containing first2014derivatives of $t$ vanish, gives $\Phi_t\,\partial_j\partial_kt+\partial_j\partial_k\Phi=0$. Hence2015$\nabla^2t(x_i)=-\nabla^2_x\Phi(t_i,x_i)/\Phi_t(t_i,x_i)$.2016%% ---- end sol/05.tex ----2017%% ---- begin sol/06.tex ----2018\section*{Chapter 6}2019 2020\sol{6.1} $\ev(\theta)=\Phi\theta$ with $\Phi$ of rank $n$. Every $y$ is a regular value;2021$M_y=\theta^*+\ker\Phi$ is an affine subspace of dimension $P-n$, where $\theta^*$ is any solution, for2022instance the minimum-norm one $\Phi^{\mathsf T}(\Phi\Phi^{\mathsf T})^{-1}y$. The Hessian is the constant2023matrix $\Phi^{\mathsf T}\Phi$, of rank $n$, with kernel $\ker\Phi$.2024 2025\sol{6.2} $\nabla\loss=(ab-1)(b,a)$, which vanishes when $ab=1$ or $(a,b)=0$. The Hessian is2026$\bigl(\begin{smallmatrix}b^2&2ab-1\\2ab-1&a^2\end{smallmatrix}\bigr)$. On $ab=1$ it is2027$\bigl(\begin{smallmatrix}b^2&1\\1&a^2\end{smallmatrix}\bigr)$, of determinant $a^2b^2-1=0$ and trace2028$a^2+b^2>0$: rank $1$, with kernel spanned by $(a,-b)$, the tangent to the hyperbola. At the origin it is2029$\bigl(\begin{smallmatrix}0&-1\\-1&0\end{smallmatrix}\bigr)$, with eigenvalues $\pm1$: a saddle of index2030$1$ with value $\frac12$. The sublevel set is $\{1-\sqrt{2c}\le ab\le1+\sqrt{2c}\}$. For $c<\frac12$ the2031lower bound is positive, and the set lies in the two quadrants $ab>0$, one component in each. For2032$c\ge\frac12$ it contains the axes and is connected. Any path from $(1,1)$ to $(-1,-1)$ must cross the set2033$ab=0$, where $\loss=\frac12$.2034 2035\sol{6.3} The flow line $\gamma$ has length at most $\ell=\sqrt{2\loss(\theta_0)/\mu}$ while it stays in $U$.2036If the closed ball of radius $\ell$ about $\theta_0$ lies in $U$, the flow line cannot leave $U$: at the2037first exit time it would be at distance at most its length, which is at most $\ell$, from $\theta_0$,2038hence still inside. So the flow line is defined for all time, has finite length, and therefore converges; the2039limit $\theta_\infty$ satisfies $|\theta_\infty-\theta_0|\le\ell$ and $\loss(\theta_\infty)=\lim2040\loss(\gamma(t))=0$.