Indexed text attachment
parametrized-neural-networks-and-their-representations-in-differential-topology.tex
Source for Parametrized Neural Networks and Their Representations in Differential Topology
View paper · Download original file
Lines 1921–2020 of 2141. Source text is displayed literally.
1921\sol{2.2} $\ker JJ^{\mathsf T}=\ker J^{\mathsf T}$ since $\langle v,JJ^{\mathsf T}v\rangle=|J^{\mathsf T}v|^2$;1922so $\rk K=\rk J^{\mathsf T}=\rk J$. For scalar output $K$ is $n\times n$, invertible if and only if $\rk J=n$,1923that is, if and only if the columns of $J$ span $\R^n$; the $k$-th column is the vector1924$\bigl(\partial F/\partial\theta_k(\theta,x_i)\bigr)_{i}$.1925 1926\sol{2.3} By \DT{Prop.~2.9} the $2$-jet of $\lambda\circ\ev$ at $\theta$ has linear part $C A$ and quadratic part1927$C\circ B+E\circ(A\times A)$, where $(A,B)$ is the $2$-jet of $\ev$ (so $A=J$ and $B=(\nabla^2\ev_k)_k$) and1928$(C,E)$ that of $\lambda$ (so $C=\nabla\lambda^{\mathsf T}$, $E=\nabla^2\lambda$). The quadratic part is1929$\sum_k\partial_k\lambda\,\nabla^2\ev_k+J^{\mathsf T}\nabla^2\lambda\,J$.1930 1931\sol{2.4} $\ev(\theta)=\Phi\theta$ with $\Phi$ the $n\times P$ matrix of rows $\phi(x_i)^{\mathsf T}$; so1932$J=\Phi$ for all $\theta$ and $K=\Phi\Phi^{\mathsf T}$, the Gram matrix of the features. The fibres of $\ev$1933are the affine subspaces $\theta_0+\ker\Phi$, of dimension $P-\rk\Phi$. The realization map is linear with1934kernel $\{\theta:\langle\theta,\phi(x)\rangle=0\ \forall x\}$, so the functional dimension is the dimension1935of the span of $\phi(X)$.1936%% ---- end sol/02.tex ----1937%% ---- begin sol/03.tex ----1938\section*{Chapter 3}1939 1940\sol{3.1} $F(\theta,x)=\sum_ja_j\sigma(\langle w_j,x\rangle+b_j)+c$. (i) Permuting the index $j$ reorders the1941sum. (ii) If $\sigma$ is odd, $(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1942(iii) If $\sigma(\lambda s)=\lambda\sigma(s)$ for $\lambda>0$, then1943$(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1944 1945\sol{3.2} The columns of $J$ belonging to $(w_j,b_j)$ are1946$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i$ and1947$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i$; if $a_j=0$ these $d+1$ columns vanish, so1948$\rk J\le P-(d+1)$. If $(w_j,b_j)=(w_k,b_k)$, the columns for $a_j$ and $a_k$ are both1949$\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i$, so two columns coincide and $\rk J\le P-1$. (In the second1950case the columns for $(w_j,b_j)$ and $(w_k,b_k)$ are also proportional, with ratio $a_j/a_k$, so the rank1951drops by $d+2$ in all.)1952 1953\sol{3.3} $\theta=(w,b,a,c)$ and $\xi=(w,b,-a,0)$. Where $wx+b>0$, $F=a(wx+b)+c$ and1954$D_\theta F=(ax,\ a,\ wx+b,\ 1)$, so $D_\theta F\cdot\xi=awx+ab-a(wx+b)=0$. Where $wx+b<0$, $F=c$ and1955$D_\theta F=(0,0,0,1)$, again orthogonal to $\xi$. Here $P=4$, $H=1$, so the functional dimension is at most1956$3$; it equals $3$ when $a\ne0$ and $w\ne0$, since the function $x\mapsto a\,\relu(wx+b)+c$ is determined by1957its breakpoint $-b/w$, the slope $aw$ on the active side, and $c$ (and the side, which is discrete).1958 1959\sol{3.4} $(2,3,1)$: one hidden layer of $3$ units, $|G|=2^3\cdot3!=48$. $(2,4,4,1)$: two hidden layers of1960$4$ units, $|G|=(2^4\cdot4!)^2=384^2=147456$.1961%% ---- end sol/03.tex ----1962%% ---- begin sol/04.tex ----1963\section*{Chapter 4}1964 1965\sol{4.1} By \cref{thm:boundary}(c), for almost every $\theta$ the set $B_\theta$ is a compact neat1966$0$-dimensional submanifold of $[-R,R]$: a finite set of interior points at which $f_\theta'\ne0$, with1967$f_\theta(\pm R)\ne0$. At each such point $f_\theta$ changes sign. So the number of zeros is even if1968$f_\theta(-R)$ and $f_\theta(R)$ have the same sign and odd otherwise.1969 1970\sol{4.2} A compact neat $1$-dimensional submanifold of the disc: a finite disjoint union of circles in the1971interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such1972configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints1973by non-crossing arcs.1974 1975\sol{4.3} $d_xF=(2x_1,-2x_2)$ vanishes only at $x=0$, where $F=c$. So $\Sigma=\{(0,0)\}$ and $\Delta=\{0\}$.1976There are two chambers, $c>0$ and $c<0$. In each the boundary is a hyperbola; for $c<0$ its branches open1977along the $x_1$-axis ($x_1^2=x_2^2-c\ge-c$) and for $c>0$ along the $x_2$-axis. The region $\{f<0\}$ is connected for1978$c<0$ (it contains the origin) and has two components for $c>0$. (The plane is not compact; the statement about chambers holds here1979because the family is a product near infinity up to scaling. Restricted to a disc the same description1980holds.)1981 1982\sol{4.4} At a triple point $p$, $f_\theta$ is transverse to $A_{\{1,2,3\}}$, the line $y_1=y_2=y_3$ in $\R^3$,1983of codimension $2$. So the map $x\mapsto(f_1-f_2,f_2-f_3)$ from the plane to $\R^2$ is a local1984diffeomorphism at $p$, and in the coordinates $(u,v)=(f_1-f_2,f_2-f_3)$ the decision regions are1985$\{u>0,\ u+v>0\}$, $\{u<0,\ v>0\}$ and $\{v<0,\ u+v<0\}$: three sectors bounded by the rays1986$\{u=0,v>0\}$, $\{v=0,u>0\}$ and $\{u+v=0,u<0\}$.1987%% ---- end sol/04.tex ----1988%% ---- begin sol/05.tex ----1989\section*{Chapter 5}1990 1991\sol{5.1} $g=(x_1^2-1)^2+x_2^2$ has $\nabla g=(4x_1(x_1^2-1),2x_2)$, vanishing at $(0,0)$ and $(\pm1,0)$. The1992Hessian is $\operatorname{diag}(12x_1^2-4,\,2)$: at $(\pm1,0)$ it is $\operatorname{diag}(8,2)$, index $0$,1993value $0$; at the origin $\operatorname{diag}(-4,2)$, index $1$, value $1$. The sublevel set $\{g\le t\}$ is1994empty for $t<0$, two discs around $(\pm1,0)$ for $0<t<1$ (on the line $x_2=0$ it is1995$\{|x_1^2-1|\le\sqrt t\}$, two intervals when $t<1$), and one disc for $t>1$. Its Euler characteristic is1996$0,2,1$, consistent with attaching two $0$-cells and then one $1$-cell.1997 1998\sol{5.2} Here $B_t$ is a finite set and $W_\gamma$ is a compact $1$-manifold with boundary $B_0\sqcup B_1$.1999A critical point of $\tau$ on a $1$-manifold is a local minimum or maximum of $t$ along a curve: two points2000of $B_t$ appear together or disappear together (surgery with $\lambda=0$ or $1$ on a $0$-manifold: add or2001remove an $S^0$). Between critical times the points move continuously. If $X$ is a circle, $f_t$ changes2002sign at each point of $B_t$ and returns to its original sign after one circuit, so $\#B_t$ is even for every2003regular $t$.2004 2005\sol{5.3} Two. By \cref{prop:chi} with $d=3$, the Euler characteristic of the region changes by2006$\pm(-1)^k$ at each surgery; the solid torus has $\chi=0$ and needs at least one $0$-handle to be non-empty,2007so at least one more handle of odd index is needed. A $0$-handle (a ball appears, $k=0$) followed by a2008$1$-handle attached to it (the ball becomes a solid torus, $k=1$) suffices. The boundary goes from empty to2009a sphere to a torus.2010 2011\sol{5.4} Write $\Phi=\Phi_\gamma$ and let $t(x)$ be defined by $\Phi(t(x),x)=0$ near $x_i$, with2012$t(x_i)=t_i$. Differentiating once, $\Phi_t\,\partial_jt+\partial_j\Phi=0$, so $\partial_jt(x_i)=0$ because2013$d_x\Phi=0$ there. Differentiating again and evaluating at $x_i$, where the terms containing first2014derivatives of $t$ vanish, gives $\Phi_t\,\partial_j\partial_kt+\partial_j\partial_k\Phi=0$. Hence2015$\nabla^2t(x_i)=-\nabla^2_x\Phi(t_i,x_i)/\Phi_t(t_i,x_i)$.2016%% ---- end sol/05.tex ----2017%% ---- begin sol/06.tex ----2018\section*{Chapter 6}2019 2020\sol{6.1} $\ev(\theta)=\Phi\theta$ with $\Phi$ of rank $n$. Every $y$ is a regular value;