Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1901–2000 of 2141. Source text is displayed literally.

1901\sol{1.4} $F(\theta,x)=\theta+x^3$. Every member has the degenerate critical point $x=0$. A function is1902Morse when its $1$-jet prolongation is transverse to the set of jets with vanishing derivative; the family1903of prolongations $(\theta,x)\mapsto(x,\theta+x^3,3x^2)$ is not transverse to $\{\text{third coordinate}=0\}$,1904since the third coordinate has vanishing derivative in both $\theta$ and $x$ at $x=0$. The parameter moves1905the value of the function but not its derivative, so the family is rich for conditions on values1906(\cref{prin:rich}) and not for conditions on first derivatives. Adding a parameter $\theta_1x$ repairs this:1907$\theta_0+\theta_1x+x^3$ is Morse for $\theta_1\ne0$.1908%% ---- end sol/01.tex ----1909%% ---- begin sol/02.tex ----1910\section*{Chapter 2}1911 1912\sol{2.1} Let $z=Wx+b$ and $h=\sigma(z)$. The lens of the layer has forward part $(W,b,x)\mapsto h$ and1913backward part1914$(W,b,x,w)\mapsto\bigl(\delta x^{\mathsf T},\ \delta,\ W^{\mathsf T}\delta\bigr)$ with1915$\delta=\sigma'(z)\odot w$: the three components are the covectors for $W$, $b$ and $x$. The output layer1916$(a,c;h)\mapsto a^{\mathsf T}h+c$ has backward part $(a,c,h,w)\mapsto(wh,\ w,\ wa)$. The loss1917$\frac12(u-y)^2$ has gradient $e=u-y$ in $u$. Composing: the covector entering the output layer is $e$, so1918$\nabla_a=e\,h$, $\nabla_c=e$, and the covector passed back is $e\,a$; then1919$\delta=\sigma'(z)\odot(e\,a)$, $\nabla_W=\delta x^{\mathsf T}$, $\nabla_b=\delta$.1920 1921\sol{2.2} $\ker JJ^{\mathsf T}=\ker J^{\mathsf T}$ since $\langle v,JJ^{\mathsf T}v\rangle=|J^{\mathsf T}v|^2$;1922so $\rk K=\rk J^{\mathsf T}=\rk J$. For scalar output $K$ is $n\times n$, invertible if and only if $\rk J=n$,1923that is, if and only if the columns of $J$ span $\R^n$; the $k$-th column is the vector1924$\bigl(\partial F/\partial\theta_k(\theta,x_i)\bigr)_{i}$.1925 1926\sol{2.3} By \DT{Prop.~2.9} the $2$-jet of $\lambda\circ\ev$ at $\theta$ has linear part $C A$ and quadratic part1927$C\circ B+E\circ(A\times A)$, where $(A,B)$ is the $2$-jet of $\ev$ (so $A=J$ and $B=(\nabla^2\ev_k)_k$) and1928$(C,E)$ that of $\lambda$ (so $C=\nabla\lambda^{\mathsf T}$, $E=\nabla^2\lambda$). The quadratic part is1929$\sum_k\partial_k\lambda\,\nabla^2\ev_k+J^{\mathsf T}\nabla^2\lambda\,J$.1930 1931\sol{2.4} $\ev(\theta)=\Phi\theta$ with $\Phi$ the $n\times P$ matrix of rows $\phi(x_i)^{\mathsf T}$; so1932$J=\Phi$ for all $\theta$ and $K=\Phi\Phi^{\mathsf T}$, the Gram matrix of the features. The fibres of $\ev$1933are the affine subspaces $\theta_0+\ker\Phi$, of dimension $P-\rk\Phi$. The realization map is linear with1934kernel $\{\theta:\langle\theta,\phi(x)\rangle=0\ \forall x\}$, so the functional dimension is the dimension1935of the span of $\phi(X)$.1936%% ---- end sol/02.tex ----1937%% ---- begin sol/03.tex ----1938\section*{Chapter 3}1939 1940\sol{3.1} $F(\theta,x)=\sum_ja_j\sigma(\langle w_j,x\rangle+b_j)+c$. (i) Permuting the index $j$ reorders the1941sum. (ii) If $\sigma$ is odd, $(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1942(iii) If $\sigma(\lambda s)=\lambda\sigma(s)$ for $\lambda>0$, then1943$(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1944 1945\sol{3.2} The columns of $J$ belonging to $(w_j,b_j)$ are1946$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i$ and1947$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i$; if $a_j=0$ these $d+1$ columns vanish, so1948$\rk J\le P-(d+1)$. If $(w_j,b_j)=(w_k,b_k)$, the columns for $a_j$ and $a_k$ are both1949$\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i$, so two columns coincide and $\rk J\le P-1$. (In the second1950case the columns for $(w_j,b_j)$ and $(w_k,b_k)$ are also proportional, with ratio $a_j/a_k$, so the rank1951drops by $d+2$ in all.)1952 1953\sol{3.3} $\theta=(w,b,a,c)$ and $\xi=(w,b,-a,0)$. Where $wx+b>0$, $F=a(wx+b)+c$ and1954$D_\theta F=(ax,\ a,\ wx+b,\ 1)$, so $D_\theta F\cdot\xi=awx+ab-a(wx+b)=0$. Where $wx+b<0$, $F=c$ and1955$D_\theta F=(0,0,0,1)$, again orthogonal to $\xi$. Here $P=4$, $H=1$, so the functional dimension is at most1956$3$; it equals $3$ when $a\ne0$ and $w\ne0$, since the function $x\mapsto a\,\relu(wx+b)+c$ is determined by1957its breakpoint $-b/w$, the slope $aw$ on the active side, and $c$ (and the side, which is discrete).1958 1959\sol{3.4} $(2,3,1)$: one hidden layer of $3$ units, $|G|=2^3\cdot3!=48$. $(2,4,4,1)$: two hidden layers of1960$4$ units, $|G|=(2^4\cdot4!)^2=384^2=147456$.1961%% ---- end sol/03.tex ----1962%% ---- begin sol/04.tex ----1963\section*{Chapter 4}1964 1965\sol{4.1} By \cref{thm:boundary}(c), for almost every $\theta$ the set $B_\theta$ is a compact neat1966$0$-dimensional submanifold of $[-R,R]$: a finite set of interior points at which $f_\theta'\ne0$, with1967$f_\theta(\pm R)\ne0$. At each such point $f_\theta$ changes sign. So the number of zeros is even if1968$f_\theta(-R)$ and $f_\theta(R)$ have the same sign and odd otherwise.1969 1970\sol{4.2} A compact neat $1$-dimensional submanifold of the disc: a finite disjoint union of circles in the1971interior and arcs with both endpoints on the boundary circle, meeting it transversally. Any such1972configuration, up to isotopy, is determined by the nesting of the circles and the pairing of the endpoints1973by non-crossing arcs.1974 1975\sol{4.3} $d_xF=(2x_1,-2x_2)$ vanishes only at $x=0$, where $F=c$. So $\Sigma=\{(0,0)\}$ and $\Delta=\{0\}$.1976There are two chambers, $c>0$ and $c<0$. In each the boundary is a hyperbola; for $c<0$ its branches open1977along the $x_1$-axis ($x_1^2=x_2^2-c\ge-c$) and for $c>0$ along the $x_2$-axis. The region $\{f<0\}$ is connected for1978$c<0$ (it contains the origin) and has two components for $c>0$. (The plane is not compact; the statement about chambers holds here1979because the family is a product near infinity up to scaling. Restricted to a disc the same description1980holds.)1981 1982\sol{4.4} At a triple point $p$, $f_\theta$ is transverse to $A_{\{1,2,3\}}$, the line $y_1=y_2=y_3$ in $\R^3$,1983of codimension $2$. So the map $x\mapsto(f_1-f_2,f_2-f_3)$ from the plane to $\R^2$ is a local1984diffeomorphism at $p$, and in the coordinates $(u,v)=(f_1-f_2,f_2-f_3)$ the decision regions are1985$\{u>0,\ u+v>0\}$, $\{u<0,\ v>0\}$ and $\{v<0,\ u+v<0\}$: three sectors bounded by the rays1986$\{u=0,v>0\}$, $\{v=0,u>0\}$ and $\{u+v=0,u<0\}$.1987%% ---- end sol/04.tex ----1988%% ---- begin sol/05.tex ----1989\section*{Chapter 5}1990 1991\sol{5.1} $g=(x_1^2-1)^2+x_2^2$ has $\nabla g=(4x_1(x_1^2-1),2x_2)$, vanishing at $(0,0)$ and $(\pm1,0)$. The1992Hessian is $\operatorname{diag}(12x_1^2-4,\,2)$: at $(\pm1,0)$ it is $\operatorname{diag}(8,2)$, index $0$,1993value $0$; at the origin $\operatorname{diag}(-4,2)$, index $1$, value $1$. The sublevel set $\{g\le t\}$ is1994empty for $t<0$, two discs around $(\pm1,0)$ for $0<t<1$ (on the line $x_2=0$ it is1995$\{|x_1^2-1|\le\sqrt t\}$, two intervals when $t<1$), and one disc for $t>1$. Its Euler characteristic is1996$0,2,1$, consistent with attaching two $0$-cells and then one $1$-cell.1997 1998\sol{5.2} Here $B_t$ is a finite set and $W_\gamma$ is a compact $1$-manifold with boundary $B_0\sqcup B_1$.1999A critical point of $\tau$ on a $1$-manifold is a local minimum or maximum of $t$ along a curve: two points2000of $B_t$ appear together or disappear together (surgery with $\lambda=0$ or $1$ on a $0$-manifold: add or