Independent scholarship · Open accessThe monthly digest

Indexed text attachment

parametrized-neural-networks-and-their-representations-in-differential-topology.tex

Source for Parametrized Neural Networks and Their Representations in Differential Topology

View paper · Download original file

Lines 1861–1960 of 2141. Source text is displayed literally.

1861positive and $9$ zero eigenvalues (\cref{thm:interp}); the inequality of \cref{prop:PL}; sublevel1862components, the barrier and the Hessians of \cref{ex:hyperbola}; \cref{prop:notproper}; \cref{exer:deeplin};1863invariance of the loss under the symmetries\\1864\ref{ch:isotopy} & random first layers of width $3$ embed a circle in $\R^5$ (\cref{prop:firstlayer}), and a1865coordinate projection that does not (\cref{exer:proj}); the convex hull property of \cref{thm:obstruction}1866for random injective networks; order preservation on the line; positivity of the Jacobian determinant and1867Liouville's formula for a planar neural ODE (\cref{prop:node}); the swap of \cref{exer:swap}; invariance of1868the linking number under a neural ODE in $\R^3$ (\cref{cor:nodeobst}); separation of nested circles after1869augmentation (\cref{cor:separate})\\1870\ref{ch:degree} & convergence of the fixed-point inversion and the determinant bound for a residual block1871(\cref{prop:iresnet}, \cref{exer:det}); a non-proper local diffeomorphism (\cref{thm:hadamard}); odd numbers1872of equilibria with signed count $1$ in one and two dimensions (\cref{thm:deq})\\1873\bottomrule1874\end{longtable}1875}1876%% ---- end ch/B-computations.tex ----1877%% ---- begin ch/C-solutions.tex ----1878\chapter{Solutions to the Exercises}\label{app:solutions}1879 1880Solutions to all exercises follow, by chapter. Where a solution relies on a computation, the computation is1881also carried out by \texttt{verify-parametrized-networks.py}.1882%% ---- end ch/C-solutions.tex ----1883%% ---- begin sol/01.tex ----1884\section*{Chapter 1}1885 1886\sol{1.1} $(2,3,1)$: $P=3\cdot3+1\cdot4=13$, $H=3$. $(2,4,4,1)$: $P=4\cdot3+4\cdot5+1\cdot5=37$, $H=8$.1887$(d,k,1)$: $P=k(d+1)+(k+1)=k(d+2)+1$, $H=k$.1888 1889\sol{1.2} Let $(\Theta,F)\colon X\to Y$, $(\Theta',G)\colon Y\to Z$, $(\Theta'',K)\colon Z\to V$. Both1890bracketings have underlying function $(\theta,\theta',\theta'',x)\mapsto K(\theta'',G(\theta',F(\theta,x)))$,1891on the parameter spaces $(\Theta\times\Theta')\times\Theta''$ and $\Theta\times(\Theta'\times\Theta'')$1892respectively. The reassociation diffeomorphism $r$ satisfies $F_{\text{left}}=F_{\text{right}}\circ(r\times1893\id_X)$, so $r$ and $r^{-1}$ are $2$-cells, mutually inverse.1894 1895\sol{1.3} The diagonal $\delta\colon\Theta\to\Theta\times\Theta$ gives a $2$-cell1896$(\Theta,(F\circ F)\circ(\delta\times\id))\Rightarrow(\Theta\times\Theta,F\circ F)$, and its source is1897$(\theta,x)\mapsto F(\theta,F(\theta,x))$, the weight-tied network. By the chain rule, the derivative of1898$\theta\mapsto G(\delta(\theta))$ is $\partial_1G+\partial_2G$ evaluated on the diagonal, for1899$G(\theta_1,\theta_2)=F(\theta_2,F(\theta_1,x))$.1900 1901\sol{1.4} $F(\theta,x)=\theta+x^3$. Every member has the degenerate critical point $x=0$. A function is1902Morse when its $1$-jet prolongation is transverse to the set of jets with vanishing derivative; the family1903of prolongations $(\theta,x)\mapsto(x,\theta+x^3,3x^2)$ is not transverse to $\{\text{third coordinate}=0\}$,1904since the third coordinate has vanishing derivative in both $\theta$ and $x$ at $x=0$. The parameter moves1905the value of the function but not its derivative, so the family is rich for conditions on values1906(\cref{prin:rich}) and not for conditions on first derivatives. Adding a parameter $\theta_1x$ repairs this:1907$\theta_0+\theta_1x+x^3$ is Morse for $\theta_1\ne0$.1908%% ---- end sol/01.tex ----1909%% ---- begin sol/02.tex ----1910\section*{Chapter 2}1911 1912\sol{2.1} Let $z=Wx+b$ and $h=\sigma(z)$. The lens of the layer has forward part $(W,b,x)\mapsto h$ and1913backward part1914$(W,b,x,w)\mapsto\bigl(\delta x^{\mathsf T},\ \delta,\ W^{\mathsf T}\delta\bigr)$ with1915$\delta=\sigma'(z)\odot w$: the three components are the covectors for $W$, $b$ and $x$. The output layer1916$(a,c;h)\mapsto a^{\mathsf T}h+c$ has backward part $(a,c,h,w)\mapsto(wh,\ w,\ wa)$. The loss1917$\frac12(u-y)^2$ has gradient $e=u-y$ in $u$. Composing: the covector entering the output layer is $e$, so1918$\nabla_a=e\,h$, $\nabla_c=e$, and the covector passed back is $e\,a$; then1919$\delta=\sigma'(z)\odot(e\,a)$, $\nabla_W=\delta x^{\mathsf T}$, $\nabla_b=\delta$.1920 1921\sol{2.2} $\ker JJ^{\mathsf T}=\ker J^{\mathsf T}$ since $\langle v,JJ^{\mathsf T}v\rangle=|J^{\mathsf T}v|^2$;1922so $\rk K=\rk J^{\mathsf T}=\rk J$. For scalar output $K$ is $n\times n$, invertible if and only if $\rk J=n$,1923that is, if and only if the columns of $J$ span $\R^n$; the $k$-th column is the vector1924$\bigl(\partial F/\partial\theta_k(\theta,x_i)\bigr)_{i}$.1925 1926\sol{2.3} By \DT{Prop.~2.9} the $2$-jet of $\lambda\circ\ev$ at $\theta$ has linear part $C A$ and quadratic part1927$C\circ B+E\circ(A\times A)$, where $(A,B)$ is the $2$-jet of $\ev$ (so $A=J$ and $B=(\nabla^2\ev_k)_k$) and1928$(C,E)$ that of $\lambda$ (so $C=\nabla\lambda^{\mathsf T}$, $E=\nabla^2\lambda$). The quadratic part is1929$\sum_k\partial_k\lambda\,\nabla^2\ev_k+J^{\mathsf T}\nabla^2\lambda\,J$.1930 1931\sol{2.4} $\ev(\theta)=\Phi\theta$ with $\Phi$ the $n\times P$ matrix of rows $\phi(x_i)^{\mathsf T}$; so1932$J=\Phi$ for all $\theta$ and $K=\Phi\Phi^{\mathsf T}$, the Gram matrix of the features. The fibres of $\ev$1933are the affine subspaces $\theta_0+\ker\Phi$, of dimension $P-\rk\Phi$. The realization map is linear with1934kernel $\{\theta:\langle\theta,\phi(x)\rangle=0\ \forall x\}$, so the functional dimension is the dimension1935of the span of $\phi(X)$.1936%% ---- end sol/02.tex ----1937%% ---- begin sol/03.tex ----1938\section*{Chapter 3}1939 1940\sol{3.1} $F(\theta,x)=\sum_ja_j\sigma(\langle w_j,x\rangle+b_j)+c$. (i) Permuting the index $j$ reorders the1941sum. (ii) If $\sigma$ is odd, $(-a_j)\sigma(-\langle w_j,x\rangle-b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1942(iii) If $\sigma(\lambda s)=\lambda\sigma(s)$ for $\lambda>0$, then1943$(a_j/\lambda)\sigma(\lambda\langle w_j,x\rangle+\lambda b_j)=a_j\sigma(\langle w_j,x\rangle+b_j)$.1944 1945\sol{3.2} The columns of $J$ belonging to $(w_j,b_j)$ are1946$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\,x_i\bigr)_i$ and1947$\bigl(a_j\sigma'(\langle w_j,x_i\rangle+b_j)\bigr)_i$; if $a_j=0$ these $d+1$ columns vanish, so1948$\rk J\le P-(d+1)$. If $(w_j,b_j)=(w_k,b_k)$, the columns for $a_j$ and $a_k$ are both1949$\bigl(\sigma(\langle w_j,x_i\rangle+b_j)\bigr)_i$, so two columns coincide and $\rk J\le P-1$. (In the second1950case the columns for $(w_j,b_j)$ and $(w_k,b_k)$ are also proportional, with ratio $a_j/a_k$, so the rank1951drops by $d+2$ in all.)1952 1953\sol{3.3} $\theta=(w,b,a,c)$ and $\xi=(w,b,-a,0)$. Where $wx+b>0$, $F=a(wx+b)+c$ and1954$D_\theta F=(ax,\ a,\ wx+b,\ 1)$, so $D_\theta F\cdot\xi=awx+ab-a(wx+b)=0$. Where $wx+b<0$, $F=c$ and1955$D_\theta F=(0,0,0,1)$, again orthogonal to $\xi$. Here $P=4$, $H=1$, so the functional dimension is at most1956$3$; it equals $3$ when $a\ne0$ and $w\ne0$, since the function $x\mapsto a\,\relu(wx+b)+c$ is determined by1957its breakpoint $-b/w$, the slope $aw$ on the active side, and $c$ (and the side, which is discrete).1958 1959\sol{3.4} $(2,3,1)$: one hidden layer of $3$ units, $|G|=2^3\cdot3!=48$. $(2,4,4,1)$: two hidden layers of1960$4$ units, $|G|=(2^4\cdot4!)^2=384^2=147456$.