Indexed text attachment
representation-theory-of-parametrized-neural-models-on-spaces.tex
Source for Representation Theory of Parametrized Neural Models on Spaces
View paper · Download original file
Lines 1121–1220 of 1343. Source text is displayed literally.
1121and let $R=k[C_2]\simeq k[u]/(u^2)$ with $u=g-1$.1122The exact sequence of $R$-modules1123\[1124 0\longrightarrow k\xrightarrow{\,1\mapsto u\,}1125 R\xrightarrow{\,\varepsilon\,}k\longrightarrow 01126\]1127splits over $k$. An $R$-linear splitting would choose a lift1128of $1$ in $R$ killed by $u$. Every such lift is $1+au$, and1129$u(1+au)=u\ne0$. Hence the sequence gives a nonzero class of1130equivariant maps $k\to k[1]$, becoming zero when the1131action is forgotten. The free resolution obtained by repeating1132multiplication by $u$ shows1133$\Ext^j_R(k,k)\simeq k$ for every $j\ge 0$.1134This gives the positive cohomological degrees in the homotopy1135fixed point spectrum of the endomorphism spectrum of $k$.1136\end{example}1137 1138The regular representation in the example and the trivial1139module are objects of the same stable coefficient category.1140To identify the action category with derived $k[G]$-modules,1141evaluation at the unique object of $BG$ is conservative and1142preserves colimits. Its left adjoint is the free action, whose1143underlying monad is $k[G]\otimes_k^L(-)$ with multiplication1144from $G$. The same monad comes from extension and1145restriction of scalars for $k\to k[G]$. Monadicity gives the1146identification \cite[\S4.7.3]{lurie-ha}. Over a ring where1147$|G|$ is a unit, the averaging idempotent makes the trivial1148module projective and the higher Ext groups in this example1149vanish. This clarifies the relation with classical averaging.1150With real or complex Hilbert features and continuous compact1151actions, averaging and decomposition follow instead from Haar1152integration under the analytic hypotheses of1153Section~\ref{sec:harmonics}.1154\par1155\section{Losses, derivatives, and symmetry}1156 1157The classifying map of a layer can be used for optimization1158when the parameter and feature objects carry differential1159structure. Work in smooth manifolds, or use smooth charts of1160bundles over a common base. Let1161$h=f(p,x)$, $y=g(q,h)$, and let a smooth loss have output1162covector $\lambda_y=d_y\ell$. The partial reverse derivatives1163of the two-layer realization are1164\[1165 \begin{aligned}1166 \lambda_q&=(D_qg)^*\lambda_y,1167 &\lambda_h&=(D_hg)^*\lambda_y,\\1168 \lambda_p&=(D_pf)^*\lambda_h,1169 &\lambda_x&=(D_xf)^*\lambda_h .1170 \end{aligned}1171\]1172The dual maps in this display are cotangent pullbacks, with1173no metric identification. The chain rule proves that they glue1174across smooth changes of charts, using the usual pullback of1175a loss along the realization. Under a tying map1176$u:P\to\prod_iP_i$ the resulting parameter covector is1177$(D u)^*(\lambda_{p_1},\ldots,\lambda_{p_n})$. In particular,1178tying two copies by a diagonal adds their covectors at the1179shared point. The same map specifies sharing in the forward1180composite of Section~\ref{sec:relative}. This is the1181cotangent form of the compatibility exploited by categorical1182formulations of backpropagation1183\cite{fong-spivak-tuyeras,cruttwell}.1184 1185A scalar expected loss can be formed from the conditional laws1186of Section~\ref{stat:section}, using a finite dataset or a1187distribution for which integration and differentiation are1188justified. Gradient vectors additionally require a positive1189metric or another update rule. For example, take a1190$G$-invariant statistical realization $r:P\to\mathcal M$,1191where $\mathcal M$ is a smooth space of laws with Fisher metric1192$F_{\mathcal M}$. The pullback $F_P=r^* F_{\mathcal M}$ is1193positive semidefinite. It annihilates vectors tangent to1194$G$-orbits and, more generally, $\ker D r$, which can be1195larger than the orbit tangent space. The degeneracies in1196Sections~\ref{stat:section} and~\ref{gauge:circle} arise even1197when the relevant symmetry group at the parameter is discrete1198or has tangent orbit of smaller dimension. The inverse-Fisher1199natural gradient is defined where the chosen statistical1200coordinates have nondegenerate information \cite{amari}.1201 1202\begin{proposition}[An equivariant gradient field]1203\label{prop:gradient}1204Suppose a compact Lie group $G$ acts smoothly on a1205finite-dimensional, paracompact parameter manifold $P$ and1206the smooth scalar loss $L$ is invariant. If $F_P$ is a smooth1207invariant positive semidefinite tensor, there exists a1208$G$-invariant Riemannian metric $j$ such that for every1209$\lambda>0$ the gradient of $L$ with respect to1210$g_\lambda=F_P+\lambda j$ is equivariant. Its local flow1211is equivariant. If the action is free, this field is1212horizontal for $P\to P/G$ in $g_\lambda$ and projects to the1213gradient of the descended loss for the induced quotient metric.1214\end{proposition}1215 1216\begin{proof}1217Choose any Riemannian metric and average its pullbacks using1218normalized Haar measure. Integration of smooth functions1219over a compact Lie group gives smoothness. Positive1220definiteness follows by averaging positive values on a