Indexed text attachment
representation-theory-of-parametrized-neural-models-on-spaces.tex
Source for Representation Theory of Parametrized Neural Models on Spaces
View paper · Download original file
Lines 1201–1300 of 1343. Source text is displayed literally.
1201 1202\begin{proposition}[An equivariant gradient field]1203\label{prop:gradient}1204Suppose a compact Lie group $G$ acts smoothly on a1205finite-dimensional, paracompact parameter manifold $P$ and1206the smooth scalar loss $L$ is invariant. If $F_P$ is a smooth1207invariant positive semidefinite tensor, there exists a1208$G$-invariant Riemannian metric $j$ such that for every1209$\lambda>0$ the gradient of $L$ with respect to1210$g_\lambda=F_P+\lambda j$ is equivariant. Its local flow1211is equivariant. If the action is free, this field is1212horizontal for $P\to P/G$ in $g_\lambda$ and projects to the1213gradient of the descended loss for the induced quotient metric.1214\end{proposition}1215 1216\begin{proof}1217Choose any Riemannian metric and average its pullbacks using1218normalized Haar measure. Integration of smooth functions1219over a compact Lie group gives smoothness. Positive1220definiteness follows by averaging positive values on a1221nonzero tangent vector. Invariance follows by the right1222invariance of compact Haar measure. This yields $j$, and the1223sum $g_\lambda$ remains invariant and positive definite.1224 1225For each $a\in G$, differentiate $L\circ a=L$ and apply1226metric invariance. Nondegeneracy of $g_\lambda$ then gives1227$D a\,\grad L(p)=\grad L(ap)$. Uniqueness for local ordinary1228differential equations supplies the flow compatibility.1229In the free case the compact action is proper, so $P\to P/G$1230is a smooth principal bundle. The loss differential vanishes1231on vertical vectors. Hence $\grad L$ is horizontal, and the1232invariant horizontal metric induces a quotient metric by1233the differential of the projection. Pairing the projected1234field with arbitrary quotient tangent vectors by horizontal1235lift gives the gradient identity on $P/G$.1236\end{proof}1237 1238For a nonfree action the equivariant evolution carries1239presentations to presentations and induces local maps of the1240quotient stack on the invariant domains of the flow. For the1241scalar deck-symmetric model, a metric can make optimization1242well defined at the origin, but the first derivative of a1243smooth loss of its law still vanishes there. Regularization1244changes the identification of covectors with vectors. It1245does not enlarge the derivative of the realization.1246 1247\section{Conclusion}1248 1249Over an ordinary smooth base the construction recovers1250smooth networks and statistical families, with cotangent1251composition and Fisher geometry when present. Over a1252homogeneous feature space it recovers induced bundles and1253intertwining layers. Allowing the weights to move contributes1254additional harmonic types, determined exactly by the tensor1255multiplicities in Theorem~\ref{thm:harmonic}. These mechanisms1256are compatible because composition and families are1257represented in the same slices.1258 1259The higher model distinguishes observable equality from1260equivalence of parameter presentations. Descent records the1261transition arrows and their coherence, while homotopy fixed1262points describe the comparison spaces for stable features.1263The circle decoder makes that distinction visible without1264changing the input or output law. For each specialization1265the relevant structures enter explicitly: a smooth site1266for smooth realizations, a probability interpretation for1267sampling, and a compact unitary action for the Hilbert1268decomposition.1269\par1270\begin{thebibliography}{99}1271\small\raggedright1272 1273\bibitem{amari}1274S.-i. Amari. Natural gradient works efficiently in learning.1275\emph{Neural Computation} 10(2), 251--276, 1998.1276\href{https://doi.org/10.1162/089976698300017746}{DOI}.1277 1278\bibitem{reprbook}1279Anonymous. \emph{Representation Theory in Higher Categories:1280From Compact Harmonic Analysis to Coherent Group Actions}.1281Higher Categorical Semantics, 2026. Chapters 2--4, 17--18, 20, and 27--28.1282\href{https://higher-categorical-semantics.pages.dev/research/representation-theory-in-higher-categories-1d29df27}{Published edition}.1283 1284\bibitem{ay-jost-le-schwachhoefer-2015}1285N. Ay, J. Jost, H. V. L\^e, and L. Schwachh\"ofer.1286Information geometry and sufficient statistics.1287\emph{Probability Theory and Related Fields} 162, 327--364, 2015.1288\href{https://doi.org/10.1007/s00440-014-0574-8}{DOI}.1289 1290\bibitem{cohen-geiger-weiler}1291T. S. Cohen, M. Geiger, and M. Weiler.1292A general theory of equivariant CNNs on homogeneous spaces.1293\emph{Advances in Neural Information Processing Systems} 32, 2019.1294\href{https://proceedings.neurips.cc/paper_files/paper/2019/hash/b9cfe8b6042cf759dc4c0cccb27a6737-Abstract.html}{Proceedings}.1295 1296\bibitem{cruttwell}1297G. S. H. Cruttwell, B. Gavranovi\'c, N. Ghani, P. Wilson, and F. Zanasi.1298Categorical foundations of gradient-based learning.1299In \emph{Programming Languages and Systems}, ESOP 2022.1300\href{https://doi.org/10.1007/978-3-030-99336-8_1}{DOI}.