Indexed text attachment
representation-theory-of-parametrized-neural-models-on-spaces.tex
Source for Representation Theory of Parametrized Neural Models on Spaces
View paper · Download original file
Lines 261–360 of 1343. Source text is displayed literally.
261evaluations and a space of weights mapping into it. An architecture262selects a classifying map, for instance by composing affine maps,263an equivariant convolution, and a nonlinearity. The ambient264exponential also includes maps that the selected architecture265cannot realize.266 267\subsection{Transport and actions}268 269In spaces, straightening identifies $\Spc_{/B}$ with270$\Fun(B,\Spc)$, where the base $B$ is regarded as an271$\infty$-groupoid. Transport acts by equivalences and the slice272exponential is computed fiberwise \cite[\S3.2]{lurie-htt}. For273a path $\gamma:b\to c$ with transports $t_X,t_Y$, exponentials274transport by $h\mapsto t_Y h t_X^{-1}$. A classifying map275is consequently a coherent family of maps276$P_b\to [X_b,Y_b]$ commuting with transport. For an ordinary277action of an abstract group $G$ on $P,X,Y$, the resulting278comparison reads279\begin{equation}280 \label{eq:equiv-eval}281 f(gp,gx)=g f(p,x).282\end{equation}283It is a condition on the entire family. At a fixed $p$, it284forces equivariance under the stabilizer $G_p$. In a smooth285site one can instead use the classifying stack of a Lie group.286Sections of that stack are torsors with smooth local287trivializations, and \eqref{eq:equiv-eval} then holds for288smooth test families. The homotopy type of a smooth base289retains path transport and may identify smoothly distinct data.290 291\subsection{Realization on a quotient}292 293Suppose a group object $G$ acts on $P$ and a realization294$r:P\to M$ is coherently invariant with trivial action on295$M$. Here $M$ can itself be a space of functions or probability296laws. The action groupoid has simplicial levels $P\times G^n$.297Its colimit is $P//G$, and the colimit universal property gives298\[299 \Map(P//G,M)\simeq \Tot\,\Map(P\times G^\bullet,M).300\]301Thus $r$ factors through $P//G$ exactly when equipped with302descent, whose first comparison identifies $r(p)$ and $r(gp)$.303Equivalences between local parameter presentations are part304of a family in the quotient. Over $\Spc$, the fiber of305$P//G\to BG$ over the base point is the parameter space $P$,306with transport the original action. For smooth $P$ and Lie307$G$, the same construction in sheaves gives the smooth quotient308stack. The example in Section~\ref{gauge:circle} distinguishes309the quotient presentations from their common observable laws.310 311\subsection{Stochastic evaluation}312\label{sec:stochastic}313 314In a cartesian closed category, a strong monad $T$ has a315natural strength $\mathrm{st}_{A,Y}:A\times T Y\to T(A\times Y)$316compatible with products, unit $\eta$, and multiplication $\mu$.317When $T$ is a probability monad, the layer $P\times X\to T Y$318assigns a probability-valued prediction. Currying represents it319by $P\to[X,T Y]$. For320$f:P\times X\to T Y$ and $g:Q\times Y\to T Z$, their321composite with parameter $P\times Q$ is322\[323 \begin{aligned}324 P\times Q\times X &\longrightarrow Q\times TY\\325 &\xrightarrow{\ \mathrm{st}\ } T(Q\times Y)326 \xrightarrow{\ T g\ } T^2 Z327 \xrightarrow{\ \mu_Z\ } TZ ,328 \end{aligned}329\]330with first map $(p,q,x)\mapsto(q,f(p,x))$.331For a formal check, fix a context $A$ and form the category332whose arrows $X\to Y$ are maps $A\times X\to Y$. It is the333co-Kleisli category of the comonad $A\times(-)$ with diagonal334and projection. Strength is a mixed distributive law335$A\times T(-)\to T(A\times(-))$ for this comonad and the monad336$T$. Its compatibility with the diagonal and projection is337product coherence and naturality in $A$. Monad unit and338multiplication compatibility are the strength axioms.339Thus $T$ acts as a monad on the category with context $A$.340Kleisli composition for that monad extends a map341$h:A\times Y\to T Z$ using strength followed by $T h$ and342$\mu_Z$. Taking $A=P\times Q$ and restricting the layers along343its two projections gives the displayed formula. Context maps344preserve the construction by naturality of strength.345A compatible system of monads on slices gives the same346construction over $B$ when their strengths and monad347operations commute with base change.348 349For instance, the finitely supported distribution monad on sets350gives finite channels by restricting the feature and output sets351to be finite. If $p,q$ are fixed, the composite has law352\[353 k_{p,q}(z\mid x)=\sum_y g_q(z\mid y) f_p(y\mid x).354\]355Dirac laws embed deterministic evaluations. Repeated sampling can356share an unknown parameter by pulling a product family back along357its diagonal. For a coin of bias $1/5$ or $4/5$ with equal prior358probability, shared-parameter sampling gives probability $17/50$359of two successes. Independently drawing the parameter for each360observation gives $1/4$. The single-observation law therefore