Independent scholarship · Open accessThe monthly digest

Indexed text attachment

operads-modules-machine-learning.tex

Source for Operads, Modules, and Differentiation in Machine Learning

View paper · Download original file

Lines 221–320 of 1174. Source text is displayed literally.

221\]222In a block followed by a scalar loss, \(G\) is the incoming223matrix cotangent. The adjoint sends it back to the shared224matrix parameter and adds the contributions of its occurrences.225\end{example}226 227\subsection{The scope of homological conversion}228 229The paired monad of Part~I, Definition~4.3 and Proposition~4.4,230assigns \((X,Y)\) the pair \((T_{\mathcal O}(X),V_X\otimes Y)\),231where \(V_X\) is the same marked-slot expression with \(X\)232in place of \(A\). Its algebras comprise an operadic algebra233and a module over it. In Part~II, Section~4, the authors234use the two-sided monadic bar construction on such data.235For a monad \(T\), a right \(T\)-functor \(F\), and a236\(T\)-algebra \(A\), the simplicial degree \(q\) is \(F T^q A\).237Its first and last faces use the right action and algebra238action, respectively. Interior faces use monad multiplication,239and degeneracies use the unit. The augmentation of240\(B_\bullet(T,T,A)\) has a simplicial contraction after forgetting241the algebra structure (Part~II, Example~4.2).242 243With a \(\Sigma\)-projective simplicial operad the conversion244theorem treats partial simplicial algebras and modules.245For an aritywise quasi-isomorphism of dg operads over246a characteristic-zero field, the dg comparison applies to247genuine algebras and their modules248(Part~II, Theorems~1.1 and 1.4). Part~V constructs249derived tensor and Hom with a cell replacement in its250\(E_\infty\)-module setting, notably Sections~5, 6, and 8.251These hypotheses govern applications to complexes, such as252coherent interfaces with derived composition. The equivalence253with modules over \(U_{\mathcal O}(A)\) by itself supplies254no canonical symmetric monoidal tensor on modules over255an arbitrary enveloping algebra. Preservation of numerical256predictions or loss requires a separate comparison intertwining257the chosen evaluations.258 259\section{Smooth semantics and a linear operadic module}260\label{sec:tangent-module}261 262\subsection{Operations, substitutions, and base points}263 264Fix a set of colors. A color \(c\) has a finite-dimensional real vector space \(X_c\) as its interpretation. Colors can include product spaces or a zero-dimensional space. Let \(P\) be a symmetric colored operad in sets. An operation \(p\in P(c_1,\dots,c_n;d)\) is interpreted by a smooth map265\[266 f_p:X_{c_1}\times\cdots\times X_{c_n}\longrightarrow X_d .267\]268For a nullary operation the product is a point. An interpretation is a smooth \(P\)-algebra when it respects the units, permutations, and simultaneous substitution. In particular, substituting \(q_i\) into input \(i\) gives the map269\[270 f_{p(q_1,\ldots,q_n)}(\boldsymbol x_1,\ldots,\boldsymbol x_n)271   =f_p\big(f_{q_1}(\boldsymbol x_1),\ldots,272             f_{q_n}(\boldsymbol x_n)\big).273\]274The tuples \(\boldsymbol x_i\) are distinct input blocks of the substituted operation. Reuse of a variable can be represented by a copy map, as in Section~\ref{sec:reverse}. An example is the operad of all smooth maps between products of the chosen spaces, with the block substitution above. Another is the free operad on typed primitives such as an activation, an affine map, or a sum.275 276Write \(\mathcal O=\mathbb R[P]\) for the real linearization with a basis element \([p]\) for every operation. The bijection between the product basis and the tensor basis gives277\(\mathbb R[S\times T]\cong\mathbb R[S]\otimes_{\mathbb R}\mathbb R[T]\).278Thus independently extending substitutions by linearity defines an operad in real vector spaces. We regard it as a dg operad in degree zero. Formal addition of operations takes place in \(\mathcal O\). Even when \(f_p+f_q\) is interpreted by some operation \(r\), the basis element \([r]\) and the formal sum \([p]+[q]\) have different actions in general.279 280For each color let281\[282 A_c=\mathbb R[X_c],\qquad283 M_c=\bigoplus_{x\in X_c}T_xX_c .284\]285Here \([x]\) denotes a basis vector of \(A_c\) and \(\iota_x\) denotes the inclusion of a tangent fiber into \(M_c\). Both use finite support. These objects capture point and first-order data algebraically. In particular, the free vector space has no prescribed completion or topology.286 287\subsection{The one-marked-input action}288 289For an input tuple \(\boldsymbol x\), let \(D_i f_p(\boldsymbol x)\) be the derivative with only the \(i\)-th input varying. It maps \(T_{x_i} X_{c_i}\) to \(T_{f_p(\boldsymbol x)}X_d\).290 291\begin{theorem}[Tangent module over the linearization]292\label{thm:tangent-module}293The formulas294\begin{align}295 A(p)([x_1],\ldots,[x_n])&=[f_p(\boldsymbol x)],\label{eq:a-act}\\296 M_i(p)([x_1],\ldots,\iota_{x_i}v,\ldots,[x_n])297   &=\iota_{f_p(\boldsymbol x)}D_i f_p(\boldsymbol x)v\label{eq:m-act}298\end{align}299extend to an \(\mathcal O\)-algebra \(A\) and an operadic \(A\)-module \(M\). For one color the action is of the form in K\v{r}\'{\i}\v{z} and May, Part~I, Definition~4.1. The colored action satisfies the same one-marked-input axioms with matching input and output colors.300\end{theorem}301 302\begin{proof}303The first formula defines a multilinear map on the tensor of all input spaces since it specifies the values on pure basis tensors. In the second formula, fix all unmarked basis points. The map into \(M_d\) is linear on each \(T_{x_i}X_{c_i}\). The direct sum property extends it to \(M_{c_i}\), and multilinearity in the other basis inputs completes the definition. Finally extend linearly in \([p]\).304 305Operad substitution holds in \(A\) because the interpretation respects substitution. For the module equation take a marked input \(j\) of \(q_i\), and put \(y_a=f_{q_a}(\boldsymbol x_a)\). The partial chain rule gives306\begin{equation}307 \begin{split}308 D_{(i,j)} f_{p(q_1,\ldots,q_n)}(\boldsymbol x)309  &=D_i f_p(\boldsymbol y)\,D_j f_{q_i}(\boldsymbol x_i).310 \end{split}311 \label{eq:partial-chain}312\end{equation}313All unmarked blocks use \(A\), so they supply the basis elements \([y_a]\). The inner marked action produces a vector in \(T_{y_i}X_{c_i}\), and the outer marked action applies the first factor in \eqref{eq:partial-chain}. This agrees with evaluating the marked slot of the substituted operation. Units and permutations respect the same formulas, since the derivative of the identity is the identity and permutation reindexes the summands. The equations extend from the generating tensors to all inputs by multilinearity.314\end{proof}315 316Equation~\eqref{eq:partial-chain} means that the derivative in a single input of a tree of operations composes along its path to the output. The unmarked branches compute the base points for the derivative along that path. In the one-color case these compositions act through the enveloping algebra \(U_{\mathcal O}(A)\) of K\v{r}\'{\i}\v{z} and May. In the colored case each one-marked map has a specified source and target color, and the relations equate two composites only when their colors match.317 318\begin{proposition}[The affine relations in a tangent fiber]319\label{prop:affine-quotient}320The smooth tangent operations on the spaces \(TX_c\) give an \(\mathcal O\)-algebra on \(\mathbb R[TX]\). There is a canonical surjective algebra map