Indexed text attachment
operads-modules-machine-learning.tex
Source for Operads, Modules, and Differentiation in Machine Learning
View paper · Download original file
Lines 1001–1100 of 1174. Source text is displayed literally.
1001Fix a rooted tree with a nonempty finite answer set \(X_v\) at each vertex.1002For an internal vertex \(v\), let \(I_v\) be the product of the1003answer sets of its children. Two evaluators have local Markov1004kernels \(K_v,\widetilde K_v\) from \(I_v\) to distributions on1005\(X_v\). At a leaf \(I_v\) is a singleton and the kernels specify1006its two answer laws. Assume the child evaluations are sampled1007independently for each evaluator, with original query and1008external random context fixed. At each vertex the evaluator1009applies its kernel to the product of those laws. Write1010\(\mu_v,\widetilde\mu_v\) for the resulting output laws. For1011probability laws on a finite set let \(\|\cdot\|_{\rm TV}\)1012denote total variation distance, defined as the supremum of the1013absolute difference in probability over all events. Put1014\begin{align*}1015\delta_v1016 &= \sup_{i\in I_v}1017 \|K_v(i)-\widetilde K_v(i)\|_{\rm TV},\\1018\eta_v1019 &= \sup_{i,j\in I_v}1020 \|K_v(i)-K_v(j)\|_{\rm TV}.1021\end{align*}1022Thus \(0\leq\eta_v\leq 1\).1023 1024\begin{proposition}[Propagation of discrepancies]1025For the root \(r\), the propagated discrepancy satisfies1026\begin{equation}1027\|\mu_r-\widetilde\mu_r\|_{\rm TV}1028 \leq1029 \sum_v \delta_v1030 \prod_{w\in\operatorname{Anc}(v)} \eta_w ,1031\label{eq:ai-tv-tree}1032\end{equation}1033where \(\operatorname{Anc}(v)\) consists of strict ancestors1034and an empty product is one.1035\end{proposition}1036 1037\begin{proof}1038Let \(e_v=\|\mu_v-\widetilde\mu_v\|_{\rm TV}\) and let1039\(\alpha_v,\widetilde\alpha_v\) be the two laws on \(I_v\).1040Independent maximal couplings of the child laws give1041\(\|\alpha_v-\widetilde\alpha_v\|_{\rm TV}1042 \leq \sum_{u}e_u\), summed over children \(u\).1043Apply \(K_v\) to a maximal coupling of the two input laws.1044When the two inputs agree, couple the output samples to be1045equal. When they differ, the two conditional output laws have a1046coupling with mismatch probability at most \(\eta_v\).1047This bounds the propagated term by1048\(\eta_v\|\alpha_v-\widetilde\alpha_v\|_{\rm TV}\).1049Replacing \(K_v\) by \(\widetilde K_v\) at fixed input law adds1050at most \(\delta_v\) by the triangle inequality under mixing.1051Hence \(e_v\leq\delta_v+\eta_v\sum_u e_u\).1052Iterating this recurrence gives \eqref{eq:ai-tv-tree}.1053\end{proof}1054 1055For example, if two leaves perturb their distributions by at1056most \(0.05\) each and a common parent uses a kernel with1057\(\eta_v=0.2\), its output moves by at most \(0.02\).1058If each row of the parent kernel also changes by at most1059\(0.01\), the bound increases to \(0.03\). The estimate holds for pairwise1060comparisons after specifying a common evaluation tree and the1061kernels being compared. In a shared-child computation or a1062program that carries conversation state between branches,1063marginal laws can agree even when joint laws differ.1064For instance, two fair bits can be independent in one1065evaluator and always equal in the other. Their marginal laws1066agree, yet exclusive-or distinguishes the evaluators.1067In that setting a valid comparison has to track the joint law1068of the consumed inputs.1069 1070The bound compares output laws. A ground-truth predicate, a1071formal checker, or a task-specific notion of utility supplies1072a separate test of the answer. A one-colour grammar with a1073constant answer at every operation can be perfectly consistent1074and wrong on every question with a different correct answer.1075Thus tree consistency can diagnose certain difficulties of1076substitution while soundness still depends on the interpretation1077of the elementary steps and on the external correctness test.1078 1079\section{Scope and outlook}1080\label{sec:scope}1081 1082The application to smooth architecture evaluation concerns the degree-zero part of the monograph's setting. The passage to \(\mathbb R[P]\) ensures that the module operators are defined on real chain complexes concentrated in degree zero. It gives a model for organizing derivatives and a check on compositional implementations. The original feature operations may be nonlinear. For a piecewise smooth layer the calculation applies on regions of differentiability. A kink, such as that of ReLU at zero, requires an additional derivative convention.1083 1084There are several different meanings of tangent structure. In the smooth calculation, a tangent space \(T_x X\) consists of directions in a parameter or state space. In the Quillen-theoretic setting, the tangent \(\infty\)-category of a presentable \(\infty\)-category at an object is the stabilization of the slice. For operadic algebras under stable hypotheses, its comparison to operadic modules is a theorem about that stabilization~\cite[Corollary~1.0.5]{harpaz2019}. This supports the role of operadic modules in homotopical linearization. Applying that theorem to learning requires a model of the architecture or data in the specified category, as well as a definition of the perturbations to be measured. The present paper directly establishes the smooth derivative interpretation and computes a separate derived symmetry example.1085 1086The clearest uses of the monograph in machine learning are the interpretation of typed architecture substitutions, the module law for one-input sensitivities, and the derived analysis of symmetry relations when ordinary balancing loses homology. For AI systems whose components return stochastic outputs, agreement between two routes is an empirical question about suitably typed kernels. The consistency calculation in Section~\ref{sec:ai-applications} supplies a diagnostic with clear sampling assumptions. The bridge in both deterministic and stochastic cases comes from the marked-slot module and the precise rule for composing it.1087 1088\paragraph{Reproducibility.} The source bundle contains the full LaTeX and a standard-library Python program for the worked numerical derivative and symmetry checks in Section~\ref{sec:examples}. These checks supplement the proofs and are limited to the worked examples.1089 1090\paragraph{Authorship.} The public byline is Anonymous. A language-model assistant helped draft, typeset, check, and revise the manuscript under the submitter's direction.1091 1092\begin{thebibliography}{12}1093\small1094\setlength{\itemsep}{2pt}1095 1096\bibitem{kriz-may}1097I.~K\v{r}\'{\i}\v{z} and J.~P. May.1098\newblock \emph{Operads, algebras, modules and motives}.1099\newblock Ast\'erisque 233, Soci\'et\'e Math\'ematique de France, 1995.1100\newblock \url{https://www.numdam.org/item/AST_1995__233__1_0/}.