Indexed text attachment
cotangent-semantics.tex
Source for Cotangent Semantics for Distributed Optimization
View paper · Download original file
Lines 1–100 of 542. Source text is displayed literally.
1\documentclass[10pt,twocolumn,letterpaper]{article}2\usepackage[T1]{fontenc}3\usepackage{lmodern}4\usepackage[letterpaper,top=.76in,bottom=.66in,left=.70in,right=.70in,columnsep=.27in]{geometry}5\usepackage{amsmath,amssymb,amsthm,mathtools}6\usepackage{booktabs,array,graphicx,xcolor}7\usepackage[font=small,labelfont=bf]{caption}8\usepackage{microtype}9\usepackage{fancyhdr}10\usepackage{tikz,pgfplots}11\pgfplotsset{compat=1.18}12\usepgfplotslibrary{groupplots}13\usepackage{ragged2e}14\usepackage[colorlinks,linkcolor=inkblue,citecolor=inkblue,urlcolor=inkblue,15 pdftitle={Cotangent Semantics for Distributed Optimization}]{hyperref}16\definecolor{inkblue}{HTML}{1C4A69}17\definecolor{graytext}{HTML}{4C5A63}18\definecolor{gaorange}{HTML}{A34925}19\definecolor{boostteal}{HTML}{13796C}20\definecolor{uniformpurple}{HTML}{76548A}21\color{black}22\setlength{\parindent}{1em}23\setlength{\parskip}{0pt}24\setlength{\headheight}{14pt}25\setlength{\textfloatsep}{13pt plus 3pt minus 3pt}26\setlength{\floatsep}{12pt plus 2pt minus 2pt}27\setlength{\intextsep}{10pt plus 2pt minus 2pt}28\setlength{\abovecaptionskip}{5pt}29\setcounter{topnumber}{3}30\renewcommand{\topfraction}{.88}31\renewcommand{\textfraction}{.08}32\renewcommand{\arraystretch}{1.15}33\pagestyle{fancy}34\fancyhf{}35\fancyhead[L]{\small\textsc{Cotangent semantics}}36\fancyhead[R]{\small\textsc{Distributed optimization}}37\fancyfoot[C]{\small\thepage}38\renewcommand{\headrulewidth}{.3pt}39\fancypagestyle{plain}{\fancyhf{}\fancyfoot[C]{\small\thepage}\renewcommand{\headrulewidth}{0pt}}40\raggedbottom41\lefthyphenmin=342\righthyphenmin=343\theoremstyle{plain}44\newtheorem{theorem}{Theorem}[section]45\newtheorem{lemma}[theorem]{Lemma}46\newtheorem{proposition}[theorem]{Proposition}47\theoremstyle{definition}48\newtheorem{definition}[theorem]{Definition}49\input{glyphtounicode}50\pdfgentounicode=151\newcommand{\R}{\mathbb R}52\newcommand{\N}{\mathbb N}53\newcommand{\id}{\mathrm{id}}54\newcommand{\Spc}{\mathcal S}55\newcommand{\curvekey}[3]{\tikz[baseline=-.45ex]{\draw[#1,#2,very thick] (0,0)--(.44,0);}\,\footnotesize #3}56\newcommand{\Graph}{\mathsf{Circ}}57\newcommand{\Sched}{\mathsf{Sched}}58\newcommand{\Lens}{\mathsf{CotLens}}59\newcommand{\Obj}{\mathcal L}60\newcommand{\KL}{\operatorname{KL}}61\newcommand{\E}{\mathbb E}62\newcommand{\softmax}{\operatorname{softmax}}63\newcommand{\MaxDerivativeError}{3.83\times 10^{-11}}64\newcommand{\FrameCount}{2,934,129}65\newcommand{\ForwardCount}{87,460}66\newcommand{\ReverseCount}{28,808}67\newcommand{\CorrelationCount}{30,720}68\newcommand{\TrafficTotal}{1.922}69\newcommand{\FrozenAccuracy}{88.91}70\newcommand{\FrozenLoss}{0.380}71\begin{document}72\twocolumn[{\begin{@twocolumnfalse}73\noindent{\color{inkblue}\small\textsc{Research manuscript}\par}74\vspace{.55em}75{\LARGE\bfseries Cotangent Semantics for Distributed Optimization\par}76\vspace{.40em}77{\large Schedule coherence and reproducible benchmarks on three QEMU guests\par}78\vspace{.55em}79{\normalsize Anonymous submission\par}80\vspace{.95em}81\noindent{\color{inkblue}\rule{\textwidth}{.55pt}}\par82\vspace{.40em}83\noindent\textbf{Abstract.}\quad84A distributed smooth program can expose its reverse dependency graph as a protocol for cotangent messages. We construct a semantics in which legal schedules of a typed acyclic circuit are related by coherent 2-cells. A symmetric monoidal reverse interpretation sends every legal schedule to the same cotangent lens, with transposed derivatives and additive cotangent fan-in. A finite-fiber base-change identity describes aggregation under reindexing, and the interpretation descends for compatible smooth families over a context site. We derive synchronous minibatch SGD from the distributed pullback and implement the forward and reverse traffic on three QEMU guests. At 18,432 training example evaluations per configuration, SGD gave the lowest mean test log loss on Iris with radial basis features, handwritten digits with radial basis features and with a tanh network, and pooled Fashion-MNIST. The comparison includes a real-coded genetic algorithm, an entropy-maximizing additive boosting rule, and uniform and frozen-feature controls. Two independently booted cluster runs reproduced all logged losses, accuracy counts, derivative checks, and application-byte counts exactly.85\par\vspace{.55em}86\noindent\textbf{Keywords.}\quad Reverse-mode differentiation, categorical semantics, distributed optimization, stochastic gradient descent, entropy regularization, reproducibility87\par\vspace{.95em}88\end{@twocolumnfalse}}]89\thispagestyle{plain}90 91\section{Introduction}92 93A distributed implementation exposes more structure than its extensional input--output map. Independent operations can finish in different orders. A feature shard can be moved to a different machine. A parameter can contribute to several outputs, so several messages must be combined before a reverse operation can execute. The semantics has to say which changes preserve the computed differential and which affect the cost of computing it. We address this question for finite smooth programs whose primal trace and vector--Jacobian products are available.94 95Reverse derivative categories provide an axiomatic setting for reverse differentiation \cite{cockett}. Parametric lenses describe how learning rules act on models \cite{cruttwell}. Adjoinable MPI studies the reversal of communication in numerical programs \cite{utke}. Our construction relates these ideas at the level of typed schedules. It gives a two-dimensional presentation of reorderings, a cotangent correctness theorem with explicit wire occurrences, and a base-change law for finite aggregation. The presentation also specifies what is preserved when a forward and reverse exchange is placed across a virtual network.96 97The experiments ask two separate questions. First, do actual cross-guest cotangents agree with independent finite differences and survive a change in scheduling? Second, what happens when full SGD, a forward-only genetic algorithm, and a boosting rule using entropy-maximizing example weights are given the same number of forward evaluations? The boosting rule adds to a finite dictionary of weak learners. Its weighting policy has a precise variational characterization, and a uniform-weight ablation isolates that policy.98 99\section{Typed circuits and their dual}100