Indexed text attachment
cotangent-semantics.tex
Source for Cotangent Semantics for Distributed Optimization
View paper · Download original file
Lines 401–500 of 542. Source text is displayed literally.
401 402\subsection{Derivative and schedule checks}403 404Before training, each task tests a regularized objective on eight rows. For four unit directions (coordinator bias, each worker block, and all blocks), the coordinator computes central differences at radius \(10^{-5}\) with distributed forward queries. It compares each estimate with the directional contraction of the distributed cotangent. The largest absolute difference is \(\MaxDerivativeError\). Reversing the two worker dispatch and collection orders, with fixed arithmetic order for reductions, changes neither the computed batch loss nor the returned gradient in these checks. This probes the implementation at finite points and is separate from the real-arithmetic proof.405 406In the reference run each worker recorded \(\ForwardCount{}\) forward requests, \(\ReverseCount{}\) full reverse requests and \(\CorrelationCount{}\) boosting statistics requests. The reconciled application traffic was \(\TrafficTotal{}\) GiB across both links, including validation and audits. The virtual switch recorded \(\FrameCount{}\) Ethernet frames from three distinct guest MAC addresses. The second run booted fresh guests, built bit-identical guest binaries and initramfs images on the recorded toolchain, and reproduced all 860 validation checkpoints, 85 test records and four derivative audit records exactly apart from the excluded timing fields. The Ethernet frame counts differ between the runs.407 408\subsection{Accuracy, loss and cost}409 410Tables~\ref{tab:accuracy} and \ref{tab:loss} give the primary comparisons. Full SGD has the lowest mean held-out log loss on all four workloads. It has the highest mean accuracy on both digit tasks and Fashion-MNIST and ties uniform boosting on Iris. The Fashion accuracy difference between SGD and entropy boosting is \(5.29\) percentage points. For Digits MLP, frozen SGD reaches \(\FrozenAccuracy{}\%\) with mean log loss \(\FrozenLoss{}\), versus \(96.09\%\) and \(0.123\) for full SGD. That control separates changing the hidden representation from changing the optimizer within a fixed dictionary. Frozen SGD uses the full reverse protocol and suppresses hidden updates, so it does not reduce the measured traffic.411 412Entropy weighting raises mean accuracy by 0.38 points on Digits MLP and 0.17 on Fashion, and lowers it by 0.67 on Iris and 0.27 on Digits RBF. Mean log loss is higher with entropy weighting on each workload. The mean normalized weighting entropy \(H(q)/\log B\) at selected configurations ranges from \(0.926\) to \(0.990\). The accuracy gains on two tasks coexist with higher mean log loss.413 414The genetic baseline consumes the fewest application bytes and has the shortest selected-configuration median training time on each workload in this run. For Digits MLP the selected medians are 2.60 s for full SGD, 1.94 s for GA and 2.38 s for entropy boosting. These durations describe the emulated single-host deployment and exclude evaluation. Table~\ref{tab:traffic} shows that returning full gradients can be materially more expensive than returning an atom summary. In SGD the reverse round also returns a parameter block on each link. Boosting sends batch weights and returns only the best atom summary from each worker. On Iris, the compact full gradients make the two budgets similar.415 416\begin{figure*}[t]417\centering418\begin{tikzpicture}419\begin{groupplot}[group style={group size=2 by 2,horizontal sep=1.0cm,vertical sep=.83cm,xlabels at=edge bottom,xticklabels at=edge bottom},420width=.429\textwidth,height=2.85cm,scale only axis,421title style={font=\small\bfseries},tick label style={font=\scriptsize},422label style={font=\footnotesize},xlabel={Forward queries},423ylabel={Validation NLL},xmin=86,xmax=395,xtick={96,192,288,384},424grid=major,grid style={gray!20},axis line style={gray!65}]425\nextgroupplot[title={Iris RBF}]426\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.15928144) (192,0.12401177) (288,0.10770155) (384,0.09312037)};427\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,0.47877783) (192,0.37839944) (288,0.33426111) (384,0.31004877)};428\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,0.14564999) (192,0.09509260) (288,0.07636548) (384,0.06614309)};429\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,0.19454720) (192,0.13487515) (288,0.10962411) (384,0.09455036)};430\nextgroupplot[title={Digits RBF},ylabel={}]431\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.14787566) (192,0.12561968) (288,0.11208309) (384,0.10090086)};432\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,2.06366710) (192,1.91597376) (288,1.74652723) (384,1.59833936)};433\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,0.64869758) (192,0.40870034) (288,0.32259549) (384,0.26248846)};434\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,0.60117870) (192,0.38101415) (288,0.29842410) (384,0.25427866)};435\nextgroupplot[title={Digits MLP}]436\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.15703193) (192,0.10624314) (288,0.08940936) (384,0.08233145)};437\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,2.30023223) (192,2.18556125) (288,2.15299041) (384,2.04114718)};438\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,0.91123377) (192,0.62583991) (288,0.51249900) (384,0.44916751)};439\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,0.89876967) (192,0.61433785) (288,0.50314124) (384,0.43958002)};440\nextgroupplot[title={Fashion linear},ylabel={}]441\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.66691827) (192,0.62112822) (288,0.60283335) (384,0.58878103)};442\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,2.24039725) (192,2.11511561) (288,1.99523192) (384,1.85760126)};443\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,1.12311189) (192,0.96246740) (288,0.92064677) (384,0.87255610)};444\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,1.06591535) (192,0.90476077) (288,0.83985042) (384,0.79850268)};445\end{groupplot}446\end{tikzpicture}\par\smallskip447\begin{tabular}{@{}llll@{}}448\curvekey{inkblue}{solid}{SGD} & \curvekey{gaorange}{solid}{Genetic algorithm} &449\curvekey{boostteal}{solid}{Entropy boosting} & \curvekey{uniformpurple}{dashed}{Uniform boosting}\\450\end{tabular}451\caption{Mean validation negative log likelihood across five seeds at four query checkpoints, using the configuration selected by final validation loss for each seed. Each panel uses its own vertical scale. The curves are retrospective summaries of the selected configurations.}452\label{fig:curves}453\end{figure*}454 455Figure~\ref{fig:curves} shows the validation trajectories in forward queries. The fixed budget captures a regime in which GA has only 48 population updates. The best GA mutation scale and SGD learning rate often lie at the upper grid boundary, so the comparison describes these particular grids. It does not estimate the best attainable score for any algorithm. The five seeds reuse one split, so their spread measures sensitivity to initialization and sampling on that split. Iris has only 30 test examples. Its observed ties and one-example changes should be read at that resolution. The Fashion workload uses pooled pixels and a subset. It does not estimate accuracy for the standard full-resolution benchmark or a modern convolutional model.456 457\section{Scope and implications}458 459A reverse interface specifies more than the presence of derivatives. It specifies the cotangent space on each port, the snapshot on which the pullback depends, and the aggregation of all uses of a wire. A synchronous implementation can therefore change its legal schedule or placement while preserving the semantic update, provided it preserves those data. A cost analysis must still retain the trace of transmissions and placement. The same lens admits different deployments with different message traffic and runtime.460 461The experiments use model sharding over one emulated LAN. There is no comparison against multiple physical hosts, decentralized consensus, stale gradient protocols, or fault-tolerant recovery. The raw transcripts are generated by the provided program, so their hashes and counter reconciliation establish repeatability and internal consistency. The rebuild and repeat-run comparison provide a direct check of the reported results. A program with data-dependent branching can use a fixed executed trace for conditional differentiation, with separate conditions needed near switching boundaries. Feedback can be unrolled to a finite acyclic program at a fixed horizon. Neither construction licenses transposing a nondifferentiable branch decision or ignoring the state needed to reverse a loop.462 463\section{Reproduction and disclosures}464 465The companion archive includes the C implementation, a Python 3 standard-library guest builder and runner, SHA-256 pins, checksum-verified prepared data shards, both runs' serial logs and manifests, the verifier, and the double-column LaTeX source. On a Linux x86-64 host, QEMU system emulation and a C compiler with static libc suffice to build and run the experiments. The archive's README gives the commands for a full cluster run, the cross-run comparison and regeneration of the manuscript tables and curves. The data shards can be regenerated from the pinned upstream datasets. The verifier treats guest identities, Ethernet frame counts and wall-clock measurements as execution evidence whose values can vary, and checks the stable numerical and application-counter fields against the reference run. No accelerator or external service is used by the guests.466 467The commissioning researcher is the responsible contributor. The manuscript uses an anonymous byline. An AI coding and writing assistant developed the manuscript and the experimental software, ran the cluster twice, checked the quantitative results and assembled the replication artifacts under the researcher's direction. The mathematical results are supported by prose proofs.468 469\begin{thebibliography}{99}470\small\RaggedRight471\bibitem{cockett} R. Cockett, G. Cruttwell, J. Gallagher, J.-S. P. Lemay,472B. MacAdam, G. Plotkin, and D. Pronk.473Reverse derivative categories.474In \emph{CSL 2020}, LIPIcs 152, article 18.475\href{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.CSL.2020.18}{doi:10.4230/LIPIcs.CSL.2020.18}.476 477\bibitem{cruttwell} G. S. H. Cruttwell, B. Gavranović, N. Ghani, P. Wilson, and F. Zanasi.478Categorical foundations of gradient-based learning.479In \emph{ESOP 2022}, LNCS 13240, pp.~1--28.480\href{https://link.springer.com/chapter/10.1007/978-3-030-99336-8_1}{doi:10.1007/978-3-030-99336-8\_1}.481 482\bibitem{utke} J. Utke, L. Hascoët, C. Hill, P. Hovland, J. Riehme, and U. Naumann.483Toward adjoinable MPI.484In \emph{PDSEC}, 2009.485\href{https://www-sop.inria.fr/tropics/papers/Utke2009TAM.html}{INRIA record}.486 487\bibitem{lurie} J. Lurie. \emph{Higher Topos Theory}.488Annals of Mathematics Studies 170, Princeton University Press, 2009.489\href{https://arxiv.org/abs/math/0608040}{arXiv:math/0608040}.490 491\bibitem{bottou} L. Bottou, F. E. Curtis, and J. Nocedal.492Optimization methods for large-scale machine learning.493\emph{SIAM Review} 60(2), pp.~223--311, 2018.494\href{https://arxiv.org/abs/1606.04838}{doi:10.1137/16M1080173}.495 496\bibitem{fritz} T. Fritz.497A synthetic approach to Markov kernels, conditional independence and theorems on sufficient statistics.498\emph{Advances in Mathematics} 370, 107239, 2020.499\href{https://arxiv.org/abs/1908.07021}{arXiv:1908.07021}.500