Indexed text attachment
cotangent-semantics.tex
Source for Cotangent Semantics for Distributed Optimization
View paper · Download original file
Lines 361–460 of 542. Source text is displayed literally.
361Digits RBF & 95.38\ensuremath{\pm}0.43 & 43.75\ensuremath{\pm}2.72 & 93.21\ensuremath{\pm}0.86 & 93.48\ensuremath{\pm}0.43 \\362Digits MLP & 96.09\ensuremath{\pm}0.68 & 26.41\ensuremath{\pm}4.46 & 88.21\ensuremath{\pm}0.87 & 87.83\ensuremath{\pm}0.40 \\363Fashion linear & 78.24\ensuremath{\pm}0.77 & 33.87\ensuremath{\pm}4.12 & 72.95\ensuremath{\pm}0.69 & 72.78\ensuremath{\pm}0.57 \\364\bottomrule365\end{tabular}366\end{table*}367 368\begin{table}[t]369\centering\small370\caption{Mean test negative log likelihood in nats per example.}371\label{tab:loss}372\setlength{\tabcolsep}{4.5pt}373\begin{tabular}{@{}lrrrr@{}}374\toprule375Workload & SGD & GA & Entropy & Uniform\\376\midrule377Iris RBF & 0.078 & 0.259 & 0.089 & 0.085 \\378Digits RBF & 0.147 & 1.646 & 0.317 & 0.314 \\379Digits MLP & 0.123 & 2.044 & 0.525 & 0.515 \\380Fashion linear & 0.621 & 1.863 & 0.893 & 0.829 \\381\bottomrule382\end{tabular}383\end{table}384 385\begin{table}[t]386\centering\small387\caption{Median training traffic in MiB at the selected configuration. The count combines both worker links in both directions.}388\label{tab:traffic}389\setlength{\tabcolsep}{4.5pt}390\begin{tabular}{@{}lrrrr@{}}391\toprule392Workload & SGD & GA & Entropy & Uniform\\393\midrule394Iris RBF & 2.28 & 1.21 & 2.38 & 2.38 \\395Digits RBF & 10.48 & 5.31 & 8.49 & 8.49 \\396Digits MLP & 19.86 & 10.00 & 13.18 & 13.18 \\397Fashion linear & 8.67 & 4.40 & 7.58 & 7.58 \\398\bottomrule399\end{tabular}400\end{table}401 402\subsection{Derivative and schedule checks}403 404Before training, each task tests a regularized objective on eight rows. For four unit directions (coordinator bias, each worker block, and all blocks), the coordinator computes central differences at radius \(10^{-5}\) with distributed forward queries. It compares each estimate with the directional contraction of the distributed cotangent. The largest absolute difference is \(\MaxDerivativeError\). Reversing the two worker dispatch and collection orders, with fixed arithmetic order for reductions, changes neither the computed batch loss nor the returned gradient in these checks. This probes the implementation at finite points and is separate from the real-arithmetic proof.405 406In the reference run each worker recorded \(\ForwardCount{}\) forward requests, \(\ReverseCount{}\) full reverse requests and \(\CorrelationCount{}\) boosting statistics requests. The reconciled application traffic was \(\TrafficTotal{}\) GiB across both links, including validation and audits. The virtual switch recorded \(\FrameCount{}\) Ethernet frames from three distinct guest MAC addresses. The second run booted fresh guests, built bit-identical guest binaries and initramfs images on the recorded toolchain, and reproduced all 860 validation checkpoints, 85 test records and four derivative audit records exactly apart from the excluded timing fields. The Ethernet frame counts differ between the runs.407 408\subsection{Accuracy, loss and cost}409 410Tables~\ref{tab:accuracy} and \ref{tab:loss} give the primary comparisons. Full SGD has the lowest mean held-out log loss on all four workloads. It has the highest mean accuracy on both digit tasks and Fashion-MNIST and ties uniform boosting on Iris. The Fashion accuracy difference between SGD and entropy boosting is \(5.29\) percentage points. For Digits MLP, frozen SGD reaches \(\FrozenAccuracy{}\%\) with mean log loss \(\FrozenLoss{}\), versus \(96.09\%\) and \(0.123\) for full SGD. That control separates changing the hidden representation from changing the optimizer within a fixed dictionary. Frozen SGD uses the full reverse protocol and suppresses hidden updates, so it does not reduce the measured traffic.411 412Entropy weighting raises mean accuracy by 0.38 points on Digits MLP and 0.17 on Fashion, and lowers it by 0.67 on Iris and 0.27 on Digits RBF. Mean log loss is higher with entropy weighting on each workload. The mean normalized weighting entropy \(H(q)/\log B\) at selected configurations ranges from \(0.926\) to \(0.990\). The accuracy gains on two tasks coexist with higher mean log loss.413 414The genetic baseline consumes the fewest application bytes and has the shortest selected-configuration median training time on each workload in this run. For Digits MLP the selected medians are 2.60 s for full SGD, 1.94 s for GA and 2.38 s for entropy boosting. These durations describe the emulated single-host deployment and exclude evaluation. Table~\ref{tab:traffic} shows that returning full gradients can be materially more expensive than returning an atom summary. In SGD the reverse round also returns a parameter block on each link. Boosting sends batch weights and returns only the best atom summary from each worker. On Iris, the compact full gradients make the two budgets similar.415 416\begin{figure*}[t]417\centering418\begin{tikzpicture}419\begin{groupplot}[group style={group size=2 by 2,horizontal sep=1.0cm,vertical sep=.83cm,xlabels at=edge bottom,xticklabels at=edge bottom},420width=.429\textwidth,height=2.85cm,scale only axis,421title style={font=\small\bfseries},tick label style={font=\scriptsize},422label style={font=\footnotesize},xlabel={Forward queries},423ylabel={Validation NLL},xmin=86,xmax=395,xtick={96,192,288,384},424grid=major,grid style={gray!20},axis line style={gray!65}]425\nextgroupplot[title={Iris RBF}]426\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.15928144) (192,0.12401177) (288,0.10770155) (384,0.09312037)};427\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,0.47877783) (192,0.37839944) (288,0.33426111) (384,0.31004877)};428\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,0.14564999) (192,0.09509260) (288,0.07636548) (384,0.06614309)};429\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,0.19454720) (192,0.13487515) (288,0.10962411) (384,0.09455036)};430\nextgroupplot[title={Digits RBF},ylabel={}]431\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.14787566) (192,0.12561968) (288,0.11208309) (384,0.10090086)};432\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,2.06366710) (192,1.91597376) (288,1.74652723) (384,1.59833936)};433\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,0.64869758) (192,0.40870034) (288,0.32259549) (384,0.26248846)};434\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,0.60117870) (192,0.38101415) (288,0.29842410) (384,0.25427866)};435\nextgroupplot[title={Digits MLP}]436\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.15703193) (192,0.10624314) (288,0.08940936) (384,0.08233145)};437\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,2.30023223) (192,2.18556125) (288,2.15299041) (384,2.04114718)};438\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,0.91123377) (192,0.62583991) (288,0.51249900) (384,0.44916751)};439\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,0.89876967) (192,0.61433785) (288,0.50314124) (384,0.43958002)};440\nextgroupplot[title={Fashion linear},ylabel={}]441\addplot[inkblue,very thick,mark=*,mark size=2pt] coordinates {(96,0.66691827) (192,0.62112822) (288,0.60283335) (384,0.58878103)};442\addplot[gaorange,very thick,mark=square*,mark size=2pt] coordinates {(96,2.24039725) (192,2.11511561) (288,1.99523192) (384,1.85760126)};443\addplot[boostteal,very thick,mark=triangle*,mark size=2pt] coordinates {(96,1.12311189) (192,0.96246740) (288,0.92064677) (384,0.87255610)};444\addplot[uniformpurple,very thick,mark=diamond*,mark size=2pt,dashed] coordinates {(96,1.06591535) (192,0.90476077) (288,0.83985042) (384,0.79850268)};445\end{groupplot}446\end{tikzpicture}\par\smallskip447\begin{tabular}{@{}llll@{}}448\curvekey{inkblue}{solid}{SGD} & \curvekey{gaorange}{solid}{Genetic algorithm} &449\curvekey{boostteal}{solid}{Entropy boosting} & \curvekey{uniformpurple}{dashed}{Uniform boosting}\\450\end{tabular}451\caption{Mean validation negative log likelihood across five seeds at four query checkpoints, using the configuration selected by final validation loss for each seed. Each panel uses its own vertical scale. The curves are retrospective summaries of the selected configurations.}452\label{fig:curves}453\end{figure*}454 455Figure~\ref{fig:curves} shows the validation trajectories in forward queries. The fixed budget captures a regime in which GA has only 48 population updates. The best GA mutation scale and SGD learning rate often lie at the upper grid boundary, so the comparison describes these particular grids. It does not estimate the best attainable score for any algorithm. The five seeds reuse one split, so their spread measures sensitivity to initialization and sampling on that split. Iris has only 30 test examples. Its observed ties and one-example changes should be read at that resolution. The Fashion workload uses pooled pixels and a subset. It does not estimate accuracy for the standard full-resolution benchmark or a modern convolutional model.456 457\section{Scope and implications}458 459A reverse interface specifies more than the presence of derivatives. It specifies the cotangent space on each port, the snapshot on which the pullback depends, and the aggregation of all uses of a wire. A synchronous implementation can therefore change its legal schedule or placement while preserving the semantic update, provided it preserves those data. A cost analysis must still retain the trace of transmissions and placement. The same lens admits different deployments with different message traffic and runtime.460