Indexed text attachment
cotangent-semantics.tex
Source for Cotangent Semantics for Distributed Optimization
View paper · Download original file
Lines 321–420 of 542. Source text is displayed literally.
321\begin{tabular}{@{}lrrr r@{}}322\toprule323Workload & Train & Val. & Test & Params.\\324\midrule325Iris RBF & 90 & 30 & 30 & 75\\326Digits RBF & 1,074 & 355 & 368 & 810\\327Digits MLP & 1,074 & 355 & 368 & 2,410\\328Fashion linear & 4,000 & 1,000 & 2,000 & 500\\329\bottomrule330\end{tabular}331\end{table}332 333\paragraph{Iris RBF.}334The three-class Iris task uses the corrected UCI rows \cite{iris}. Train/validation/test indices are fixed by a stratified 60/20/20 split. The four input measurements are standardized using training statistics. The head uses eight training landmarks per class, giving 24 Gaussian radial basis features.335 336\paragraph{Digits RBF and MLP.}337The 1,797 rows of the UCI Optical Recognition of Handwritten Digits test file \cite{digits} are split into new, disjoint 60/20/20 partitions by class. Thus these accuracies are for the supplied split of that file. Inputs are 8-by-8 pixel counts divided by 16. The kernel model uses 80 training landmarks, eight per class. The neural model standardizes its 64 inputs with training statistics and learns 32 tanh units in two shards.338 339\paragraph{Fashion linear.}340We take 400 training and 100 validation images per class from the official Fashion-MNIST training set and 200 test images per class from its official test set \cite{fashion}. Nonoverlapping \(4\times4\) average pooling produces 49 features from each \(28\times28\) grayscale image. Training-only standardization follows. The head is ten-class softmax with 49 features and a bias. This is an image classification workload with a deliberately small linear head.341 342The RBF feature for landmark \(c_j\) is \(\exp(-\gamma\|x-c_j\|^2)\). The program sets \(\gamma\) to the reciprocal of the median squared distance of 512 reproducibly sampled training pairs. It standardizes each feature using training statistics. This realizes a finite kernel classifier with \(K_{\mathrm{fin}}(x,x')=\langle\phi(x),\phi(x')\rangle\). It uses regularization in feature coefficient space. Kernel feature methods provide the relevant comparison class \cite{rahimi}, while this construction uses fixed training landmarks. The archive records all split indices, landmark rows and bandwidths, together with the code that regenerates the feature shards. Table~\ref{tab:data} gives the model sizes.343 344\subsection{Budgets and selection protocol}345 346We run each workload at seeds \(11,23,37,53,71\) with the fixed splits. A configuration receives 384 forward queries of 48 examples each, or 18,432 example evaluations. GA spends eight queries on one common batch for each of 48 generations. Each other optimizer consumes 384 independently sampled batches and one forward query per batch. Thus the accounting matches forward exposure, including all GA candidate evaluations. It permits different independent draws, backward work and wall time. For all methods \(\lambda=10^{-4}\), and the same seed sets the common initial parameter vector. GA's remaining initial individuals perturb that vector. The two MLP boosting controls and frozen SGD share the same hidden-feature initialization as full SGD.347 348Each primary method has three hyperparameter choices. SGD starts at \(\eta_0\in\{.04,.12,.36\}\) and uses \(\eta_s=\eta_0/\sqrt{1+s/192}\) at zero-based step \(s\). GA uses initial mutation scale in \(\{.04,.12,.30\}\), annealed by \(1-.6s/384\) where \(s\) is the number of prior candidate evaluations. For the bias and hidden-bias entries the relative mutation scale is 1, for input weights it is the inverse square root of input width, and for output weights it is the inverse square root of total feature or hidden width. The entropy variants use \(\tau\in\{.5,1,2\}\) and the uniform control has one configuration. No test metric enters model selection. For each method and seed we select the lowest validation log loss after query 384. Intermediate validation checkpoints at 96, 192 and 288 show the course of training. The test set is evaluated once per selected configuration. Evaluation, audit and tuning overhead are included in the cluster-wide counts and excluded from the per-configuration training budgets.349 350\section{Results and replication}351 352\begin{table*}[t]353\centering\small354\caption{Held-out accuracy in percent. Entries are mean \(\pm\) sample standard deviation across five seeds on fixed splits. The uniform column uses the same weak atoms as entropy boosting.}355\label{tab:accuracy}356\begin{tabular}{@{}lrrrr@{}}357\toprule358Workload & SGD & Genetic algorithm & Entropy boosting & Uniform boosting\\359\midrule360Iris RBF & 96.67\ensuremath{\pm}0.00 & 90.00\ensuremath{\pm}2.36 & 96.00\ensuremath{\pm}1.49 & 96.67\ensuremath{\pm}0.00 \\361Digits RBF & 95.38\ensuremath{\pm}0.43 & 43.75\ensuremath{\pm}2.72 & 93.21\ensuremath{\pm}0.86 & 93.48\ensuremath{\pm}0.43 \\362Digits MLP & 96.09\ensuremath{\pm}0.68 & 26.41\ensuremath{\pm}4.46 & 88.21\ensuremath{\pm}0.87 & 87.83\ensuremath{\pm}0.40 \\363Fashion linear & 78.24\ensuremath{\pm}0.77 & 33.87\ensuremath{\pm}4.12 & 72.95\ensuremath{\pm}0.69 & 72.78\ensuremath{\pm}0.57 \\364\bottomrule365\end{tabular}366\end{table*}367 368\begin{table}[t]369\centering\small370\caption{Mean test negative log likelihood in nats per example.}371\label{tab:loss}372\setlength{\tabcolsep}{4.5pt}373\begin{tabular}{@{}lrrrr@{}}374\toprule375Workload & SGD & GA & Entropy & Uniform\\376\midrule377Iris RBF & 0.078 & 0.259 & 0.089 & 0.085 \\378Digits RBF & 0.147 & 1.646 & 0.317 & 0.314 \\379Digits MLP & 0.123 & 2.044 & 0.525 & 0.515 \\380Fashion linear & 0.621 & 1.863 & 0.893 & 0.829 \\381\bottomrule382\end{tabular}383\end{table}384 385\begin{table}[t]386\centering\small387\caption{Median training traffic in MiB at the selected configuration. The count combines both worker links in both directions.}388\label{tab:traffic}389\setlength{\tabcolsep}{4.5pt}390\begin{tabular}{@{}lrrrr@{}}391\toprule392Workload & SGD & GA & Entropy & Uniform\\393\midrule394Iris RBF & 2.28 & 1.21 & 2.38 & 2.38 \\395Digits RBF & 10.48 & 5.31 & 8.49 & 8.49 \\396Digits MLP & 19.86 & 10.00 & 13.18 & 13.18 \\397Fashion linear & 8.67 & 4.40 & 7.58 & 7.58 \\398\bottomrule399\end{tabular}400\end{table}401 402\subsection{Derivative and schedule checks}403 404Before training, each task tests a regularized objective on eight rows. For four unit directions (coordinator bias, each worker block, and all blocks), the coordinator computes central differences at radius \(10^{-5}\) with distributed forward queries. It compares each estimate with the directional contraction of the distributed cotangent. The largest absolute difference is \(\MaxDerivativeError\). Reversing the two worker dispatch and collection orders, with fixed arithmetic order for reductions, changes neither the computed batch loss nor the returned gradient in these checks. This probes the implementation at finite points and is separate from the real-arithmetic proof.405 406In the reference run each worker recorded \(\ForwardCount{}\) forward requests, \(\ReverseCount{}\) full reverse requests and \(\CorrelationCount{}\) boosting statistics requests. The reconciled application traffic was \(\TrafficTotal{}\) GiB across both links, including validation and audits. The virtual switch recorded \(\FrameCount{}\) Ethernet frames from three distinct guest MAC addresses. The second run booted fresh guests, built bit-identical guest binaries and initramfs images on the recorded toolchain, and reproduced all 860 validation checkpoints, 85 test records and four derivative audit records exactly apart from the excluded timing fields. The Ethernet frame counts differ between the runs.407 408\subsection{Accuracy, loss and cost}409 410Tables~\ref{tab:accuracy} and \ref{tab:loss} give the primary comparisons. Full SGD has the lowest mean held-out log loss on all four workloads. It has the highest mean accuracy on both digit tasks and Fashion-MNIST and ties uniform boosting on Iris. The Fashion accuracy difference between SGD and entropy boosting is \(5.29\) percentage points. For Digits MLP, frozen SGD reaches \(\FrozenAccuracy{}\%\) with mean log loss \(\FrozenLoss{}\), versus \(96.09\%\) and \(0.123\) for full SGD. That control separates changing the hidden representation from changing the optimizer within a fixed dictionary. Frozen SGD uses the full reverse protocol and suppresses hidden updates, so it does not reduce the measured traffic.411 412Entropy weighting raises mean accuracy by 0.38 points on Digits MLP and 0.17 on Fashion, and lowers it by 0.67 on Iris and 0.27 on Digits RBF. Mean log loss is higher with entropy weighting on each workload. The mean normalized weighting entropy \(H(q)/\log B\) at selected configurations ranges from \(0.926\) to \(0.990\). The accuracy gains on two tasks coexist with higher mean log loss.413 414The genetic baseline consumes the fewest application bytes and has the shortest selected-configuration median training time on each workload in this run. For Digits MLP the selected medians are 2.60 s for full SGD, 1.94 s for GA and 2.38 s for entropy boosting. These durations describe the emulated single-host deployment and exclude evaluation. Table~\ref{tab:traffic} shows that returning full gradients can be materially more expensive than returning an atom summary. In SGD the reverse round also returns a parameter block on each link. Boosting sends batch weights and returns only the best atom summary from each worker. On Iris, the compact full gradients make the two budgets similar.415 416\begin{figure*}[t]417\centering418\begin{tikzpicture}419\begin{groupplot}[group style={group size=2 by 2,horizontal sep=1.0cm,vertical sep=.83cm,xlabels at=edge bottom,xticklabels at=edge bottom},420width=.429\textwidth,height=2.85cm,scale only axis,