Independent scholarship · Open accessThe monthly digest

Indexed text attachment

cotangent-semantics.tex

Source for Cotangent Semantics for Distributed Optimization

View paper · Download original file

Lines 281–380 of 542. Source text is displayed literally.

281\subsection{Placement and message accounting}282 283Every run boots three independent Alpine Linux 3.23.8 initramfs guests, one virtual CPU and 192 MiB each, on QEMU 10.0.13 with TCG \cite{qemu}. The coordinator \(C\) owns labels, the global bias, parameter snapshots, the loss and optimizer state. Workers \(W_1,W_2\) own feature shards and their local forward and reverse computations. The host builds the guests and relays Ethernet frames on an isolated virtual LAN. No optimization or scoring runs on the host. Guests exchange TCP messages over emulated e1000 interfaces. Each worker records its own application-byte and operation counts, which the verifier reconciles against the coordinator's records.284 285\begin{figure}[t]286\centering287\begin{tikzpicture}[x=1cm,y=1cm,288 vm/.style={draw=inkblue,rounded corners=2pt,fill=inkblue!5,minimum width=2.05cm,289             minimum height=.60cm,align=center,font=\footnotesize},290 tag/.style={font=\scriptsize,text=graytext,fill=white,inner sep=1pt}]291 \node[vm] (one) at (-2.13,0) {\(W_1\)\\feature shard 1};292 \node[vm] (two) at (2.13,0) {\(W_2\)\\feature shard 2};293 \node[vm,minimum width=2.3cm] (core) at (0,1.54) {\(C\)\\loss and updates};294 \draw[->,thick,inkblue] (one.north east) -- (core.south west)295       node[tag,pos=.51,above left]{partial logits};296 \draw[->,thick,inkblue] (two.north west) -- (core.south east)297       node[tag,pos=.51,above right]{partial logits};298 \draw[->,thick,boostteal] (core.south) -- (one.east)299       node[tag,pos=.53,below right]{cotangent};300 \draw[->,thick,boostteal] (core.south) -- (two.west)301       node[tag,pos=.53,below left]{cotangent};302\end{tikzpicture}303\caption{Primal outputs and their reverse seeds on three guests. Requests carry batch indices and the parameter snapshot. In the reverse round, the workers return their parameter-block pullbacks.}304\label{fig:cluster}305\end{figure}306 307Each worker emits \(z_i\in\R^K\) per row. The coordinator uses \(z=(b+z_1)+z_2\). For a linear shard the worker computes \(z_i=W_i^\top x_i\) and its pullback accumulates \(x_i\bar z^\top\). For the MLP it computes \(z_i=V_i^\top\tanh(A_i x+b_i)\), with 16 hidden units per worker. Both MLP workers receive the same preprocessed inputs and own disjoint hidden/output blocks. For \(h_i=\tanh(A_i x+b_i)\), its local formulas are308\begin{align*}309 \bar V_i &= h_i \bar z^\top,\qquad310 t_i = (V_i\bar z)\odot(1-h_i^2),\\311 \bar A_i &= t_i x^\top,\qquad \bar b_i = t_i .312\end{align*}313The coordinator sums its bias cotangents over rows and adds the ridge gradient once, after collecting all parameter blocks. The tape stores hidden activations and the snapshot used for each worker. Cross-guest messages carry a 20-byte header on requests and arrays of indices or binary64 values as appropriate. Bytes reported below are application payload and framing over both links, excluding the TCP/IP and Ethernet headers.314 315\subsection{Datasets and hypotheses}316 317\begin{table}[t]318\centering\small319\caption{Fixed training, validation and test splits. Counts refer to the data loaded by the guests.}320\label{tab:data}321\begin{tabular}{@{}lrrr r@{}}322\toprule323Workload & Train & Val. & Test & Params.\\324\midrule325Iris RBF       & 90    & 30    & 30    & 75\\326Digits RBF     & 1,074 & 355   & 368   & 810\\327Digits MLP     & 1,074 & 355   & 368   & 2,410\\328Fashion linear & 4,000 & 1,000 & 2,000 & 500\\329\bottomrule330\end{tabular}331\end{table}332 333\paragraph{Iris RBF.}334The three-class Iris task uses the corrected UCI rows \cite{iris}. Train/validation/test indices are fixed by a stratified 60/20/20 split. The four input measurements are standardized using training statistics. The head uses eight training landmarks per class, giving 24 Gaussian radial basis features.335 336\paragraph{Digits RBF and MLP.}337The 1,797 rows of the UCI Optical Recognition of Handwritten Digits test file \cite{digits} are split into new, disjoint 60/20/20 partitions by class. Thus these accuracies are for the supplied split of that file. Inputs are 8-by-8 pixel counts divided by 16. The kernel model uses 80 training landmarks, eight per class. The neural model standardizes its 64 inputs with training statistics and learns 32 tanh units in two shards.338 339\paragraph{Fashion linear.}340We take 400 training and 100 validation images per class from the official Fashion-MNIST training set and 200 test images per class from its official test set \cite{fashion}. Nonoverlapping \(4\times4\) average pooling produces 49 features from each \(28\times28\) grayscale image. Training-only standardization follows. The head is ten-class softmax with 49 features and a bias. This is an image classification workload with a deliberately small linear head.341 342The RBF feature for landmark \(c_j\) is \(\exp(-\gamma\|x-c_j\|^2)\). The program sets \(\gamma\) to the reciprocal of the median squared distance of 512 reproducibly sampled training pairs. It standardizes each feature using training statistics. This realizes a finite kernel classifier with \(K_{\mathrm{fin}}(x,x')=\langle\phi(x),\phi(x')\rangle\). It uses regularization in feature coefficient space. Kernel feature methods provide the relevant comparison class \cite{rahimi}, while this construction uses fixed training landmarks. The archive records all split indices, landmark rows and bandwidths, together with the code that regenerates the feature shards. Table~\ref{tab:data} gives the model sizes.343 344\subsection{Budgets and selection protocol}345 346We run each workload at seeds \(11,23,37,53,71\) with the fixed splits. A configuration receives 384 forward queries of 48 examples each, or 18,432 example evaluations. GA spends eight queries on one common batch for each of 48 generations. Each other optimizer consumes 384 independently sampled batches and one forward query per batch. Thus the accounting matches forward exposure, including all GA candidate evaluations. It permits different independent draws, backward work and wall time. For all methods \(\lambda=10^{-4}\), and the same seed sets the common initial parameter vector. GA's remaining initial individuals perturb that vector. The two MLP boosting controls and frozen SGD share the same hidden-feature initialization as full SGD.347 348Each primary method has three hyperparameter choices. SGD starts at \(\eta_0\in\{.04,.12,.36\}\) and uses \(\eta_s=\eta_0/\sqrt{1+s/192}\) at zero-based step \(s\). GA uses initial mutation scale in \(\{.04,.12,.30\}\), annealed by \(1-.6s/384\) where \(s\) is the number of prior candidate evaluations. For the bias and hidden-bias entries the relative mutation scale is 1, for input weights it is the inverse square root of input width, and for output weights it is the inverse square root of total feature or hidden width. The entropy variants use \(\tau\in\{.5,1,2\}\) and the uniform control has one configuration. No test metric enters model selection. For each method and seed we select the lowest validation log loss after query 384. Intermediate validation checkpoints at 96, 192 and 288 show the course of training. The test set is evaluated once per selected configuration. Evaluation, audit and tuning overhead are included in the cluster-wide counts and excluded from the per-configuration training budgets.349 350\section{Results and replication}351 352\begin{table*}[t]353\centering\small354\caption{Held-out accuracy in percent. Entries are mean \(\pm\) sample standard deviation across five seeds on fixed splits. The uniform column uses the same weak atoms as entropy boosting.}355\label{tab:accuracy}356\begin{tabular}{@{}lrrrr@{}}357\toprule358Workload & SGD & Genetic algorithm & Entropy boosting & Uniform boosting\\359\midrule360Iris RBF & 96.67\ensuremath{\pm}0.00 & 90.00\ensuremath{\pm}2.36 & 96.00\ensuremath{\pm}1.49 & 96.67\ensuremath{\pm}0.00 \\361Digits RBF & 95.38\ensuremath{\pm}0.43 & 43.75\ensuremath{\pm}2.72 & 93.21\ensuremath{\pm}0.86 & 93.48\ensuremath{\pm}0.43 \\362Digits MLP & 96.09\ensuremath{\pm}0.68 & 26.41\ensuremath{\pm}4.46 & 88.21\ensuremath{\pm}0.87 & 87.83\ensuremath{\pm}0.40 \\363Fashion linear & 78.24\ensuremath{\pm}0.77 & 33.87\ensuremath{\pm}4.12 & 72.95\ensuremath{\pm}0.69 & 72.78\ensuremath{\pm}0.57 \\364\bottomrule365\end{tabular}366\end{table*}367 368\begin{table}[t]369\centering\small370\caption{Mean test negative log likelihood in nats per example.}371\label{tab:loss}372\setlength{\tabcolsep}{4.5pt}373\begin{tabular}{@{}lrrrr@{}}374\toprule375Workload & SGD & GA & Entropy & Uniform\\376\midrule377Iris RBF & 0.078 & 0.259 & 0.089 & 0.085 \\378Digits RBF & 0.147 & 1.646 & 0.317 & 0.314 \\379Digits MLP & 0.123 & 2.044 & 0.525 & 0.515 \\380Fashion linear & 0.621 & 1.863 & 0.893 & 0.829 \\