COMPILING TENSOR GRAPHS AND THEIR ADJOINTS FOR HETEROGENEOUS CLUSTERS
A November 2026 Survey and Higher Categorical Synthesis

Anonymous submission draft. Evidence cutoff: 3 November 2026.
Complete manuscript: output/heterogeneous-cluster-adjoints.pdf.
Length: 50 pages; approximately 19,800 words of main text; 99 references.

This synopsis accompanies the full manuscript. It is not the complete paper.

QUESTION

What can current languages, compilers, and runtimes do to compile supported tensor graphs and their reverse-mode pullbacks onto heterogeneous CPUs and GPUs connected through TCP/IP, GPU-aware transports, or remote-memory models, while reducing bespoke distributed metaprogramming?

SURVEY FINDING

The evidence supports a layered answer. Global-array systems simplify partitioning; automatic parallelizers search bounded strategy spaces; programmable training DAGs separate strategy from execution; communication-aware kernel compilers generate overlap; and actor/task runtimes simplify orchestration. No inspected system establishes a universally optimizing compiler for every effectful graph on every hardware and network mixture.

The survey emphasizes 2025-2026 developments, including Hetu v2/HSPMD, HexiScale's 2026 revision, JAX's explicit cotangent distribution, PyTorch spmd_types and experimental AutoParallel, Piper, DITRON, Syncopate, MPK, contemporary Monarch and TorchComms work, Burn's remote/collective direction, Reactant multi-host execution, and Decoupled DiLoCo. Earlier work is retained when it establishes an architectural lineage.

SEMANTIC CONTRIBUTION

The manuscript separates a logical tensor space X from physical storage D_l using linear encoding E_l and decoding Q_l with Q_l E_l = identity. If a smooth physical program P satisfies Q_m P E_l = f on a domain, then its logical pullback is E_l^* DP(E_l x)^* Q_m^*. This makes the cotangent interpretation of replication and partial contributions explicit.

It derives the transposes of linear collectives, including broadcast/sum, all-gather/reduce-scatter, permutations, repeated-index gather/scatter-add, and scaled all-reduce. It proves independent-event schedule equivalence under explicit access and effect assumptions. It also gives a smooth counterexample showing that small forward-value error alone does not bound derivative error.

Execution plans are organized through typed interfaces, event dependencies, representation changes, and justified rewrites. A sheaf construction on dependency-closed event contexts proves gluing of compatible logical histories, without claiming liveness or physical memory coherence. Indexed effects organize buffer leases, completion, epochs, and commits. Higher structures are proposed only after their required data and assumptions are stated.

ENGINEERING CONTRIBUTION

The proposed contract separates model, strategy, and deployment artifacts. It records logical types, layouts, cotangent roles, residual lifetimes, parameter epochs, collective participants, and completion rules. Generated strategies are checked independently of their authoring tool. TCP staging, RDMA ordering, bounded resources, retries, and optimizer commits remain explicit.

EVIDENCE AND LIMITS

This is a source-audited narrative survey, not a registered systematic review. Reported benchmark numbers retain their source-specific metrics and scope. Preprints, proceedings papers, engineering reports, design proposals, and living documentation are labeled distinctly. No new cluster benchmark, executable compiler implementation, formal-kernel check, or publication outcome is claimed. The proposed evaluation agenda covers graph diversity, heterogeneous hardware, ordinary networks, numerical pullback checks, recovery, and metaprogramming effort.

AUTHORSHIP AND DESTINATIONS

AI assistance is disclosed. The responsible human submitter should review the complete manuscript. The same work is intended for Higher Categorical Semantics and Agentic Meta-Journal; the package asserts no submission, acceptance, or review at either destination.
