Entropy, Sources, and Channels in Higher Categories
This monograph is a higher-categorical treatment of the territory mapped by A. I. Khinchin's Mathematical Foundations of Information Theory (Dover, 1957; two papers of 1953 and 1956): the entropy of finite schemes and its uniqueness, stationary and ergodic sources, McMillan's theorem, channels with memory, Feinstein's lemma, and Shannon's coding theorems. It is an independent work. It reproduces none of that book's text or proofs verbatim, and a concordance (Appendix B) lets the two be read side by side.
The attached PDF has 13 chapters, four appendices and 117 references in 120 pages. Each topic is stated classically, with a proof or a reference, and then re-read in a language in which its structure is explicit. Every numbered statement carries a status tag: classical, literature, reformulation, observation or programme. Appendix D gives solutions to all 60 exercises. The sections below summarize the main re-readings.
The chain rule is a cocycle condition
Khinchin singles out $H(AB)=H(A)+H_A(B)$ as the essential property of entropy. On the category of finite probability spaces and measure-preserving maps, with $H(f)=H(p)-H(q)$ for $f:(X,p)\to(Y,q)$, it says that entropy is a functor to $([0,\infty),+)$, and $H_A(B)$ is the entropy of a projection. On a finite sample space $\Omega$ the partitions form a commutative idempotent monoid acting on functions of probability laws by conditioning,
$$(Y.f)(P)=\sum_{B\in Y}P(B)\,f(P|_B),$$
and the property is the 1-cocycle condition $f[YZ]=f[Y]+Y.f[Z]$ of Baudot and Bennequin. The book proves, in a finite and explicit form, that for $|\Omega|\ge 3$ every measurable 1-cocycle is a multiple of entropy, so $H^0\cong\mathbb{R}$ and $H^1\cong\mathbb{R}\cdot[H]$. Symmetry of entropy is not assumed; it follows from $YZ=ZY$. With weights $P(B)^\alpha$ the constants are no longer invariant, the Tsallis entropy is the coboundary of a constant, and both cohomology groups vanish. Shannon entropy is then the obstruction to deforming the invariant constant along the family of actions: with $\delta'$ the derivative of the coboundary at $\alpha=1$, one has $\delta' 1=-\ln 2\cdot H$. Over $\mathbb{F}_p$ the same pattern gives Kontsevich's truncated logarithm $\sum_{k=1}^{p-1}x^k/k$ and Leinster's entropy modulo a prime.
Sources, codes and channels are one category
Let $\mathsf{Dyn}$ have standard Borel spaces with an automorphism as objects and equivariant Markov kernels as morphisms. It is a Markov category. A stationary source is a state on $(A^{\mathbb{Z}},T)$, a stationary channel is a morphism, and a code is a deterministic morphism:
$$\begin{CD} A^{\mathbb{Z}} @>{\nu}>> B^{\mathbb{Z}} \\ @V{T}VV @VV{T}V \\ A^{\mathbb{Z}} @>{\nu}>> B^{\mathbb{Z}} \end{CD}$$
Khinchin's compound source is the composite $\omega=(\mathrm{id}\otimes\nu)\circ\mathrm{copy}\circ\mu$, and his theorem that it is stationary is a three-step diagram chase: $\nu$ is equivariant, $T$ is deterministic and so commutes with copying, and $\mu$ is invariant. A channel without anticipation is a natural transformation between towers $(A^n)_n\to(B^n)_n$ in $\mathsf{FinStoch}$, the stochastic counterpart of causal functions as morphisms in the topos of trees.
A gap in Khinchin's ergodicity theorem
Khinchin states that an ergodic source connected to a stationary channel of finite memory gives an ergodic compound source. With his definition of memory this is false: the channel $\nu_x=\tfrac12\delta_{\mathbf 0}+\tfrac12\delta_{\mathbf 1}$, which ignores its input and emits one random bit forever, has memory $0$ in his sense and never gives an ergodic compound source. Breiman, Takano and Feinstein noticed the problem and added an independence condition on the outputs. The categorical formulation shows where the hypothesis must go. The sequence space $B^{\mathbb{Z}}$ is the cofree system on $B$ for maps (which is the Curtis–Hedlund–Lyndon theorem) but not for kernels, because an infinite tensor product is not a categorical product. A stationary channel therefore has two parts, the dependence of each output block on the input and the dependence among outputs, and Khinchin's condition constrains only the first.
McMillan's theorem from two idempotents
In the dagger category of probability spaces and kernels up to almost sure equality, conditioning on a $\sigma$-algebra is a dagger idempotent, and a filtration is an increasing sequence of idempotents (Perrone and Van Belle). With $i_n=-\log\mu_n(x_0\dots x_{n-1})$ and $g_k=-\log\mu(x_0\mid x_{-k},\dots,x_{-1})$, the chain rule gives
$$i_n=\sum_{k=0}^{n-1}g_k\circ T^k .$$
McMillan's theorem in mean, $\tfrac1n i_n\to H(\mu)$ in $L^1$, then follows from two soft facts and one estimate: increasing idempotents converge to their supremum (Lévy's theorem in mean), time averages converge to the idempotent of the invariant $\sigma$-algebra (von Neumann), and $\mu\{g_k>t\}\le|A|\,2^{-t}$. Doob's and Birkhoff's almost sure theorems, on which Khinchin's thirty-page proof rests, are not needed for the statement he proves. The book also records where the method stops: almost sure convergence needs maximal inequalities, which the categorical results do not supply.
Entropy and capacity are ranks
An $(M,\varepsilon)$-code for $W$ is a pair $e:[M]\to X$, $d:Y\to[M]$ with $d\circ W\circ e$ within $\varepsilon$ of the identity. Feinstein's lemma is proved in its one-shot form, with the information density, and gives
$$C(W)=\lim_n\tfrac1n\log\max\{M:\ [M]\text{ is an }\varepsilon\text{-retract of }W^{\otimes n}\},$$
while McMillan's theorem gives the dual statement that $H(\mu)$ is the growth rate of the least $M$ such that the block law $\mu_n$ is an $\varepsilon$-retract of $[M]$. Shannon's first theorem is the composition of the two retractions. The converse is formal: any function $\Phi$ on morphisms that is monotone under composition, subadditive under $\otimes$ and Fano-continuous at identities bounds the rate, and capacity is the least such function. The book also presents polar codes, where the conservation law $I(W^-)+I(W^+)=2I(W)$ is the chain rule along an isomorphism and polarization needs only the orthogonality of martingale increments.
What the tolerance does
Without $\varepsilon$ the limits exist but are governed by families of invariants: zero-error capacity is a minimum over Strassen's asymptotic spectrum of graphs (Zuiddam), and exact compression is governed by Rényi entropies of orders in $[0,1]$. After conditioning on a set of probability $1-\varepsilon$, all Rényi rates of an ergodic source coincide. Shannon entropy is what remains when the spectrum collapses. In ergodic theory the collapse has already happened: by the theorems of Sinai and Krieger, entropy is both a subrank and a rank in the preorder of factor maps.
Status and authorship
The contribution is the unified development, the reformulations above with proofs from the cited results, the worked examples, the exercises with solutions, and a closing assessment with eight open problems. The statements tagged observation are elementary and, to the author's knowledge, not in the literature in this form: entropy as a deformation obstruction, the limit–colimit comparison for $L^r$-bounded martingales, the universal property of capacity, and the collapse of Rényi rates. No cited theorem is claimed as new.
Limitations. The genuinely higher structure is in Part I (operads, cohomology in a ringed presheaf topos); Parts II and III use monoidal, enriched and dagger categories, which are low-dimensional. Feinstein's lemma, the strong converse and the regularity theorem for the fundamental equation are given categorical statements, not categorical proofs. Proofs and bibliographic details have not been independently refereed.
The attached verify.py uses only the Python standard library and a fixed seed. Its 91 checks recompute every numerical example and the numerical answers in the solutions, and test the finite structural claims (the conditioning action and $\delta\delta=0$, the cocycle and Tsallis identities, entropy modulo $p$, data processing and submodularity, block entropies of Markov and hidden Markov sources, the greedy construction in Feinstein's lemma on random channels, capacities, polarization, the pentagon) on explicit and random instances. This is evidence about the examples, not proof of the theorems. The LaTeX source is attached as a single UTF-8 file marked for text indexing.
Generative AI (Anthropic's Claude) assisted with drafting, literature recall, mathematical consistency checks, computation and typesetting, under the direction of the submitting author, who is responsible for the text and uses the public byline "Anonymous".