Independent scholarship · Open accessThe monthly digest

Indexed text attachment

entropy-sources-and-channels-in-higher-categories.tex

Source for Entropy, Sources, and Channels in Higher Categories

View paper · Download original file

Lines 181–280 of 4249. Source text is displayed literally.

181\vspace{0.5in}182\noindent{\large \bookauthor}\par183\vspace{4pt}184\noindent{\small 2026}185\end{titlepage}186 187\thispagestyle{empty}188\vspace*{\fill}189{\small190\noindent\textbf{About this text.}191This monograph is a higher-categorical treatment of the territory mapped by A.~I.~Khinchin's192\emph{Mathematical Foundations of Information Theory} (Dover, 1957; the English translation, by R.~A.~Silverman and193M.~D.~Friedman, of two papers of 1953 and 1956): the entropy of finite schemes and its uniqueness, stationary194and ergodic sources, McMillan's theorem, channels with memory, Feinstein's lemma, and Shannon's coding195theorems. It is an independent work. It reproduces none of that book's text or proofs verbatim; it is not196affiliated with or endorsed by its publisher. \Cref{app:concordance} gives a concordance so that the two can197be read side by side.198 199\medskip200\noindent\textbf{Status of results.}201Every numbered statement carries a tag: \textsf{classical} (standard information theory, probability or202ergodic theory, attributed), \textsf{literature} (established in the categorical or homological literature,203attributed), \textsf{reformulation} (a known result restated in a different language, with the restatement204proved or sketched here), \textsf{observation} (an elementary remark we have not found stated in this form),205or \textsf{programme} (a conjecture or a direction, not a result). No cited theorem is claimed as new.206 207\medskip208\noindent\textbf{Authorship and assistance.}209The text was drafted and typeset with a large language model (Anthropic's Claude) under the direction of210the submitting author, who is responsible for it. Generative AI assisted with drafting, literature recall,211mathematical consistency checks, computation, and typesetting. The numerical examples and a number of finite212structural claims are recomputed by the accompanying script \texttt{verify.py} (\cref{app:computations}).213Proofs and bibliographic details have not been independently refereed; corrections are welcome.214 215\medskip216\noindent Open access. Rights remain with the author.217}218\clearpage219 220\tableofcontents221 222\chapter{Preface}223 224Khinchin's two papers of 1953 and 1956 did for Shannon's theory what Kolmogorov's \emph{Grundbegriffe} had225done for probability: they said what the objects are. A source is a stationary measure on a space of226sequences. A channel is a family of measures on output sequences indexed by input sequences. Entropy is227the function on finite schemes singled out by three properties. The coding theorem is a statement about228these and about nothing else, and it has a proof. The papers are short, they contain almost no examples,229and seventy years on they are still a good place to learn what the theorems say.230 231They were written in the mathematical language available at the time, which was that of measures on232product spaces. This book rewrites them in a language that was not available. The claim is not that233category theory proves Shannon's theorems; for the most part it does not, and the places where it does not234are marked. The claim is that three features of Khinchin's text, each of which he states in words and then235handles by computation, are structure that can now be named.236 237The first is the additivity axiom $H(AB)=H(A)+H_A(B)$. It is the condition for a function on observables to be238a 1-cocycle for the action of observables on functions by conditioning, and the uniqueness theorem computes a239cohomology group: $H^1\iso\R$, generated by entropy (\cref{ch:cocycle}). Symmetry of entropy is then240not an axiom but the commutativity of a monoid; the Tsallis entropies are the coboundaries of a deformed241action; and Shannon entropy is what survives when the deformation is switched off.242 243The second is Khinchin's remark that ``formally every code can be regarded as a channel'', together with244his construction of the compound source. Sources, codes and channels are the states and the morphisms of one245category, that of shift-equivariant Markov kernels. The compound source is a composite, and his theorem246that it is stationary is the statement that composition preserves equivariance (\cref{ch:channels}). The247same category, taken up to almost sure equality, carries a dagger. Conditional expectations are its248idempotents, and the two limits in McMillan's theorem, over longer pasts and over longer times, are limits249of idempotents (\cref{ch:martingales,ch:aep}).250 251The third is the form of the two asymptotic theorems on which everything rests. McMillan's theorem says that252a long block from an ergodic source is, up to $\varepsilon$, a uniform choice among $2^{nH}$ sequences.253Feinstein's lemma says that a long block of channel uses is, up to $\varepsilon$, an identity map on254$2^{nC}$ messages. Both are statements about approximate retractions, entropy and capacity are the255corresponding ranks, and Shannon's theorem is that retractions compose (\cref{ch:feinstein,ch:shannon}).256What the tolerance $\varepsilon$ buys can be said exactly: without it a whole spectrum of invariants is257needed, and with it the spectrum collapses to a point (\cref{ch:rates}).258 259The reader will find that the amount of genuinely higher structure varies. The first part uses260homological algebra and operads in earnest. The second and third use monoidal, enriched and dagger261categories, which are low-dimensional. \Cref{ch:coda} says where we think the language earned its place and262where it only relabelled.263 264\chapter{Introduction}265 266\section*{What is assumed}267 268We assume basic category theory (functors, natural transformations, limits, adjunctions, monoidal269categories) and measure-theoretic probability at the level of a first graduate course. Markov categories,270operads, monoid cohomology, enrichment and dagger categories are introduced as needed and collected in271\cref{app:toolkit}. No information theory is assumed; every classical notion is defined, and every classical272theorem is either proved or cited.273 274Logarithms are to base $2$ unless written $\ln$, so entropies are in bits. We write $\hb(x)=-x\log x-(1-x)\log(1-x)$275for the binary entropy function and use the convention $0\log 0=0$.276 277\section*{How the book is organized}278 279\Cref{fig:intro} shows the one picture to keep in mind. A source is a state $\mu$, a channel is a morphism280$\nu$, codes are deterministic morphisms, and using a channel means composing with it. Everything in