Indexed text attachment
entropy-sources-and-channels-in-higher-categories.tex
Source for Entropy, Sources, and Channels in Higher Categories
View paper · Download original file
Lines 1049–1148 of 4249. Source text is displayed literally.
1049\[ \partial(xy)=x\,\partial(y)+\partial(x)\,y, \]1050but it is not additive, and entropy is its defect of additivity: for a distribution,1051$H(p)=\sum_i\partial(p_i)-\partial\bigl(\sum_ip_i\bigr)$, the last term being $\partial(1)=0$. The chain rule is1052then one line,1053\[ \sum_{i,j}\partial(q_ip_{ij})=\sum_iq_i\sum_j\partial(p_{ij})+\sum_i\partial(q_i)\sum_jp_{ij}. \]1054This way of seeing entropy does not need the real numbers. Let $p$ be a prime. On the integers the Fermat1055quotient $\partial_p(a)=(a-a^p)/p$ satisfies $\partial_p(ab)\equiv a\,\partial_p(b)+\partial_p(a)\,b\pmod p$ and fails to1056be additive.1057 1058\begin{definition}\label{def:modp}1059Let $\pi\in\Fp^n$ with $\sum_i\pi_i=1$, and choose integers $a_i$ representing the $\pi_i$. The1060\term{entropy of $\pi$ modulo $p$} is1061\[ H_p(\pi)=\frac1p\Bigl(1-\sum_ia_i^p\Bigr)\bmod p\ \in\Fp. \]1062\end{definition}1063 1064It is well defined because $a^p$ modulo $p^2$ depends only on $a$ modulo $p$.1065 1066\begin{theorem}\label{thm:modp}1067\Sli[\cite{Kon95,Lei21p}] $H_p$ satisfies the chain rule1068$H_p\bigl(\gamma\circ(\pi^1,\dots,\pi^n)\bigr)=H_p(\gamma)+\sum_i\gamma_iH_p(\pi^i)$, and every function on such1069``distributions'' with values in $\Fp$ satisfying the chain rule is a multiple of it; no regularity1070hypothesis is needed. For odd $p$,1071\[ H_p(x,1-x)=\sum_{k=1}^{p-1}\frac{x^k}{k}, \]1072a truncated logarithm, which satisfies \eqref{eq:FEITH} for all $x,y\in\Fp\setminus\{1\}$.1073\end{theorem}1074 1075The polynomial was found by Kontsevich, who observed that it satisfies the fundamental equation; Leinster1076identified the function it comes from and proved the uniqueness. The real and the mod $p$ theories are1077parallel. Whether there is a single cohomological statement specializing to both is, as far as we know,1078open (\cref{ch:coda}).1079 1080\section{The general setting, and higher degrees}1081 1082We have worked with all partitions of one finite set. Baudot and Bennequin, and in greater generality1083Vigneaux, consider an \term{information structure}: a poset $\Sinf$ of observables in which the product of1084two elements exists whenever they have a common refinement in $\Sinf$, with a functor of laws and a1085presheaf of functions on it. For each $X$ the observables refined by $X$ form a monoid $\Sinf_X$; the1086monoid algebras $\mathcal{A}_X=\R[\Sinf_X]$ form a presheaf of rings $\mathcal{A}$ on $\Sinf$; and the1087functions form a presheaf of $\mathcal{A}$-modules.1088 1089\begin{theorem}\label{thm:topos}1090\Sli[\cite{Vig20}] Information cohomology is $\Ext^\bullet_{\mathcal{A}}(\R,\Fun)$, computed in the abelian1091category of $\mathcal{A}$-modules in the topos of presheaves on $\Sinf$. A relative bar resolution of the1092trivial module $\R$ yields the complex of \cref{def:complex}. If every observable in $\Sinf$ can be written1093as a product of two others in a nondegenerate way, $H^1$ is a product of copies of $\R$, one for each1094connected component of $\Sinf$, each generated by a multiple of entropy.1095\end{theorem}1096 1097The generality matters when not all observables are jointly measurable, as in quantum theory or in the1098contextual situations studied by sheaf-theoretic methods. For our purposes the finite full structure is1099enough.1100 1101About degrees above one much less is known. If the conditioning action is replaced by the trivial action,1102the coboundary of entropy is $H[Y]+H[Z]-H[YZ]$, the mutual information, and the higher alternating sums are1103the multivariate mutual informations \cite{BB15}; these fit together into Hu's theorem that entropy1104extends to a signed measure on the Boolean algebra generated by the observables \cite{Hu62,Yeu08}. For the1105conditioning action itself we know of no general computation of $H^n(\Omega)$ for $n\ge2$.1106 1107\section*{Notes}1108 1109Information cohomology was introduced by Baudot and Bennequin \cite{BB15}; the topos-theoretic foundations,1110the bar resolution, and the treatment of the Tsallis case are due to Vigneaux \cite{Vig20,Vig19}, who also1111treats Gaussian laws, where differential entropy and the dimension appear as cocycles. The fundamental1112equation has a long history, for which see \cite{AD75}; its generalization of degree $\alpha$ is due to1113Dar\'oczy \cite{Dar70}, and the entropies $S_\alpha$ to Havrda and Charv\'at \cite{HC67} and Tsallis1114\cite{Tsa88}. Cathelineau \cite{Cat88} met the equation in the homology of $\mathrm{SL}_2$, which is the1115origin of the connection with polylogarithms pursued in \cite{Kon95,EVG02}. Entropy modulo a prime is1116developed by Leinster \cite{Lei21p}.1117 1118\section*{Exercises}1119 1120\begin{exercise}1121Show that $Y._\alpha(Z._\alpha f)=(YZ)._\alpha f$ for every $\alpha>0$, and that the constant function $1$ is1122invariant under the $\alpha$-action iff $\alpha=1$.1123\end{exercise}1124 1125\begin{exercise}1126Verify directly that $S_\alpha[YZ]=S_\alpha[Y]+Y._\alpha S_\alpha[Z]$. For independent $Y$ and $Z$ deduce1127$S_\alpha[YZ]=S_\alpha[Y]+S_\alpha[Z]-(\alpha-1)S_\alpha[Y]S_\alpha[Z]$.1128\end{exercise}1129 1130\begin{exercise}\label{exer:two}1131Let $|\Omega|=2$. Describe all 1-cocycles and compute $H^1(\Omega)$.1132\end{exercise}1133 1134\begin{exercise}1135Show that $h(x)=x$ satisfies \eqref{eq:FEITH}, and that $\hb$ satisfies it by computing $H(x,y,1-x-y)$ in two1136ways. Why does $h(x)=x$ not contradict \cref{thm:H1}?1137\end{exercise}1138 1139\begin{exercise}1140Prove the congruence $\partial_p(ab)\equiv a\,\partial_p(b)+\partial_p(a)\,b\pmod p$. Compute $H_5(2,4)$ from the1141definition and from the truncated logarithm. Compute $H_3(2,2)$ and $H_3(1,1,2)$, and check the chain rule for1142$(1,1,2)=(2,2)\circ\bigl((2,2),(1)\bigr)$ in $\mathbb{F}_3$.1143\end{exercise}1144%% ---- end ch/03-cocycle.tex ----1145%% ---- begin ch/04-inequalities.tex ----1146\chapter{Divergence and the Elementary Inequalities}\label{ch:inequalities}1147 1148Khinchin's second paper opens with a chapter of inequalities: two generalizations of Shannon's inequality