Information theory and measure theory

If we associate the existence of sets ${\tilde {X}}$ and ${\tilde {Y}}$ with arbitrary discrete random variables X and Y, somehow representing the information borne by X and Y, respectively, such that:

• $\mu ({\tilde {X}}\cap {\tilde {Y}})=0$ whenever X and Y are independent, and
• ${\tilde {X}}={\tilde {Y}}$ whenever X and Y are such that either one is completely determined by the other (i.e. by a bijection);

where $\mu$ is a measure over these sets, and we set:

$H(X)=\mu ({\tilde {X}}),$

$H(Y)=\mu ({\tilde {Y}}),$

$H(X,Y)=\mu ({\tilde {X}}\cup {\tilde {Y}}),$

$H(X|Y)=\mu ({\tilde {X}}\,\backslash \,{\tilde {Y}}),$

$I(X;Y)=\mu ({\tilde {X}}\cap {\tilde {Y}});$

we find that Shannon's "measure" of information content satisfies all the postulates and basic properties of a formal measure over sets. This can be a handy mnemonic device in some situations. Certain extensions to the definitions of Shannon's basic measures of information are necessary to deal with the σ-algebra generated by the sets that would be associated to three or more arbitrary random variables. (See Reza pp. 106-108 for an informal but rather complete discussion.) Namely $H(X,Y,Z,\cdots )$ needs to be defined in the obvious way as the entropy of a joint distribution, and an extended transinformation $I(X;Y;Z;\cdots )$ defined in a suitable manner (left as an exercise for the ambitious reader) so that we can set:

$H(X,Y,Z,\cdots )=\mu ({\tilde {X}}\cup {\tilde {Y}}\cup {\tilde {Z}}\cup \cdots ),$

$I(X;Y;Z;\cdots )=\mu ({\tilde {X}}\cap {\tilde {Y}}\cap {\tilde {Z}}\cap \cdots );$

in order to define the (signed) measure over the whole σ-algebra. (It is interesting to note that the mutual information of three or more random variables can be negative as well as positive: Let X and Y be two independent fair coin flips, and let Z be their exclusive or. Then $I(X;Y;Z)=-1$ bit.)

This formulation reiterates and clarifies the fundamental properties of these basic concepts of information theory.