[26] Prove that the entropy function H has the following four properties:

(a) For given n and n i=1 pi = 1, the function H(p1, p2,...,pn) takes its largest value for pi = 1/n (i = 1, 2,...,n). That is, the scheme with the most uncertainty is the one with equally likely outcomes.

(b) H(X, Y ) = H(X) + H(Y |X). That is, the uncertainty in the product of schemes x and y equals the uncertainty in scheme x plus the uncertainty of y given that x occurs.

(c) H(p1, p2,...,pn) = H(p1, p2,...,pn, 0). That is, adding the impossible event or any number of impossible events to the scheme does not change its entropy.

(d) H(p1, p2,...,pn) = 0 iff one of the numbers p1, p2,...,pn is one and all the others are zero. That is, if the result of the experiment can be predicted beforehand with complete certainty, then there is no uncertainty as to its outcome. In all other cases the entropy is positive.

Comments. Source: [C.E. Shannon, Bell System Tech. J., 27(1948), 379–

423, 623–656].