Let X be a random variable consisting of two outcomes X1 and X2, both occurring with positive probability p(X1) = p and p(X2) = 1 p. The (Shannon) entropy of X is defined to be H(X) = p log2 1 p + (1 p) log2 1 1 p = p log2 (p) (1 p) log2 (1 p) . Note that H(X) > 0 since p and 1p are both strictly between 0 and 1, so their reciprocals exceed 1, and hence log2 (1/p) and log2 (1/(1 p)) are both positive.

(a) (2 marks) Suppose p(X1) = 1/4 and p(X2) = 3/4. Calculate H(X).

(b) (8 marks) Prove that if H(X) is maximal, then both outcomes are equally likely. Hint: First year calculus: consider H(X) as a function of p and determine for which value of p it takes on its maximum. Then find that maximum. (c) (2 marks) What is the maximal value of H(X)?