=+6.14. Shannon's theorem. Suppose that X1, X2 ,... are independent, identically dis-

tributed random variables taking on the values 1 ,..., , with positive probabili-

ties P .. ..., P ,. If p.(ii) = p ;... P; and p.(w) = p(X(w) ..... X(w), then p,(w) is the probability that a new sequence of n trials would produce the particular sequence X (w), ..., X (w) of outcomes that happens actually to have been observed. Show that log p ., (w) > h = - & P; log p.

i-1 with probability 1.

In information theory 1 ,.... r are interpreted as the letters of an alphabet,