Information Entropy

Problem 3. Networking in English Letters and bits

Alice transfers information in English alphabets to Bob. There are 26 distinct alphabet letters; only alphabets are treated and capitalization is ignored. Part a uses the letters so that they are equally frequent while Parts d-f use the letters for English language.

a. Suppose Alice and Bob use the English letters so that they are uniformly distributed, i.e., each letter occurs equally frequently. What is the information entropy of a letter in this case?

b. Alice transforms each letter to bits and transfer the corresponding bits to Bob for each letter. Assuming that Alice cannot send a fraction of bits and she wants to be efficient (the less number of bits the better), how many bits are needed per letter?

c. Alice takes a ten-letter sequence and transforms the sequence to bits, as opposed to transforming them letter-by-letter (as in Part a). Assuming that Alice sends many letters and she wants to be efficient (the less number of bits the better), how many bits are needed per letter?

d. Alice and Bob use the letters for English language. The information entropy for one letter yields the information entropy of X bits. Compare X to the answers in Part a and Part b (it is not required to actually solve for X). Explain why it is bigger, smaller, or the same.

e. Given X from Part d and that Alice and Bob both use English language, Alice appends a 1-Byte header for every ten letters where the Byte header is used to indicate the sender (i.e., each sender has one unique Byte). If Alice is the only sender in the network, what is the information entropy for sending ten letters (including the header)?

f. Now there are 16 senders (Alice and 15 others) in the network. Answer the same question in Part e. Assume that the sixteen senders are equally active in sending letters and that their networking timing is random.