Please answer question $4$

$4)$ Answer why entropy is maximized in a uniform distribution. $(25$points$)$

Self$-$Information

In information theory, the entropy of a random variable is the average level of "information",

"surprise", or "uncertainty" inherent to the variable's possible outcomes.

The self$-$information is a measure of the information content associated with the outcome of a

random variable. The self$-$information of an event $x=x$ is defined as:

$I(x)=-lo{g}_{2}P(x=x)$

The choice of base for log$,$ the logarithm, varies for different applications. Base $2$ gives the unit of

bits. We can quantify the amount of uncertainty in an entire probability distribution using the

Shannon entropy.

Shannon Entropy

Given a discrete random variable $x,$ with possible outcomes $x_1,$dots,$x_n,$ which occur with probability

$P(x=x_1),$dots,$P(x=x_n)$ the entropy of $x$ is formally defined as:

$H(x)=-{\displaystyle \underset{i=1}{\overset{n}{}}}P(x=x_i)lo{g}_{2}P(x=x_i)$

where ${\displaystyle \underset{?}{\overset{?}{}}}$ denotes the sum over the variable's possible values. An equivalent definition of entropy is

the expected value of self$-$information of a variable.

Problem:

Study Shannon Entropy yourself in more detail and calculate the entropy of two random

variables $x$ and $Y,$ respectively.

Random variable $x$ is a uniform random variable with $N=8,$ i$.$e$.,xU(8).(25$ points$)$

Random variable $Y$ has the following probability mass function $(25$ points$)$:

$P(x=1)=\frac{1}{2},P(x=2)=\frac{1}{4},P(x=3)=\frac{1}{8},P(x=4)=\frac{1}{16},$

$P(x=5)=\frac{1}{64},P(x=6)=\frac{1}{64},P(x=7)=\frac{1}{64},P(x=8)=\frac{1}{64}.$

Which random variable gives a higher entropy, $x$ or $Y?(25$ points$)$

Answer why entropy is maximized in a uniform distribution. $(25$ points$)$