Consider the infinite MDP with discount factor $<mn\; id="EditorElement\_13265610">1</mn>$ illustrated in Figure $1.$ It consists of $3$ states,

and rewards are given upon taking an action from the state. From state $s_0,$ action $a_1$ has zero

immediate reward and causes a deterministic transition to state $s_1$ where there is reward $+1$ for

every time step afterwards $($regardless of action$).$ From state $s_0,$ action $a_2$ causes a deterministic

transition to state $s_2$ with immediate reward of $\frac{{}^2}{1-}$ but state $s_2$ has zero reward for every

time step afterwards $($regardless of action$).$

Figure $1$: infinite $3-$state MDP

$($a$)$ What is the total discounted return $({\displaystyle \underset{t=0}{\overset{}{}}}{}^t{r}_t)$ of taking action $a_1$ from state $s_0$ at time step

$t=0?[5pts]$

$($b$)$ What is the total discounted return $({\displaystyle \underset{t=0}{\overset{}{}}}{}^t{r}_t)$ of taking action $a_2$ from state $s_0$ at time step

$t=0?$ What is the optimal action? $[5$ pts$]$

$($c$)$ Assume we initialize value of each state to zero, $($i$.$e$.$ at iteration $n=0,$AAs:$V_n=0(s)=0).$

Show that value iteration continues to choose the sub$-$optimal action until iteration $n^{**}$ where,

$n^{**}\frac{log(1-)}{log}\frac{1}{2}log(\frac{1}{1-})\frac{1}{1-}$

Thus, value iteration has a running time that grows faster than $\frac{1}{1-}.($You just need to

show the first inequality$)[10$ pts$]$