Problem $1.(50$pt$)$ Given a Markov stationary policy $\backslash$pi $,$ consider the policy evaluation problem to compute v$^\backslash$pi $.$ For example, we can apply the temporal difference $($TD$)$ learning algorithm given by v$\_$t$+1($s$)=$v$\_$t$($s$)+\backslash$alpha $\backslash$delta $\_$t$($s$)\_\{$s$\_$t$=$s$\},$ where $\backslash$delta $\_$t:$=$r$\_$t$+\backslash$gamma v$\_$t$($s$\_$t$+1)-$v$\_$t$($s$\_$t$)$ is known as TD error. Alternatively, we can apply the n$-$step TD learning algorithm given by v$\_$t$+1($s$)=$v$\_$t$($s$)+\backslash$alpha $($G$\_$t$^($n$)-$v$\_$t$($s$))\_\{$s$\_$t$=$s$\},$ where G$\_$t$^($n$)$:$=$r$\_$t$+\backslash$gamma r$\_$t$+1+...+\backslash$gamma $^$n$-1$ r$\_$t$+$n$-1+\backslash$gamma $^$n v$\_$t$^\backslash$pi $($s$\_$t$+$n$)$ for n$=1,2,...$ Note that $\backslash$delta $\_$t$=$ G$\_$t$^(1)-$v$\_$t$($s$\_$t$).$ The n$-$step TD algorithms for n$<\backslash$infty use bootstrapping. Therefore, they use biased estimate of v$^\backslash$pi $.$ On the other hand, as n $->\backslash$infty $,$ the n$-$step TD algorithm becomes a Monte Carlo method, where we use an unbiased estimate of v$^\backslash$pi $.$ However, these approaches delay the update for n stages and we update the value function estimate only for the current state. As an intermediate step to address these challenges, we first introduce the $\backslash$lambda $-$return algorithm given by v$\_$t$+1($s$)=$v$\_$t$($s$)+\backslash$alpha $($G$\_$t$^\backslash$lambda $-$v$\_$t$($s$))\_\{$s$\_$t$=$s$\},$ where given $\backslash$lambda in $[0,1],$ we define G$\_$t$^\backslash$lambda :$=(1-\backslash$lambda $)\_$n$=1^\backslash$infty $\backslash$lambda $^$n$-1$ G$\_$t$^($n$)$ taking a weighted average of G$\_$t$^($n$),$s$.($a$)$ By the definition of G$\_$t$^($n$),$ we can show that G$\_$t$^($n$)=$r$\_$t$+\backslash$gamma G$\_$t$+1^($n$-1).$ Derive an analogous recursive relationship for G$\_$t$^\backslash$lambda and G$\_$t$+1^\backslash$lambda $.($b$)$ Show that the term G$\_$t$^\backslash$lambda $-$v$\_$t$($s$)$ in the $\backslash$lambda $-$return update can be written as the sum of TD errors. The TD algorithm, Monte Carlo method and $\backslash$lambda $-$return algorithm looks forward to approximate v$^\backslash$pi $.$ Alternatively, we can look backward via the eligibility trace method. TheTD$(\backslash$lambda $)$