q$1.$ Consider the following MDP$,$ in which all of the transitions are deterministic.

States: $s0,s1,s2$

Actions: $[$a$0,$ a$1]$

Transitions: $[($s$0,$a$0,$s$0),($s$0,$a$1,$s$1),($s$1,$a$0,$s$0),($s$1,$a$1,$s$2),($s$2,$a$0,$s$2),($s$2,$a$1,$s$2)]$

Rewards: $R(s0,a0)=1,R(s0,a1)=-1,R(s1,a0)=2,R(s1,a1)=-1,R(s2,a0)=0,R(s2,a1)=4$

We have the following policy that maps states to actions:

$\frac{?}{PI}(s0)=a1$

$\frac{?}{PI}(s1)=a1$

$\frac{?}{PI}(s2)=a1$

The policy will be executed from state $s0.$ Which technique is most appropriate to calculate the reward that will be gained?

Select one:

a$.$ Policy Iteration

b$.$ Value Iteration

c$.$ Policy Evaluation q$2.$ The value of each state is initially set to $0.$

V$($s$)=0,$ for all s

Apply a single iteration of the Bellman Backup with discount factor gamma$=0\mathrm{.}5$ to update the estimated value of each state under the policy from question $1.$

What is the estimated value of state s$0?$ q$3.$ Perform a second iteration to improve the value estimates.

What is the new estimated value of state s$1?$ q$4.$ True or false: If the only difference between two MDPs is the value of the discount factor then they must have the same optimal policy. q$5.$ True or false: For an infinite$-$horizon MDP with a finite number of states and actions, and discount factor $(0$ gamma value iteration is guaranteed to converge.