MDPs and RL

A new golfer, Mr$.$ Roboto, is playing the Masters tournament. Mr$.$ Roboto$$s game can be

represented as a MDP with the following information:

State Space: $\{$ Tee$-$T$,$ Fairway$-$F$,$

Sand$-$S$,$ Green$-$G $\}$

Actions: $\{$ Conservative$-$C$,$

Risky$-$R $\}$ Shot

Initial State: Tee

Terminal State: Green

With a reward function $(*$ is a wildcard, or

don$$t care$)$:

s R$(,,$ s$)$

Fairway $0$

Sand $-2$

Green $3$

Transition Model

s a s$$ T $($s$,$ a$,$ s$)$

Tee Conservative Fairway $0\mathrm{.}9$

Tee Conservative Sand $0\mathrm{.}1$

Tee Risky Green $0\mathrm{.}3$

Tee Risky Sand $0\mathrm{.}7$

Fairway Conservative Green $0\mathrm{.}8$

Fairway Conservative Sand $0\mathrm{.}2$

Fairway Risky Green $0\mathrm{.}6$

Fairway Risky Sand $0\mathrm{.}4$

Sand Conservative Sand $0\mathrm{.}1$

Sand Conservative Fairway $0\mathrm{.}9$

Sand Risky Fairway $0\mathrm{.}7$

Sand Risky Green $0\mathrm{.}3$

$($a$)(3$ points$)$ Consider the policy of always taking the conservative shot. Assume $\backslash$gamma $=1.$ Perform

two Bellman updates to compute the values of this policy. Use the formula for a value of a