Question 2. Consider an MDP with 3 states, A. B and C; and 2 actions Clockwise and Counterclockwise. We do not know the transition function or the reward function for the MDP, but instead, we are given samples of what an agent experiences when it interacts with the environment (although, we do know that we do not remain in the same state after taking an action). In this problem, we will first estimate the model (the transition function and the reward function), and then use the estimated model to find the optimal actions. To find the optimal actions, model-based RL proceeds by computing the optimal V or Q value function with respect to the estimated T and R. This could be done with any of value iteration, policy iteration, or Q-value iteration, we will go with Q value iteration in this exercise. Consider the following episodes that the agent encountered: sr B0.0 B B10.0 | | B B -2.0 B -2.0 B -2.0 B -2.0 B -2.0 A Counterclockwise C1.0 B Counterclockwise A 1.0 C Counterclockwise B-2.0 A Counterclockwise C1.0B Counterclockwise A 1.0C Counterclockwise A 0.0O A Counterclockwise B 0.0B Counterclockwise C 0.0CCounterclockwise A 0.0 A Counterclockwise B 0.0B Counterclockwise A 1.0 C Counterclockwise B-2.0 A Counterclockwise C1.0B Counterclockwise A 1.0C Counterclockwise A 0.0O Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise A -9.0C C 0.0C A 9.0 C c0.0 C A -9.0C Clockwise Clockwise Clockwise Clockwise Clockwise C -2.0B B10.0 | | B Question 2. Consider an MDP with 3 states, A. B and C; and 2 actions Clockwise and Counterclockwise. We do not know the transition function or the reward function for the MDP, but instead, we are given samples of what an agent experiences when it interacts with the environment (although, we do know that we do not remain in the same state after taking an action). In this problem, we will first estimate the model (the transition function and the reward function), and then use the estimated model to find the optimal actions. To find the optimal actions, model-based RL proceeds by computing the optimal V or Q value function with respect to the estimated T and R. This could be done with any of value iteration, policy iteration, or Q-value iteration, we will go with Q value iteration in this exercise. Consider the following episodes that the agent encountered: sr B0.0 B B10.0 | | B B -2.0 B -2.0 B -2.0 B -2.0 B -2.0 A Counterclockwise C1.0 B Counterclockwise A 1.0 C Counterclockwise B-2.0 A Counterclockwise C1.0B Counterclockwise A 1.0C Counterclockwise A 0.0O A Counterclockwise B 0.0B Counterclockwise C 0.0CCounterclockwise A 0.0 A Counterclockwise B 0.0B Counterclockwise A 1.0 C Counterclockwise B-2.0 A Counterclockwise C1.0B Counterclockwise A 1.0C Counterclockwise A 0.0O Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise Clockwise A -9.0C C 0.0C A 9.0 C c0.0 C A -9.0C Clockwise Clockwise Clockwise Clockwise Clockwise C -2.0B B10.0 | | B
