Consider the following deterministic MDP with 1-dimensional continuous states and actions and a finite task horizon:

State Space S: R

Action Space A: R

Reward Function: R(s, a, s') = −qs² − ra² where r > 0 and q ≥ 0 Deterministic Dynamics/Transition Function: s' = cs + da (i.e., the next state s' is a deterministic function of the action a and current state s)

Task Horizon: T ∈ N

Discount Factor: γ = 1 (no discount factor)

Hence, we would like to maximize a quadratic reward function that rewards small actions and staying close to the origin. In this problem, we will design an optimal agent π^∗_t and also solve for the optimal agent’s value function V^∗_t for all timesteps.

By induction, we will show that V^∗_t is quadratic. Observe that the base case t = 0 trivially holds because V^∗₀ (s) = 0 For all parts below, assume that V^∗_t (s) = −p_ts² (Inductive Hypothesis). 

a. (i) Write the equation for V^∗ _t+1(s) as a function of s, q, r, a, c, d, and pt . If your expression contains max, you do not need to simplify the max. 

(ii) Now, solve for π^∗ _t+1(s). Recall that you can find local maxima of functions by computing the first derivative and setting it to 0. 

b. Assume π^∗ _t+1 = k_t+1s for some k_t+1 ∈ R. Solve for p_t+1 in V^∗ _t+1(s) = −p_t+1s².