Semi$-$Gradient Update

This problem presents a brief glimpse of the problems that can arise in off$-$policy learning with function approximation, through the concepts that have been introduced so far. If

you would like a more detailed discussion on these issues, you may refer to Chapter $11.$ Let us now apply semi$-$gradient TD learning from Chapter $9$ with batch updates $($Section

$6\mathrm{.}3)$ to the following value$-$function approximation problem, based on a problem known as Baird's Counterexample:

In the above diagram, each circle is a state. The arrows represent some possible transitions between states.

The formulas shown in each state are for their values in terms of some parameters $w_i,i=0,1,$dots,$6$ that comprise the value function approximator that we wish to learn.

Consider that we see each of the transitions shown above exactly once in a batch of data.

The reward for each transition is $0,$ and the discount factor $=0\mathrm{.}95.$

We update the weights in the function approximation using semi$-$gradient $TD(0).$

Before the update, the weights are: $(1,1,1,1,1,1,5),$ with $w_6=5.$

With a learning rate $=0\mathrm{.}1,$ what will be the weights after the update?

Consider the updates happening in a batch, with the individual updates summed up and applied to the weights after computing the update for all the six transitions. Enter your

answers to $3$ decimal places.

$0$ points earned

$w_0$

$($no answer$)$

$6$

$0$ points earned

$7$

$0$ points earned

$8$

$0$ points earned

$($no answer$)$

$($no answer$)$

Results for question $9.$

$9$

$0$ points earned

Incorrect answer:

$($no answer$)$

Results for question $10.$

$10$

$0$ points earned

Incorrect answer:

$($no answer$)$

Results for question $11.$

$11$

$0$ points earned

Incorrect answer:

$($no answer$)$

Feedback

General Feedback

Using the update rule given in page $203$ of the textbook in the algorithm $($not the algorithm itself, as this is a batch update$),$ the contribution to the update from the leftmost transition is

Similarly, the contributions from the other transitions from left to right are:

And for the bottom "loop" transition, it is

$\backslash [[-0\mathrm{.}070,0,0,0,0,0,-0\mathrm{.}035]^\{$T$\}\backslash$

Adding the above updates together and adding the total update to gives the updated weights.

Results for question $12.$

$12$

$0$ points earned

After this update, what will be the new value of the state whose value is calculated as $?$

Enter your answer to $3$ decimal places.

Incorrect answer:

$($no answer$)$

Results for question $13.$

$13$

$0$ points earned

And what will be the value of the state whose value is calculated as $?$

Enter your answer to $3$ decimal places.

Incorrect answer:

$($no answer$)$