Considering Netflix's personalised video recommendation for an individual, where Netflix

suggests a genre from the available list of genres $(\{$a$,$ b$,$ c $\})$ and observes the reward

$($satisfactory rating$)\{0,1,2,3\}$ from three individuals $\{$ p$,$ q$,$ r $\}.$ It is desired to learn the most

suitable recommendations for the individuals from the following episodes $\{$E$1...$ E$3\}$ of the trial run:

E$1$: p$,$ a$,0$ ; q$,$ c$,2$ ; p$,$ b$,0$ ; q$,$ c$,3$ ; r$,$ c$,2$; r$,$ b$,2$ ; q$,$ c$,1$;

E$2$: q$,$ b$,3$ ; r$,$ a$,2$ ; q$,$ c$,1$ ; p$,$ a$,0$ ; q$,$ a$,2$; p$,$ c$,2$ ; r$,$ b$,1$;

E$2$: r$,$ a$,2$ ; p$,$ b$,1$ ; p$,$ a$,2$ ; q$,$ c$,3$ ; q$,$ b$,3$; r$,$ a$,3$ ; q$,$ c$,0$;

Assume the initial policy to be $($p$)=$ a ; $($q$)=$ a; $($r$)=$ c; the initial Q$($s$,$ a$)$ for all $($s$,$a$)$ are $0$s;

and the initial V$($s$)$ for all s to be $0$s $[$ DRL Exam Question $4$ ; $22-$July$-2023]$

$($a$)$ Compute first visit and every visit prediction for all the states. Which of these values

converge faster to V$*($s$)?$ Explain $[3$ M$]$

$($b$)$ Generate a random episode and find out the important sampling ratio for E$2.$ Explain how

the importance sampling ratio and model dynamics $($when available$)$ are related. $[2$ M$]$

$($c$)$ Explain two approaches discussed in the class that help an agent learning policy using

monte$-$carlo methods continue exploring. $[1\mathrm{.}5$ M$]$

$($d$)$ Is it mandatory for the off$-$policy learning method to have non$-$zero probabilities for the

choice of all actions $?$ Explain. $[1\mathrm{.}0$ M$]$

Page $(2/2)$