Suppose that you observe the following $6$ episodes from a Markov Decision process

with three states $A,B,$ and $C$ with a single action for every state. We use no

discounting $($i$.$e$.=1)$ :

A$,0,$ C$,0,$ A$,0,$ C$,1$

$A,0,C,1$

$B,0,C,0$

B$,$ O$,$ C$,$ O$,$ A$,$ O$,$ C$,0$

C$,1$

C$,1$