Q. 4(Discounted rewards).Let{Xn}be a Markov chain with transition matrixP. Whenever the Markov chain is in the statei, you get a rewardf(i). However, we must discount the value of rewards we will get in the future (a dollar you receive next year is worth less than a dollar you receive now, because if you receive the dollar now you can put it in your bank account and earn interest on it). Therefore, the total value of the rewards you will receive in the future is

r(i)=Enf(X),

in n=0

where0< <1is the discount factor (that is,1/is the interest rate on your bank account). We can naturally viewrandfas vectors as explained in class.

1. Show that the vectorrsatisfies the following equation:r=f+P r.
2. Suppse that{Xn}is a Markov chain on two statesD={1,2}with

f(1)=1, f(2)=2, =0.5, P11=P22=0.75, P12=P21=0.25.

Use the above formula to computer(1)andr(2).