$7.($Noisy flatmate$)$ You receive the following letter:

Dear Friend, Some time ago, I moved in with a friend but found out that the friend had notorious

sardonic laughter. As a result, it is hardly habitable and I cannot leave the house. There is hope,

however, by actually interacting I have found that this sardonic laughter is subject to certain laws,

obscure but infallible, and that the laughter can be affected by my playing the piano or burning

incense sticks. In each minute, the laughter occurs or not, it shows a randomness. What it will do

during the ensuing minute depends, in the following exact way, on what has been happening during

the preceding minute: Whenever there is laughter, it will continue in the succeeding minute unless I

play the piano, in which case it will stop. But continuing to play the piano does not keep the house

quiet $($You know how poor at playing piano.$).$ I notice, however, that whenever I burn incense sticks

when the house is quiet and do not play the piano it remains quiet for the next minute. At this

minute of writing, the laughter is going on$.$ Please tell me what manipulations of burning incense

stick and playing piano I should make to get that house quiet, and to keep it so$.$

Sincerely,

At Wits End

$($a$)$ Formulate this problem as an MDP $($for the sake of uniformity, formulate it as a continuing

discounted problem, with $=0\mathrm{.}9.$ Let the reward be $+1$ on any transition into the silent

state, and $-1$ on any transition into the laughing state.$)$ Explicitly give the state set, action

sets, and state transition, and reward function.

$($b$)$ Starting with a simple policy of always burning incense, and not playing piano, perform a

couple of policy iterations.

$($c$)$ Finally, what is your advice to At Wits End?