1 A Bit of Everything (Markov chain)

1 A Bit of Everything Suppose that X0,X1, . .. is a Markov chain with nite state space S = {1,2, .. . ,n}, where n > 2, and transition matrix P. Suppose further that 1 P(1,i) = H for all states if and P(j,j1)=l forallstatesjal, with P(i, j) = O everywhere else. (a) Prove that this Markov chain is irreducible and aperiodic. (b) Suppose you start at state 1. What is the distribution of T, where T is the number of transitions until you leave state 1 for the rst time? (c) Again starting from state 1, what is the expected number of transitions until you reach state 71 for the rst time? (d) Again starting from state 1, what is the probability you reach state 72 before you reach state 2? (e) Compute the stationary distribution of this Markov chain. (f) Suppose now you start in state n. What is the expected number of transitions until you return to state n for the rst time