Problem 4 Consider a sequence of i.i.d. random variables {Zi , i = 1, 2, . . .}, where Zi denotes the value obtained from a roll of a fair 8-sided die (the sides of the die are labeled 1, 2, 3, . . . , 8, respectively). For each n = 1, 2, 3, . . . , define X(n) = max{Z1, Z2, . . . , Zn}, (1) i.e., X(1) = Z1, X(2) = max{Z1, Z2} and so on. Also define X(0) = 1. (a)Show that the discrete-time process {X(n), n = 0, 1, 2, . . .} is a (discrete-time) Markov chain. Determine the state-space and draw the state-transition diagram for the Markov chain. (Equivalently, provide the transition probability matrix). Is this Markov Chain irreducible? (b)Consider the 8-sided die problem description in (3). Suppose that at each time-step n, the state gets reset to 01 0 with probability 0.1, independent of any other event. Now determine the Markov chain state-space and transition probabilities. Is this irreducible?

Random Probability and statistics