Question
5 Consider the success run chain in Example 8.2.16. Suppose that the chain has been running for a while and is currently in state 10.
5 Consider the success run chain in Example 8.2.16. Suppose that the chain has been
running for a while and is currently in state 10. (a) What is the expected number of
steps until the chain is back at state 10? (b) What is the expected number of times the
chain visits state 9 before it is back at 10?
16 Consider a version of the success run chain in Example 8.2.16 where we disregard
sequences of consecutive tails, in the sense that for example T, T T, T T T, and so on,
all simply count as T. Describe this as a Markov chain and examine it in terms of
irreducibility, recurrence, and periodicity. Find the stationary distribution and compare
with Example 8.2.16. Is it the limit distribution?
17 Reversibility. Consider an ergodic Markov chain, observed at a late timepoint n. If
we look at the chain backward, we have the backward transition probability qij =
P(Xn?1 = j|Xn = i). (a) Express qij in terms of the forward transition probabilities
and the stationary distribution ?. (b)If the forward and backward transition probabilities
are equal, the chain is called reversible. Show that this occurs if and only if ?ipij =
?jpji for all states i, j (this identity is usually taken as the definition of reversibility).
(c) Show that if a probability distribution ? satisfies the equation ?ipij = ?jpji for all
i, j, then ? is stationary.
18 The intuition behind reversibility is that if we are given a sequence of consecutive states
under stationary conditions, there is no way to decide whether the states are given in
forward or backward time. Consider the ON/OFF system in Example 8.2.4; use the
definition in the previous problem to show that it is reversible and explain intuitively.
19 For which values of p is the following matrix the transition matrix of a reversible Markov
chain? Explain intuitively.
P =
0 p 1 ? p
1 ? p 0 p
p 1 ? p 0
!
20 Ehrenfest model of diffusion. Consider two containers containing a total of N gas
molecules, connected by a narrow aperture. Each time unit, one of the N molecules is
chosen at random to pass through the aperture from one container to the other. Let Xn
be the number of molecules in the first container. (a) Find the transition probabilities
for the Markov chain {Xn}. (b) Argue intuitively why the chain is reversible and why
the stationary distribution is a certain binomial distribution. Then use Problem 17 to
show that it is indeed the stationary distribution. (c) Is the stationary distribution also
the limit distribution?
21 Consider an irreducible and positive recurrent Markov chain with stationary distribution
? and let g : S ? R be a real-valued function on the state space. It can be shown that
1
n
Xn
k=1
g(Xk)
P?
X
j?S
g(j)?j
for any initial distribution, where we recall convergence in probability from Section 4.2.
This result is reminiscent of the law of large numbers, but the summands are not i.i.d.
We have mentioned that the interpretation of the stationary distribution is the long-term
proportion of time spent in each state. Show how a particular choice of the function g
above gives this interpretation (note that we do not assume aperiodicity)
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started