To prove Eq. (7.25), define the following notation:

X j

i

≡ time spent in state i on the j th visit to this state;

Ni(m) ≡ number of visits to state i in the first m transitions In terms of this notation, write expressions for

(a) the amount of time during the first m transitions that the process is in state i;

(b) the proportion of time during the first m transitions that the process is in state i.

Argue that, with probability 1,

(c)

Ni (m)

j=1 X

j i

Ni(m)

→μi as m→∞

(d) Ni(m)/m→πi as m→∞.

(e) Combine parts (a), (b), (c), and

(d) to prove Eq. (7.25).