[25] Let an experiment in which the outcomes are 0 or 1 with fixed probability p for outcome 1 and 1 − p for outcome 0 be repeated n times. Such an experiment consists of a sequence of Bernoulli trials generated by a (p, 1 − p) Bernoulli process, explained at the beginning of this section.

Show that for each  > 0 the probability that the number Sn of outcomes 1 in the first n trials of a single sequence of trials satisfies n(p − ) <

Sn < n(p + ) goes to 1 as n goes to infinity.

Comments. This is J. Bernoulli’s law of large numbers [Ars Conjectandi, Basel, 1713, Part IV, Ch. 5, p. 236], the so-called weak law of large numbers. This law shows that with great likelihood in a series of n trials the proportion of successful outcomes will approximate p as n grows larger. The following interpretation of the weak law is false: “if Alice and Buck toss a perfect coin n times, then we can expect Alice to be in the lead roughly half of the time, regardless of who wins.” It can be shown that if Buck wins, then it is likely that he has been in the lead for practically the whole game. Thus, contrary to common belief, the time average of Sn (1 ≤ n ≤ m) over an individual game of length m has nothing to do with the so-called ensemble average of the different Sn’s associated with all possible games (the ensemble consisting of 2n games)
at a given moment n, which is the subject of the weak law. Source: [W.
Feller, An Introduction to Probability Theory and Its Applications, Vol.
1, Wiley, 1968].