Mindy is playing a gambling game of 3 rounds. She will start with 4 chips, and she can wager any number of chips she has on hand at each round. With probability 0.45, she will win the bet and receive a number of additional chips equal to her wager. Otherwise, with probability 0.55 she will lose all the chips wagered, and of course, the game is over if she runs out of chips.

Mindy wants to choose the betting strategy that will lead to the highest expected number of chips at the end of the 4 rounds.

(a) Formulate Mindy’s task as a MDP with multiple states and stages, including identifying all the elements of definition 9.44 .

(b) Sketch a digraph depicting your model of part (a). You need not insert all parameter details, but do show exemplars of all transition arcs with the decision to which they are attached, the reward they would realize, and the associated probability.

(c) Form the functional equations for all states and stages.

(d) Establish that the optimal (expected value) bet at every state is simply to wager just 1 chip, which produces the expected value overall of 3.70 chips.