3 Consider a machine that may be in any one of the states 0, 1, 2, . . . . At the beginning of each month, the state of the machine is observed, and it is decided whether to replace or keep the machine. If the machine is replaced, a new state 0 machine arrives instantaneously. It costs R dollars to replace a machine. Each month that a state i machine is in operation, a maintenance cost of c(i) is incurred. If a machine is in state i at the beginning of a month, then with probability pij, the machine will begin the next month in state j. At the beginning of the first month, we own a state i0 machine.

Assuming that the interest rate is 12% per year, formulate a dynamic programming recursion that could be used to minimize the expected discounted cost incurred during the next T months. Note that if we replace a machine at the beginning of a month, we incur a maintenance cost of c(0)

during the month, and with probability p0i, we begin the next month with a state i machine.