A dam is used for generating electric power and for irrigation. The dam has a capacity of 4 units of water. Assume that the volume of water stored in the dam is always an integer. At the beginning of every week, a volume of water denoted by W_n flows into the dam. The stationary probability distribution of W_n, which is an independent, identically distributed, and integer random variable, is given below.

The dam has the following policy for releasing water at the beginning of every week. If the volume of water stored in the dam plus the volume flowing into the dam at the beginning of the week exceeds 2 units, then 2 units of water are released. The first unit of water released is used to generate electricity, which is sold for \($5\). The second unit released is used for irrigation, which earns \($4\). If the volume of water stored in the dam plus the volume flowing into the dam at the beginning of the week equals 2 units, then only 1 unit of water is released. The 1 unit of water released is used to generate electricity, which is sold for \($5\). No water is released for irrigation. The volume of water stored in the dam is normally never allowed to drop below 1 unit to provide a reserve in the event of a natural disaster. Hence, if the volume of water stored in the dam plus the volume flowing into the dam at the beginning of the week is less than 2 units, no water is released. If the volume of water stored in the dam plus the volume flowing into the dam at the beginning of the week exceeds 6 units, then 2 units of water are released to generate electricity and for irrigation. In addition, surplus water is released through the spillway and lost, causing flood damage at a cost of \($3\) per unit.  

The volume of water in the dam can be modeled as a Markov chain. Let the state X_n be the volume of water in the dam at the beginning of week n.

(a) Assuming that this release policy is followed, model the volume of water in the dam and the associated rewards as a four-state recurrent MCR. Construct the transition probability matrix and the reward vector.

(b) Execute value iteration to fi nd the vector of expected total rewards that will be earned in every state after 3 weeks. Assume zero terminal rewards at the end of week 3. 

(c) Find the expected average reward, or gain.

(d) Find the vector of expected total discounted rewards using a discount factor of α = 0.9.  

Transcribed Image Text:

Volume W of water flowing into dam at start of week n, W = k 012 3 Probability, P(W, k) 0.2 0.3 0.4 0.1 =