(1) You are a consultant and have to decide whether or not to accept a consulting contract the moment it comes in. Each day, you get at most one contract offer; this happens with probability 1),, = 0.4; with probability 1 p0 = 0.6, you get no offer that day. Each offer has two parameters: the amount of money that you will be paid and the number of days that you will have to devote to this project. During the time you are on a contract, you cannot accept any additional offers; you are expected to devote your work time entirely to that accepted contract. Offers not accepted expire immediately and are lost forever. Offers accepted are paid for, in full, right away (even if multiple days are required to full the order). The discount factor is at = 0.95 per day. The offer characteristics are shown in the following table. Note that the probabilities are conditioned on there being an offer (which probability is pa). So, your probability of getting an offer for $ 250 on a given day is 0.31:0, for example. Name Amount Days Probability 01 $ 250 1 0.3 02 $ 800 4 0.2 03 $ 1000 6 0.3 04 $ 1200 8 0.2 You are trying to maximize your discounted reward. (In class, we dealt in terms of minimizing expected cost; here, we are maximizing expected reward.) Do the following. (a) Dene the state of your system and the possible actions in each state. You should be embedding a Markov chain at the start of each day. (b) Calculate the state transition probabilities for each action. (c) Write a program to obtain the expected reward in each state, using the policy iteration approach we discussed in class and run it to obtain the optimal action in each state and the value of each state. (The value of a state will be the expected total discounted reward of being in that state from that point in time to innity.)