4. (25 points) In each week, a computer manufacturer's product either sells well or it sells poorly. If the product sells well in a given week and the computer manufacturer takes no further action, he earns 6,000 during that week and the product will still sell well in the next week with probability 0.5 (i.e., with probability 0.5, the product will sell poorly in the next week). Alternatively, the computer manufacturer can choose to advertise the product when it sells well. Advertising costs 2,000, but increases the probability that the product will sell well during the next week to 0.8. If the product sells poorly in a given week and the computer manufacturer takes no further action, he earns' -3,000, and the product will sell well in the next week with probability 0.4. Alternatively, the computer manufacturer can choose to do research to improve his product when it sells poorly. Doing research costs 2,000, but increases the probability that the product will sell well during the next week to 0.7. The computer manufacturer aims to maximize his long-term discounted earnings, using a weekly discount factor of 0.9. (a) 4 pt Model this problem as a Markov decision problem. What do you choose as states, decisions and optimal value function? (b)-4 pt Determine the direct rewards and transition probabilities for each state and decision. (c)-4 pt Formulate the corresponding optimality equations explicitly for this specific problem (i.e., not in generic form). The computer manufacturer currently advertises when the product sells well, and does research when the product sells poorly. (d)6 pt Use the policy iteration algorithm to determine whether or not the computer manufacturer's current policy is optimal. (e)-3 pt Formulate the linear program that the computer manufacturer could have used to determine the optimal policy. (f)-4 pt In the optimal solution to this linear program, which constraints are binding (i.e., have no slack), and what is the objective function value? 4. (25 points) In each week, a computer manufacturer's product either sells well or it sells poorly. If the product sells well in a given week and the computer manufacturer takes no further action, he earns 6,000 during that week and the product will still sell well in the next week with probability 0.5 (i.e., with probability 0.5, the product will sell poorly in the next week). Alternatively, the computer manufacturer can choose to advertise the product when it sells well. Advertising costs 2,000, but increases the probability that the product will sell well during the next week to 0.8. If the product sells poorly in a given week and the computer manufacturer takes no further action, he earns' -3,000, and the product will sell well in the next week with probability 0.4. Alternatively, the computer manufacturer can choose to do research to improve his product when it sells poorly. Doing research costs 2,000, but increases the probability that the product will sell well during the next week to 0.7. The computer manufacturer aims to maximize his long-term discounted earnings, using a weekly discount factor of 0.9. (a) 4 pt Model this problem as a Markov decision problem. What do you choose as states, decisions and optimal value function? (b)-4 pt Determine the direct rewards and transition probabilities for each state and decision. (c)-4 pt Formulate the corresponding optimality equations explicitly for this specific problem (i.e., not in generic form). The computer manufacturer currently advertises when the product sells well, and does research when the product sells poorly. (d)6 pt Use the policy iteration algorithm to determine whether or not the computer manufacturer's current policy is optimal. (e)-3 pt Formulate the linear program that the computer manufacturer could have used to determine the optimal policy. (f)-4 pt In the optimal solution to this linear program, which constraints are binding (i.e., have no slack), and what is the objective function value