Consider the following two investment decisions: buy a certified deposit (CD) which will pay you a 10% annual interest rate, and buy some stocks which will either give you a 30% annual return (with 0.7 probability) or incur a 10% loss for you (with 0.3 probability). In terms of expected utility, you should of course buy stocks However, suppose you are conservative and cannot tolerate any loss of your principal. In this case, you have no choice but to deposit your money into a CD. Now consider this problem for a 3 years time span. You start with 1 unit of money. Each year you can choose only one way to invest your entire fund (meaning you are not allowed to diversify, like 50% CD, 50% stocks). You definitely don't want to lose any money at the end of the 3 years but you are okay with a "temporary" loss during the time (for example, it is okay for you to lose some money in the first year as long as you can make it up later). Formalize this problem as a Markov decision process and compute its optimal policy. Hints A state needs to contain information about how many money you have in this state. In other words, a state needs to encode the history: which actions you have done so far and what their outcomes are if the actions are non-deterministic. Consider the following two investment decisions: buy a certified deposit (CD) which will pay you a 10% annual interest rate, and buy some stocks which will either give you a 30% annual return (with 0.7 probability) or incur a 10% loss for you (with 0.3 probability). In terms of expected utility, you should of course buy stocks However, suppose you are conservative and cannot tolerate any loss of your principal. In this case, you have no choice but to deposit your money into a CD. Now consider this problem for a 3 years time span. You start with 1 unit of money. Each year you can choose only one way to invest your entire fund (meaning you are not allowed to diversify, like 50% CD, 50% stocks). You definitely don't want to lose any money at the end of the 3 years but you are okay with a "temporary" loss during the time (for example, it is okay for you to lose some money in the first year as long as you can make it up later). Formalize this problem as a Markov decision process and compute its optimal policy. Hints A state needs to contain information about how many money you have in this state. In other words, a state needs to encode the history: which actions you have done so far and what their outcomes are if the actions are non-deterministic