$(20$ points$)$ Biopharmaceuticals are drugs produced in a biomanufacturing process, usually

using mammalian cells. The production process of the active pharmaceutical ingredient

Page $2$

$($API$)$ consists of an upstream stage, in which the cells are fermented in batches to produce

the API, and the downstream process which captures and purifies the API after batch

harvest.

We focus on the upstream stage where each batch of a particular product is fermented for

approximately $10-14$ days and a new batch is started once the previous was harvested. Note

that a batch can only be harvested in the morning, i$.$e$.,$ the decision to harvest can only

be made once a day. The API formation during a batch fermentation follows a sigmoidal

curve with slower formation rates at the beginning and at the end of the process. The

uncertain product formation rate, i$.$e$.,$ the quantity of product added during the next day

of fermentation, follows a Normal distribution $N((s),(s))$ with the mean formation rate

$(s)$ and the standard deviation $(s)$ depending on the current product concentration.

The product concentration is $x_{\text{m}\text{i}\text{n}}=10\frac{mg}{L}$ at the beginning, as the batch is being trans$-$

ferred from a previous fermentation stage. The maximum product concentration that can

be achieved is $x_{\text{m}\text{a}\text{x}}=2150\frac{mg}{L}.$ The volume of the bioreactor is assumed to be constant at

$V_B=15,000L.$ We consider fermentation cost per day of $C_F=-208,$ fixed batch harvest

cost of $C_H=-3508$ and a revenue of $C_R=0\mathrm{.}0001\frac{s}{mg}.$ The operational question arising

is the optimal batch harvest time point assuming that a harvested batch is immediately

replaced with a new batch.

$($a$)(8$ points$)$ The state space needs to represent the current product concentration in the

bioreactor. Hence, it can be any value in the interval $[x_{\text{m}\text{i}\text{n}},x_{\text{m}\text{a}\text{x}}].$ What is the problem

with such a state space if we want to solve the problem using dynamic programming,

e$.$g$.,$ policy iteration? Briefly explain how this problem could be addressed and which

additional modelling parameter needs to be introduced. Is the solution we obtain from

the policy iteration algorithm still optimal? Name a class of solution algorithms that

would not have a problem with this kind of state space.

$($b$)(12$ points$)$ Define the state space, action space and reward function of an MDP model

for this problem using formal mathematical notation and briefly explaining it$.$

$(10$ points$)$ Short quiz questions

$($a$)(2$ points$)$ In online scheduling, a production plan is created for the planning horizon

considering future uncertainties. As the outcomes of uncertain parameter are already

considered, later rescheduling is not needed. True or False?

$($b$)(2$ points$)$ An undiscounted $(=1)$ sequential decision model acts greedily, i$.$e$.,$ chooses

the best action in the current state that maximizes the immediate reward. True or

False?

$($c$)(2$ points$)$ In the value iteration algorithm solving an MDP$,$ the value function repre$-$

sents the expected long$-$term cumulative rewards for each state. True or False?

$($d$)(2$ points$)$ Capacity consumption constraints as we know them from mathematical pro$-$

gramming can be considered in an MDP by defining state space$-$dependent action spaces

$A(s)$ if the available machine capacity is part of the state space. True or False?

$($e$)(2$ points$)$ For an MDP problem with $2,000$ states, $10$ possible actions in every state and

$10,000$ entries in the reward matrix, what is the dimension of the transition probability

matrix?