python help q $3$

Policy Iteration vs Value Iteration

In Part $3,$ you will compare the convergence time of policy iteration and value iteration. Both techniques are guaranteed to converge to the optimal policy in a finite number of steps, but we will see that the number of steps required can be considerably different.

$3.$A $-$ Create Environment

Create a $30$x$30$ instance of the FrozenPlatform environment with sp$\_$range$=[0,0\mathrm{.}2],$ a start position of $1($which is the default$),$ no holes, and with random$\_$state$=1.$ You do not need to display the environment.

$[]$

$3.$B $-$ Policy Iteration

Create an instance of the DPAgent class for the environment created in Step $3.$A$.$ Set gamma$=1$ and random$\_$state$=1.$ Run policy iteration with the default parameters.

$[]$

$3.$C $-$ Value Iteration

Create annother instance of the DPAgent class for the environment created in Step $3.$A$.$ Set gamma$=1$ and random$\_$state$=1.$ Run value iteration with the default parameters.

$[]$

$3.$D $-$ Algorithm Comparison

In the previous steps, you should have noticed that policy iteration had a considerably longer runtime than value iteration. You will now explore how these runtimes depend on environment size.

Starter code has been provided to you for this step. The code is intended to use a loop to create FrozenPlatform environments of size $2$x$2,3$x$3,4$x$4,$ and so on up to $25$x$25.$ Both policy iteration and value iteration will be applied to each environment. The time$()$ function from the time module will be used to calculate the runtime for each algorithm, storing the results in two different lists.

After the loop is complete, the cell should output the following two messages, with the blanks filled in with the appropriate values, rounded to $4$ decimal places.

$\%\%$time

# $3.$D

rng $=$ range$(\_\_\_\_\_\_,\_\_\_\_\_\_)$

pol$\_$iter$\_$times $=[]$

val$\_$iter$\_$times $=[]$

#np$.$random.seed$(1)$

for i in tqdm$($rng$)$:

temp$\_$fp $=$ FrozenPlatform$($

rows$=\_\_\_\_\_\_,$ cols$=\_\_\_\_\_\_,$ sp$\_$range$=[0\mathrm{.}1,0\mathrm{.}4],$ holes$=0,$ random$\_$state$=$i

$)$

t$0=$ time.time$()$

temp$\_$dp $=$ DPAgent$($env$=$temp$\_$fp$,$ gamma$=1,$ random$\_$state$=$i$)$

temp$\_$dp$.$policy$\_$iteration$($report$=$False$)$

delta$\_$t $=$ time.time$()-$ t$0$

$\_\_\_\_\_\_.$append$($delta$\_$t$)$

t$0=$ time.time$()$

temp$\_$dp $=$ DPAgent$($env$=$temp$\_$fp$,$ gamma$=1,$ random$\_$state$=$i$)$

temp$\_$dp$.$value$\_$iteration$($report$=$False$)$

delta$\_$t $=$ time.time$()-$ t$0$

$\_\_\_\_\_\_.$append$($delta$\_$t$)$

print$($f$\text{'}$Average time for policy iteration: $\{$np$.$mean$(\_\_\_\_\_\_)$:$\mathrm{.}4$f$\}\text{'})$

print$($f$\text{'}$Average time for value iteration: $\{$np$.$mean$(\_\_\_\_\_\_)$:$\mathrm{.}4$f$\}\text{'})$

Visualizing Results

Use Matplotlib to create a figure with two line plots on a single axis. The y$-$values for each line plot should come from the runtime lists created in the previous cell. The x$-$values should be the associated environment sizes $(2$ through $25).$ Create the figure according to the following specifications.

Set the figsize to $[6,3].$

The title should read "Runtime Comparison".

The x and y axes should be labeled "Environment Size" and "Runtime $($in seconds$)",$ respectively.

Add a legend with labels "Policy Iteration" and "Value Iteration" to explain which line corresponds to which algorithm.

Add a grid to your plot.