Consider an infinite$-$horizon MDP M $=$S$,$ A$,$ P$,$ R$,\backslash$gamma $,$ and assume that S and A are

finite and $\backslash$gamma $<1.$ Define Q$$

to be the optimal state$-$action value Q$$

$($s$,$ a$)=$ Q$\backslash$pi $($s$,$ a$)$

where $\backslash$pi

$$

is the optimal policy. Assume we have an estimate Qe of Q$$

$,$ and Qe is bounded

by $\backslash$infty norm as follows: