Tightness of Trained RNN Language Models $(10$ pts$)$

In this question, we investigate the tightness of RNN language models based on the Elman RNN$.$ We

will construct a non$-$tight RNN language model step$-$by$-$step to get a qualitative understanding of the

requirements for RNN language models to be tight and some scenarios when they can be non$-$tight.

First, recall the definition of the Elman RNN recurrence with the ReLU activation function and the

one$-$hot encoding function J$$K: