Problem $3$: TokenizationIf your regular expression is right, all the tests defined below should pass:

init$\_$test$()$

TEST$\_$EXAMPLES

$(\text{'}$This is a test!$\text{'},[\text{'}$This$\text{'},\text{'}$is$\text{'},\text{'}$a$\text{'},$ 'test', $\text{'}!\text{'}]),$

$(\text{'}$Is this a test?$\text{'},[\text{'}$Is$\text{'},$ 'this', $\text{'}$a$\text{'},$ 'test', $\text{'}?\text{'}]),$

$("$I don't think this is a test", $[\text{'}$I$",\text{'}$do$","$n$\text{'}$t$",$ 'think', 'this', $\text{'}$is$",\text{'}$a$\text{'},$ 'test"$]),$

$"$Th$$y ph$$ c$$ c$$a t$$i l$$y $$ c$$a l$$n$",$

$[\text{'}$Th$$y$\text{'},$ 'phi', $\text{'}$ca$\text{'},\text{'}$c$$a$\text{'},\text{'}$t$$i$\text{'},\text{'}$l$\text{'},\text{'}$y$\text{'},\text{'}\text{'},\text{'}$c$$a$\text{'},\text{'}$l$$n$\text{'}]),$

$[\text{'}$Is$\text{'},\text{'}$it$\text{'},$ 'legal', $\text{'}$to$\text{'},$ 'shout', "The word 'very' is very over$-$used",

$["$The word 'very' is very over$-$used",$\text{'}$ 'word', $"\text{'}",$ 'very', $"\text{'}.",\text{'}$is$\text{'},$ 'very', 'over', $\text{'}-\text{'},$ 'used'$]),$

I don't think we'll"ve been there yet",

$[\text{'}$I$\text{'},\text{'}$do$\text{'},"$n$\text{'}$t$",$ 'think', $\text{'}$we$\text{'},"\text{'}11","\text{'}$ve$",$ 'been', 'there', 'yet'$]),$

$("$Give me $12$ apples, please", $[\text{'}$Give$\text{'},\text{'}$me$\text{'},\text{'}12\text{'},$ 'apples', $\text{'},\text{'},$ 'please'$]),$

$"$A $20\%$ tip on a $$30$ tab is $6$ dollars",

$[\text{'}$A$\text{'},\text{'}20\%\text{'},$ 'tip', $\text{'}$on$\text{'},\text{'}$a$\text{'},\text{'}$$$30",$ 'tab', $\text{'}$is$\text{'},\text{'}6\text{'},$ 'dollars'$]),$

Qpytest.mark.parametrize$(\text{'}$text$,$ toks', TEST$\_$EXANPLES$)$

def test$\_$tokenizer$($text$,$ toks$)$:

test$\_$tokenizer$($text$,$ toks$)$: assert tokenize$($text$)=$ toks

run$\_$test$()$

If your regular expression is right, all the tests defined below should pass:

init$\_$test$()$

TEST$\_$EXAMPLES

$(\text{'}$This is a test!$\text{'},[\text{'}$This$\text{'},\text{'}$is$\text{'},\text{'}$a$\text{'},$ 'test', $\text{'}!\text{'}]),$

$(\text{'}$Is this a test?$\text{'},[\text{'}$Is$\text{'},$ 'this', $\text{'}$a$\text{'},$ 'test', $\text{'}?\text{'}]),$

$("$I don't think this is a test", $["$I$\text{'},\text{'}$do$\text{'},"$n$\text{'}$t$",$ 'think', 'this', $\text{'}$is$\text{'},\text{'}$a$\text{'},$ 'test'$]),$

$("$Th$$y phi c$$ c$$a t$$i l$$y $$ c$$a l$$n$",$

$[\text{'}$Th$$y$\text{'},$ 'phi', $\text{'}$c$\text{'},\text{'}$c$$a$\text{'},\text{'}$t$$i$\text{'},\text{'}$l$\text{'},\text{'}$y$\text{'},\text{'}\text{'},\text{'}$c$$a$\text{'},\text{'}$l$$n$\text{'}]),$

$("$Is it legal to shout $\text{'}$ 'Fire!$\text{'}$ in a crowded theater?",

$[\text{'}$Is$\text{'},\text{'}$it$\text{'},$ 'legal', $\text{'}$to$\text{'},$ 'shout', $"\text{'}.",$ 'Fire', $\text{'}!\text{'},"\text{'}.\text{'},\text{'}$in$\text{'},\text{'}$a$\text{'},$ 'crowded', 'theater', $\text{'}?\text{'}]),$

$("$The word 'very' is very over$-$used",

$[\text{'}$The$\text{'},$ 'word', $"\text{'}",$ 'very', $"\text{'}",\text{'}$is$\text{'},$ 'very', 'over', $\text{'}-\text{'},$ 'used'$]),$

$("$I don't think we'll've been there yet",

$[\text{'}$I$\text{'},\text{'}$do$\text{'},"$n$\text{'}$t$",$ 'think', $\text{'}$we$\text{'},"\text{'}11","\text{'}$ve$",$ 'been', 'there", 'yet"$]),$

$("$Give me $12$ apples, please", $[\text{'}$Give$\text{'},\text{'}$me$\text{'},\text{'}12\text{'},$ 'apples', $\text{'},\text{'},$ 'please'$]),$

$("$A $20\%$ tip on a $$30$ tab is $6$ dollars",

$[\text{'}$A$\text{'},\text{'}20\%\text{'},$ 'tip', $\text{'}$on$\text{'},\text{'}$a$\text{'},\text{'}$ $${30}^{\text{'}},$ 'tab', $\text{'}$is$\text{'},\text{'}6\text{'},$ 'dollars'$]),$

$("$They$\text{'}$re going to pay us $10\%$ of $$120,000$ by Jun $4,2021",$

$[\text{'}$They$\text{'},"\text{'}$re$",$ 'going', $\text{'}$to$\text{'},$ 'pay', $\text{'}$us$\text{'},\text{'}10\%\text{'},\text{'}$of$\text{'},\text{'}$$$120,000\text{'},\text{'}$by$\text{'},$ 'Jun', $\text{'}4\text{'},\text{'},\text{'},\text{'}2021\text{'}]),$

Qpytest.mark.parametrize$(\text{'}$text$,$toks', TEST$\_$EXAMPLES$)$

def test$\_$tokenizer$($text$,$ toks$)$:

assert tokenize$($text$)==$ toks

run$\_$test$()$

Modify this expression so that it meets the following additional requirements:

the punctuation marks $\text{'}$ and $\text{'}\text{'}($left double apostrophe and right double apostrophe$)$ should be single tokens

like n$\text{'}$t$,$ the contractions $\text{'}$ve$,\text{'}11,\text{'}$re$,$ and $\text{'}$s should be seperate tokens

numbers should be separate tokens, where:

@ a number may start with $ or end with $\%$

a number may start with or contain a comma but may not end with one $($technically$,$ number tokens shouldn't start with a comma but it's okay if your transducer allows it$)$

tok$\_$patterns $=\{\}$

# insert spaces before and after punctuation

tok$\_$patterns$[\text{'}$punct$\text{'}]=$

FST$.$re$($r$"$$$^$rewrite$(\text{'}\text{'}$: $\text{'}\text{'}[!?,]\text{'}\text{'}$: $\text{'}\text{'})")$

# insert space before n$\text{'}$t

tok$\_$patterns$[\text{'}$contract$\text{'}]=$

FST$.$re$($r$"$$$^$rewrite$(\text{'}\text{'}$ : $\text{'}\text{'}\frac{n}{\frac{?}{?}}\text{'}$t$)")$

tokenizer $=$

FST$.$re$("$$punct @ $contract", tok$\_$patterns$)]$

def tokenize$($s$)$:

$s=$ list$($tokenizer$.$generate$($s$))$

if len$(s)==1$ :

return $s[].$split$()$

else:

return None

$0\mathrm{.}0s$