Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

Please provide the code to find out how many word types account for a third of all word tokens in Brown import nltk from nltk.corpus

image text in transcribed

Please provide the code to find out how many word types account for a third of all word tokens in Brown

import nltk from nltk.corpus import brown Vow to get the list of word tokens in brown, do: >>> bw = brown.words() The name bw has now been set to the list of all the word tokens in Brown. To get a data structure with all the word counts, do: >>> fd = nltk. FreqDist(bw) Having done all that you can now execute the following to find the top 10 most frequent words and their counts: >>> fd.most common (10) [(u'the', 62713), (u',',58334), (u'.', 49346), (u'of', 36080), (u'and', 27915), (u'to', 25732), (u'a', 21881), (u'in', 19536), (u'that', 10237), (u'is', 10011)] This is a Python dictionary containing the count of all the word types in Bown. To find the number of word to- kens in Brown, do: >>> len (bw) To find the number of word types in Brown, just get the length of the dictionary: >>> len(fd) To get the count of the word type 'computer' do: >>> fd['computer'] 13 So there are 13 token of the word type 'computer' in the Brown corpus. This is unigram count for computer'. Jnigram counts are just word counts. Bigram counts will be counts opf word pairs, trigram counts, counts of word riples, and so on. You will find it useful to review Chapter One of the NLTK book. t is handy to ignore case for purposes of this exercise. To do that, just lowercase all the words in Brown before making the frequency distribution. This changes the number of word types, but not the number of word tokens: >>> bw = [w.lower() for win bw] >>> len(bw) >>> fd = nltk. FreqDist(b) >>> len(fd) 49815 To hand in

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

Securing SQL Server Protecting Your Database From Attackers

Authors: Denny Cherry

2nd Edition

1597499471, 978-1597499477

More Books

Students also viewed these Databases questions

Question

8. Explain the relationship between communication and context.

Answered: 1 week ago

Question

d. How were you expected to contribute to family life?

Answered: 1 week ago