# Плавное введение в natural language processing (nlp)

## Final words

It’s useful to understand how TF-IDF works so that you can gain a better understanding of how machine learning algorithms function. While machine learning algorithms traditionally work better with numbers, TF-IDF algorithms help them decipher words by allocating them a numerical value or vector. This has been revolutionary for machine learning, especially in fields related to NLP such as text analysis.

In text analysis with machine learning, TF-IDF algorithms help sort data into categories, as well as extract keywords. This means that simple, monotonous tasks, like tagging support tickets or rows of feedback and inputting data can be done in seconds.

Every wondered how Google can serve up information related to your search in mere seconds? Well, now you know. Text vectorization transforms text within documents into numbers, so TF-IDF algorithms can rank articles in order of relevance.

## Term Frequency

This measures the frequency of a word in a document. This highly depends on the length of the document and the generality of word, for example a very common word such as “was” can appear multiple times in a document. but if we take two documents one which have 100 words and other which have 10,000 words. There is a high probability that the common word such as “was” can be present more in the 10,000 worded document. But we cannot say that the longer document is more important than the shorter document. For this exact reason, we perform a normalization on the frequency value. we divide the the frequency with the total number of words in the document.

Recall that we need to finally vectorize the document, when we are planning to vectorize the documents, we cannot just consider the words that are present in that particular document. If we do that, then the vector length will be different for both the documents, and it will not be feasible to compute the similarity. So, what we do is that we vectorize the documents on the vocab. vocab is the list of all possible words in the corpus.

When we are vectorizing the documents, we check for each words count. In worst case if the term doesn’t exist in the document, then that particular TF value will be 0 and in other extreme case, if all the words in the document are same, then it will be 1. The final value of the normalised TF value will be in the range of . 0, 1 inclusive.

TF is individual to each document and word, hence we can formulate TF as follows.

If we already computed the TF value and if this produces a vectorized form of the document, why not use just TF to find the relevance between documents? why do we need IDF?

Let me explain, though we calculated the TF value, still there are few problems, for example, words which are the most common words such as “is, are” will have very high values, giving those words a very high importance. But using these words to compute the relevance produces bad results. These kind of common words are called stop-words, although we will remove the stop words later in the preprocessing step, finding the importance of the word across all the documents and normalizing using that value represents the documents much better.

## Times up.

The current answer is option 3. Break it in sentences .

Why ? cuz when you break a document in multiple sentences, each sentence has multiple words which represent provide some context to sentences and these sentences as a whole provide some context to the document and then we can ask the machine questions like,

By evaluating TF-IDF or a number of “the words used in a sentence vs words used in overall document”, we understand —

1. how useful a word is to a sentence (which helps us understand the importance of a word in a sentence).
2. how useful a word is to a document (which helps us understand the important words with more frequencies in a document).
3. helps us ignore words that are misspelled (using n-gram technique) an example of which I am covering below

Imagine in a document you misspelled ‘example’ as ‘exaple’ and you forgot to go back and change it before giving it to a machine to read —

In case of BOW, both ‘example’ and ‘exaple’ would be treated as different words and given the same importance because their frequency is same.

But in case of TD-IDF because of a score of IDF, this mistake is corrected because we know example as a word is more important than exaple, so we treat it like a non useful word.

Now because of these scores our machine has a better understanding of these documents and can be asked to compare these documents, find similar documents, find opposite documents, find similarities in document and can be used by machine to recommend you what to read next, cool right?

Now, I am guessing you need a minute to go back and grasp this concept again before I tell you how to do it, ofcourse I’ll take up an example so if you’re conceptually hazy but almost clear you’ll definitelly be alright once you practise with the example.

## What is the way of finding TF-IDF of a document?

The process to find meaning of documents using TF-IDF is very similar to Bag of words,

1. Clean data / Preprocessing — Clean data (standardise data) , Normalize data( all lower case) , lemmatize data ( all words to root words ).
2. Tokenize words with frequency
3. Find TF for words
4. Find IDF for words
5. Vectorize vocab

(if you’re unfamiliar with what these are, I recommend reading the article BOW I shared on top to get a clear understanding of how to do these).

I’ll be using these techniques to cover the example below so I hope you’re familiar with them.

## Структура формулы

TF (term frequency — частота слова) — отношение числа вхождений некоторого слова к общему числу слов документа

Таким образом, оценивается важность слова ti{\displaystyle t_{i}} в пределах отдельного документа.

tf(t,d)=nt∑knk{\displaystyle \mathrm {tf} (t,d)={\frac {n_{t}}{\sum _{k}n_{k}}}} ,

где nt{\displaystyle n_{t}} есть число вхождений слова t{\displaystyle t} в документ, а в знаменателе — общее число слов в данном документе.

IDF (inverse document frequency — обратная частота документа) — инверсия частоты, с которой некоторое слово встречается в документах коллекции. Основоположником данной концепции является Карен Спарк Джонс. Учёт IDF уменьшает вес широкоупотребительных слов. Для каждого уникального слова в пределах конкретной коллекции документов существует только одно значение IDF.

idf(t,D)=log⁡|D||{di∈D∣t∈di}|{\displaystyle \mathrm {idf} (t,D)=\log {\frac {|D|}{|\{\,d_{i}\in D\mid t\in d_{i}\,\}|}}} ,

где

• |D| — число документов в коллекции;
• |{di∈D∣t∈di}|{\displaystyle |\{\,d_{i}\in D\mid t\in d_{i}\,\}|} — число документов из коллекции D{\displaystyle D}, в которых встречается t{\displaystyle t} (когда nt≠{\displaystyle n_{t}\neq 0}).

Выбор основания логарифма в формуле не имеет значения, поскольку изменение основания приводит к изменению веса каждого слова на постоянный множитель, что не влияет на соотношение весов.

Таким образом, мера TF-IDF является произведением двух сомножителей:

tf-idf⁡(t,d,D)=tf⁡(t,d)×idf⁡(t,D){\displaystyle \operatorname {tf-idf} (t,d,D)=\operatorname {tf} (t,d)\times \operatorname {idf} (t,D)}

Большой вес в TF-IDF получат слова с высокой частотой в пределах конкретного документа и с низкой частотой употреблений в других документах.

## Характеристики TF

Размеры TF-карты значительно меньше mini-SD и RC-MMS – 15х11х1 мм. Во время разработки компания SanDisk встроила чип MLC, поэтому габариты получились рекордно маленькими.

Перед покупкой карты емкостью 32 ГБ и более, убедитесь, что ваше устройство поддерживает SDXC. Если ваше устройство «потянет» 128 ГБ, то лучше выбрать именно такую емкость, а не 64 ГБ. Последняя обойдется дороже.

Раньше скорость приема у MicroSD/TF составляла всего 5,3 Мбит/с. Со временем скорость модифицировали до 10 Мбит/с, а в названии появилась надпись «Ultra II».

Распространенные классы, то есть градации минимальных скоростей для записи, бывают от 2-го до 16-го. Чем выше класс, тем быстрее работает карта. Притом скорость чтения всегда превосходит этот показатель.

Как выбрать класс TF (MicroSD):

— 2 – для аудио-, видеоплееров и небольших гаджетов;

— 4 – для «мыльниц», делающих снимки JPG, и видеокамер;

— 6 – для полупрофессиональных зеркальных фотоаппаратов, записывающих в JPG и RAW;

— 10 – для записи Full HD в формате RAW, просмотра фильмов и прохождения игр на планшете или ноутбуке.

Также есть UHS Speed Class 1 (U1) и UHS Speed Class 3 (U3). U1 позволяет записывать FullHD-видео, а скорость не опускается ниже 10 МБ/с. U3 – скорость не менее 30 МБ/с, плюс возможность записи 4К видео.

Иногда производители указывают скоростные показатели в виде множителей. Тогда нужно умножить приведенное значение, например, 40х, на 0,15. Получится 40 х 0,15 = 6 МБ/с.

В TF встроена защита содержимого от изменения/стирания данных. Эта функция объясняется тем, что получить доступ к информации могут только авторизированные пользователи.

TF-карта может работать в режиме чтения. Для отключения записи контента есть обратимые и необратимые команды хоста.

Важная особенность – TF-карта полностью совместима с MicroSD. Это значит, что устройство, работающее с MicroSD, будет работать и с TF. Интересно, что у всех стандартов обычно работает только прямая совместимость, то есть поддержка старых форматов.

## Applications of TF-IDF

Determining how relevant a word is to a document, or TD-IDF, is useful in many ways, for example:

Information retrieval

TF-IDF was invented for document search and can be used to deliver results that are most relevant to what you’re searching for. Imagine you have a search engine and somebody looks for LeBron. The results will be displayed in order of relevance. That’s to say the most relevant sports articles will be ranked higher because TF-IDF gives the word LeBron a higher score.

It’s likely that every search engine you have ever encountered uses TF-IDF scores in its algorithm.

Keyword Extraction

TF-IDF is also useful for extracting keywords from text. How? The highest scoring words of a document are the most relevant to that document, and therefore they can be considered keywords for that document. Pretty straightforward.

## Let’s cover an example of 3 documents —

Document 1 It is going to rain today.

Document 2 Today I am not going outside.

Document 3 I am going to watch the season premiere.

To find TF-IDF we need to perform the steps we laid out above, let’s get to it.

## Step 2 Find TF

Document 1—

Find it’s TF = (Number of repetitions of word in a document) / (# of words in a document)

TF for sentence 1

Continue for rest of sentences —

TF for the document

## Step 3 Find IDF

Find IDF for documents (we do this for feature names only/ vocab words which have no stop words )

IDF =Log[(Number of documents) / (Number of documents containing the word)]

IDF for document

## Step 5 Compare results and use table to ask questions

Remember, the final equation = TF-IDF = TF * IDF

You can easily see using this table that words like ‘it’,’is’,’rain’ are important for document 1 but not for document 2 and document 3 which means Document 1 and 2&3 are different w.r.t talking about rain.

You can also say that Document 1 and 2 talk about something ‘today’, and document 2 and 3 discuss something about the writer because of the word ‘I’.

This table helps you find similarities and non similarities btw documents, words and more much much better than BOW.

## What is TF-IDF (and what search engines have to do with it)?

TF-IDF (term frequency — inverse document frequency) is a statistical measure usually used in information retrieval and text mining to evaluate how important a term is to a particular document in a collection of documents. It has a long history in different research fields, such as linguistics and information architecture, due to its ability to facilitate the analysis of massive sets of documents in a short amount of time.

Search engines often use different variants of the TF-IDF algorithm as a part of their ranking mechanism. By giving documents a relevance score, they manage to deliver «rubbish-free» search results in milliseconds.

For example, TF-IDF has long been a part of Google’s ranking mechanism. Google uses TF-IDF to determine which terms are topically relevant (or irrelevant) by analyzing how often a term appears on a page (term frequency — TF) and how often it’s expected to appear on an average page, based on a larger set of documents (inverse document frequency — IDF).

To determine how relevant a given page is, Google analyzes the pages in its index against a number of specific features it considers relevant to the query.

Since most online content is text, these features, most probably, are the presence or absence of certain terms and phrases on the page. And not only their presence, but their prominence on this page as opposed to other pages across the web.

This is where the TF-IDF algorithm might come in handy. It measures the average use frequency for this particular term on the whole web as well as sets a benchmark to stop words to provide even greater prominence.

Let’s see how the TF-IDF formula works.

## Устранение неполадок при открытии файлов TF

### Общие проблемы с открытием файлов TF

#### Turbo Profiler не установлен

Дважды щелкнув по файлу TF вы можете увидеть системное диалоговое окно, в котором сообщается «Не удается открыть этот тип файла». В этом случае обычно это связано с тем, что на вашем компьютере не установлено Turbo Profiler для %%os%%. Так как ваша операционная система не знает, что делать с этим файлом, вы не сможете открыть его дважды щелкнув на него.

Совет: Если вам извстна другая программа, которая может открыть файл TF, вы можете попробовать открыть данный файл, выбрав это приложение из списка возможных программ.

#### Установлена неправильная версия Turbo Profiler

В некоторых случаях у вас может быть более новая (или более старая) версия файла Turbo Profiler Configuration, не поддерживаемая установленной версией приложения. При отсутствии правильной версии ПО Turbo Profiler (или любой из других программ, перечисленных выше), может потребоваться загрузить другую версию ПО или одного из других прикладных программных средств, перечисленных выше. Такая проблема чаще всего возникает при работе в более старой версии прикладного программного средства с файлом, созданным в более новой версии, который старая версия не может распознать.

Совет: Иногда вы можете получить общее представление о версии файла TF, щелкнув правой кнопкой мыши на файл, а затем выбрав «Свойства» (Windows) или «Получить информацию» (Mac OSX).

Резюме: В любом случае, большинство проблем, возникающих во время открытия файлов TF, связаны с отсутствием на вашем компьютере установленного правильного прикладного программного средства.

Даже если на вашем компьютере уже установлено Turbo Profiler или другое программное обеспечение, связанное с TF, вы все равно можете столкнуться с проблемами во время открытия файлов Turbo Profiler Configuration. Если проблемы открытия файлов TF до сих пор не устранены, возможно, причина кроется в других проблемах, не позволяющих открыть эти файлы. Такие проблемы включают (представлены в порядке от наиболее до наименее распространенных):

## The mechanics of TF-IDF

By now you’ve noticed that there are two terms in the notion. While term frequency is more or less clear, what is that mysterious inverse document frequency?

TF-IDF can be calculated according to the following formula:

Don’t worry, you do not have to calculate everything yourself; there are tools to do that for you. However, before using any tool, you should understand that TF-IDF value is not just a crafty form of keyword density. Here’s how it works:

Term Frequency (TF)

At first glance, the metric is clear: how frequently a term appears in a document. It’s calculated according to the following formula (and don’t worry, I will do the math for you):

For example, if you have a page of 1,000 words where your keyword appears 10 times, its term frequency will be 4.32/9.97=0.43 (if you use log base 2 in the formula).

If you make your keyword appear twice as much in the same document, its term frequency won’t change much, it will be 5.32/9.97=0.53 (log base 2 again).

Term frequency reflects whether you are using a particular keyword too often or too rarely. However, on its own, it’s pretty useless because you need to measure term’s importance, not just the frequency of its uses. Otherwise, function words would rule the search. To prevent it, we need IDF.

Inverse Document Frequency (IDF)

This metric helps understand the real value of a particular keyword. It measures the ratio of the total number of documents in a set to the number of documents that actually contain this keyword. The formula goes like this:

If the keyword is a common word, most probably it will be used in a large amount of documents. As a result, its IDF value will be tiny, and if we multiply TF by it, the value won’t change much. And vice versa, if the term is found only in a few documents, its IDF value will be much larger resulting in a larger TDF-IDF score.

So you see, unlike keyword density that only reflects how stuffed your text is with a particular keyword, TF-IDF comes as a more advanced and sophisticated metric that reflects the importance of a given keyword to a given page. It scales down the prominence of unimportant words and phrases, while rare, meaningful terms are scaled up in importance.

Having this thought in mind, let’s check out what TF-IDF has to do with SEO.

## Calculate TermFrequency and generate a matrix

We’ll find the TermFrequency for each word in a paragraph.

Now, remember the definition of TF,

TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document)

Here, the document is a paragraph, the term is a word in a paragraph.

Now the resultant matrix would look something like this:

{'\nThose Who Are ': {'resili': 0.03225806451612903, 'stay': 0.03225806451612903, 'game': 0.03225806451612903, 'longer': 0.03225806451612903, '“': 0.03225806451612903, 'mountain': 0.03225806451612903}, 'However, I real': {'howev': 0.07142857142857142, ',': 0.14285714285714285, 'realis': 0.07142857142857142, 'mani': 0.07142857142857142, 'year': 0.07142857142857142}, 'Have you experi': {'experienc': 0.25, 'thi': 0.25, 'befor': 0.25, '?': 0.25}, 'To be honest, I': {'honest': 0.2, ',': 0.2, '’': 0.2, 'answer': 0.2, '.': 0.2}, 'I can’t tell yo': {'’': 0.1111111111111111, 'tell': 0.1111111111111111, 'right': 0.1111111111111111, 'cours': 0.1111111111111111, 'action': 0.1111111111111111, ';': 0.1111111111111111, 'onli': 0.1111111111111111, 'know': 0.1111111111111111, '.': 0.1111111111111111}}

If we compare this table with the table we’ve generated in step 2, you will see the words having the same frequency are having the similar TF score.

## Preprocessing

Finally, we are going to put in all those preprocessing methods above in another method and we will call that preprocess method.

def preprocess(data):    data = convert_lower_case(data)    data = remove_punctuation(data)    data = remove_apostrophe(data)    data = remove_single_characters(data)    data = convert_numbers(data)    data = remove_stop_words(data)    data = stemming(data)    data = remove_punctuation(data)    data = convert_numbers(data)

If you look closely, few of the preprocessing methods are repeated again. As discussed, this just helps clean the data little deep. Now we need to read the documents and store their title and the body separately as we are going to use it later. In our problem statement we have very different types of documents, this can cause few errors in reading the documents due to encoding compatibility. to resolve this, just use encoding=”utf8″, errors=’ignore’ in the open() method.

## Document Frequency

This measures the importance of document in whole set of corpus, this is very similar to TF. The only difference is that TF is frequency counter for a term t in document d, where as DF is the count of occurrences of term t in the document set N. In other words, DF is the number of documents in which the word is present. We consider one occurrence if the term consists in the document at least once, we do not need to know the number of times the term is present.

To keep this also in a range, we normalize by dividing with the total number of documents. Our main goal is to know the informativeness of a term, and DF is the exact inverse of it. that is why we inverse the DF

## Step 2: Extracting Title & Body:

There is no specific way to do this, this totally depends on the problem statement at hand and on the analysis we do on the dataset.

As we have already found that the titles and the document names are in the index.html, we need to extract those names and titles. We are lucky that html has tags which we can use as patterns to extract our required content.

Before we start extracting the titles and file names, as we have different folders, first let’s crawl to the folders to later read all the index.html files at once.

 for x in os.walk(str(os.getcwd())+’/stories/’)]

os.walk gives us the files in the directory, os.getcwd gives us the current directory and title and we are going to search in the current directory + stories folder as our data files are in the stories folder.

Now we can find that folders give extra for the root folder, so we are going to remove it

folders = folders)-1]

the above code removes the last character for the 0th index in folders, which is the root folder

Now, let’s crawl through all the index.html to extract their titles. To do that we need to find a pattern to take out the title. As this is in html, our job will be little simpler.

let’s see…

We can clearly observe that each file name is enclosed between (><A HREF=”) and (”) and each title is between (<BR><TD>) and (\n)

We will use simple regular expression to retrieve the name and title. The following code gives the list of all the values that match that pattern. so names and titles variables have the list of all names and titles.

names = re.findall(‘><A HREF=”(.*)”>’, text)titles = re.findall(‘<BR><TD> (.*)\n’, text)

Now that we have code to retrieve the values from index, we just need to iterate to all the folders and get the title and file name from all the index.html files

dataset = []for i in folders:    file = open(i+"/index.html", 'r')    text = file.read().strip()    file.close()    file_name = re.findall('><A HREF="(.*)">', text)    file_title = re.findall('<BR><TD> (.*)\n', text)    for j in range(len(file_name)):        dataset.append((str(i) + str(file_name), file_title))

This prepares the indexes of the dataset, which is a tuple of location of file and its title. There is a small issue, the root folder index.html also has folders and its links, we need to remove those

simply use a conditional checker to remove it.

if c == False:    file_name = file_name    c = True

## Step 3: Preprocessing

Preprocessing is one of the major steps when we are dealing with any kind of text models. During this stage we have to look at the distribution of our data, what techniques are needed and how deep we should clean.

This step never has a one hot rule, and totally depends on the problem statement. Few mandatory preprocessing are converting to lowercase, removing punctuation, removing stop words and lemmatization/stemming. In our problem statement it seems like the basic preprocessing steps will be sufficient.

## How is TF-IDF calculated?

TF-IDF for a word in a document is calculated by multiplying two different metrics:

• The term frequency of a word in a document. There are several ways of calculating this frequency, with the simplest being a raw count of instances a word appears in a document. Then, there are ways to adjust the frequency, by length of a document, or by the raw frequency of the most frequent word in a document.
• The inverse document frequency of the word across a set of documents. This means, how common or rare a word is in the entire document set. The closer it is to 0, the more common a word is. This metric can be calculated by taking the total number of documents, dividing it by the number of documents that contain a word, and calculating the logarithm.
• So, if the word is very common and appears in many documents, this number will approach 0. Otherwise, it will approach 1.

Multiplying these two numbers results in the TF-IDF score of a word in a document. The higher the score, the more relevant that word is in that particular document.

To put it in more formal mathematical terms, the TF-IDF score for the word t in the document d from the document set D is calculated as follows:

Where:

## Challenge is to use these sentences and find words which provide meaning to these sentences using TF-IDF, ok?

Let’s begin

#Part 1 Declaring all documents and assigning to a Vocab document

Document1= “It is going to rain today.”Document2= “Today I am not going outside.”Document3= “I am going to watch the season premiere.”Doc = print(Doc)Output>>>

#Part 2 —intializing TFIDFVectorizer

from sklearn.feature_extraction.text import TfidfVectorizervectorizer = TfidfVectorizer()

Simple how easy to deploy TF-IDF , right ?

#Part 3 — Getting feature names of final words that we will use to tag documents

analyze = vectorizer.build_analyzer()print(‘Document 1’,analyze(Document1))print(‘Document 2’,analyze(Document2))print(‘Document 3’,analyze(Document3))print(‘Document transform’,X.toarray())Output>>>Document 1 Document 2 Document 3  Document transform   ]

See how each sentence is broken in words and each word is represented as a number for the machine, I’ve broken both above.

#Part 4 — Vectorizing or creating a matrix of all three documents and finding feature names

X = vectorizer.fit_transform(Doc)print(vectorizer.get_feature_names())Output>>>

The output signifies the important words which add context to 3 sentences. These are the words that are important in all 3 sentences and now you can ask questions of whatever nature you like to the machine, stuff like

Because the machine has a score to help aid with these questions, TF-IDF proves a great tool to train machine to answer back in case of chatbots as well.

If you would like to view the full code —

Go checkout my Github here > Check Bag of words code.

## TF-card — что это такое? Описание

TransFlash была анонсирована Ассоциацией SD Card в качестве третьей карты памяти форм-фактора в семействе Secure Digital после карты памяти SD и miniSD. После принятия стандарта TransFlash корпорация изменила имя продуктового решения на microSD. Объем TF card — что это? MicroSD имеет те же размеры и спецификации, что и TransFlash, и поэтому обе карты памяти полностью совместимы друг с другом.

Карты TransFlash (TF) и microSD почти одинаковы и обычно могут использоваться взаимозаменяемо, за исключением того, что у microSD есть поддержка режима SDIO, а TF данной возможности не имеет. SDIO позволяет картам microSD выполнять задания без памяти, такие как Bluetooth, GPS и

«The most fun you can have online» — PC GamerIs now FREE! There’s no catch! Play as much as you want, as long as you like!

The most highly-rated free game of all time! One of the most popular online action games of all time, Team Fortress 2 delivers constant free updates—new game modes, maps, equipment and, most importantly, hats. Nine distinct classes provide a broad range of tactical abilities and personalities, and lend themselves to a variety of player skills.

New to TF? Don’t sweat it! No matter what your style and experience, we’ve got a character for you. Detailed training and offline practice modes will help you hone your skills before jumping into one of TF2’s many game modes, including Capture the Flag, Control Point, Payload, Arena, King of the Hill and more.

Make a character your own! There are hundreds of weapons, hats and more to collect, craft, buy and trade. Tweak your favorite class to suit your gameplay style and personal taste. You don’t need to pay to win—virtually all of the items in the Mann Co. Store can also be found in-game.

## 3.2 Zipf’s law

Distributions like those shown in Figure are typical in language. In fact, those types of long-tailed distributions are so common in any given corpus of natural language (like a book, or a lot of text from a website, or spoken words) that the relationship between the frequency that a word is used and its rank has been the subject of study; a classic version of this relationship is called Zipf’s law, after George Zipf, a 20th century American linguist.

Zipf’s law states that the frequency that a word appears is inversely proportional to its rank.

Since we have the data frame we used to plot term frequency, we can examine Zipf’s law for Jane Austen’s novels with just a few lines of dplyr functions.

The column here tells us the rank of each word within the frequency table; the table was already ordered by so we could use to find the rank. Then, we can calculate the term frequency in the same way we did before. Zipf’s law is often visualized by plotting rank on the x-axis and term frequency on the y-axis, on logarithmic scales. Plotting this way, an inversely proportional relationship will have a constant, negative slope.

Figure 3.2: Zipf’s law for Jane Austen’s novels

Notice that Figure is in log-log coordinates. We see that all six of Jane Austen’s novels are similar to each other, and that the relationship between rank and frequency does have negative slope. It is not quite constant, though; perhaps we could view this as a broken power law with, say, three sections. Let’s see what the exponent of the power law is for the middle section of the rank range.

Classic versions of Zipf’s law have

\ and we have in fact gotten a slope close to -1 here. Let’s plot this fitted power law with the data in Figure to see how it looks.

Figure 3.3: Fitting an exponent for Zipf’s law with Jane Austen’s novels

С этим читают