Part-of-Speech Tagging

In this lesson, we’re going to learn about the textual analysis methods part-of-speech tagging and keyword extraction. These methods will help us computationally parse sentences and better understand words in context.


[Charles] Babbage, who called [Ada Lovelace] the “enchantress of numbers,” once wrote that she “has thrown her magical spell around the most abstract of Sciences and has grasped it with a force which few masculine intellects (in our own country at least) could have exerted over it.

—Claire Cain Miller, “Ada Lovelace,” NYT Overlooked Obituaries

She “ PRON has AUX thrown VERB her DET magical ADJ spell NOUN around ADP the DET most ADV abstract ADJ of ADP Sciences. PROPN nsubj aux poss amod dobj prep det advmod pobj prep pobj

Why is Part-of-Speech Tagging Useful?

I don’t mean to go all Language Nerd on you, but parts of speech are important. Even if they seem kind of boring. Parts of speech are the grammatical units of language — such as (in English) nouns, verbs, adjectives, adverbs, pronouns, and prepositions. Each of these parts of speech plays a different role in a sentence.

https://imgs.xkcd.com/comics/language_nerd.png

By computationally identifying parts of speech, we can start computationally exploring syntax, the relationship between words — rather than only focusing on words in isolation, as we did with tf-idf. Though parts of speech may seem pedantic, they help computers (and us) crack at that ever-elusive abstract noun — meaning.

spaCy and Natural Language Processing (NLP)

To computationally identify parts of speech, we’re going to use the natural language processing library spaCy. For a more extensive introduction to NLP and spaCy, see the previous lesson.

To parse sentences, spaCy relies on machine learning models that were trained on large amounts of labeled text data. The English-language spaCy model that we’re going to use in this lesson was trained on an annotated corpus called “OntoNotes”: 2 million+ words drawn from “news, broadcast, talk shows, weblogs, usenet newsgroups, and conversational telephone speech,” which were meticulously tagged by a group of researchers and professionals for people’s names and places, for nouns and verbs, for subjects and objects, and much more.

Install spaCy

To use spaCy, we first need to install the library.

!pip install -U spacy

Import Libraries

Then we’re going to import spacy and displacy, a special spaCy module for visualization.

import spacy
from spacy import displacy
from collections import Counter
import pandas as pd
pd.set_option("max_rows", 400)
pd.set_option("max_colwidth", 400)

We’re also going to import the Counter module for counting nouns, verbs, adjectives, etc., and the pandas library for organizing and displaying data (we’re also changing the pandas default max row and column width display setting).

Download Language Model

Next we need to download the English-language model (en_core_web_sm), which will be processing and making predictions about our texts. This is the model that was trained on the annotated “OntoNotes” corpus. You can download the en_core_web_sm model by running the cell below:

!python -m spacy download en_core_web_sm
Requirement already satisfied: en_core_web_sm==2.1.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.1.0/en_core_web_sm-2.1.0.tar.gz#egg=en_core_web_sm==2.1.0 in /Users/melaniewalsh/anaconda3/lib/python3.7/site-packages (2.1.0)
✔ Download and installation successful
You can now load the model via spacy.load('en_core_web_sm')

Note: spaCy offers models for other languages including German, French, Spanish, Portuguese, Italian, Dutch, Greek, Norwegian, and Lithuanian. Languages such as Russian, Ukrainian, Thai, Chinese, Japanese, Korean and Vietnamese don’t currently have their own NLP models. However, spaCy offers language and tokenization support for many of these language with external dependencies — such as PyviKonlpy for Korean or Jieba for Chinese.

Load Language Model

Once the model is downloaded, we need to load it with spacy.load() and assign it to the variable nlp.

nlp = spacy.load('en_core_web_sm')

Create a Processed spaCy Document

Whenever we use spaCy, our first step will be to create a processed spaCy document with the loaded NLP model nlp(). Most of the heavy NLP lifting is done in this line of code. After processing, the document object will contain tons of juicy language data — named entities, sentence boundaries, parts of speech — and the rest of our work will be devoted to accessing this information.

To test out spaCy’s part-of-speech tagging, we’ll begin by processing a sample sentence from Ada Lovelace’s obituary:

“[Charles] Babbage, who called [Ada Lovelace] the “enchantress of numbers,” once wrote that she “has thrown her magical spell around the most abstract of Sciences and has grasped it with a force which few masculine intellects (in our own country at least) could have exerted over it.

This sentence makes for an interesting example because it is syntactically complex and because it includes contains difficultly ambiguous words such as “spell,” “abstract,” and “force.”

sample = """She “has thrown her magical spell around the most abstract of Sciences."""
document = nlp(sample)

spaCy Part-of-Speech Tagging

POS

Description

Examples

ADJ

adjective

big, old, green, incomprehensible, first

ADP

adposition

in, to, during

ADV

adverb

very, tomorrow, down, where, there

AUX

auxiliary

is, has (done), will (do), should (do)

CONJ

conjunction

and, or, but

CCONJ

coordinating conjunction

and, or, but

DET

determiner

a, an, the

INTJ

interjection

psst, ouch, bravo, hello

NOUN

noun

girl, cat, tree, air, beauty

NUM

numeral

1, 2017, one, seventy-seven, IV, MMXIV

PART

particle

’s, not,

PRON

pronoun

I, you, he, she, myself, themselves, somebody

PROPN

proper noun

Mary, John, London, NATO, HBO

PUNCT

punctuation

., (, ), ?

SCONJ

subordinating conjunction

if, while, that

SYM

symbol

$, %, §, ©, +, −, ×, ÷, =, :), 😝

VERB

verb

run, runs, running, eat, ate, eating

X

other

sfpksdpsxmsa

SPACE

space

Above is a POS chart taken from spaCy’s website, which shows the different parts of speech that spaCy can identify as well as their corresponding labels. To quickly see spaCy’s POS tagging in action, we can use the spaCy module displacy on our sample document with the style= parameter set to “dep” (short for dependency parsing):

#Set some display options for the visualizer
options = {"compact": True, "distance": 90, "color": "yellow", "bg": "black", "font": "Gill Sans"}

displacy.render(document, style="dep", options=options)
She “ PRON has AUX thrown VERB her DET magical ADJ spell NOUN around ADP the DET most ADV abstract ADJ of ADP Sciences. PROPN nsubj aux poss amod dobj prep det advmod pobj prep pobj

As you can see, spaCy has correctly identified that “spell” and “force” are nouns in our sample sentence:

for token in document:
    if token.pos_ == "NOUN":
        print(token, token.pos_)
spell NOUN
force NOUN
intellects NOUN
country NOUN

But if we look at the same words in a different context — in a sentence that I made up — spaCy can identify when these words have changed grammatical roles and meanings.

You shouldn’t force someone to learn how to spell Babbage. They just need practice. You can’t abstract it.

document = nlp("You shouldn't force someone to learn how to spell Babbage. They just need practice. You can't abstract it.")
for token in document:
    if token.pos_ == "VERB":
        print(token, token.pos_)
force VERB
learn VERB
spell VERB
need VERB
abstract VERB

Where previously spaCy had identified “force” and “spell” as nouns, here spaCy correctly identifies the words “force,” “spell,” and “abstract” as verbs.

Get Part-Of-Speech Tags

To get part of speech tags for every word in a document, we have to iterate through all the tokens in the document and pull out the .pos_ attribute for each token. We can get even finer-grained dependency information with the attribute .dep_.

for token in document:
    print(token.text, token.pos_, token.dep_)
You PRON nsubj
should AUX aux
n't ADV neg
force VERB ROOT
someone NOUN dobj
to PART aux
learn VERB xcomp
how ADV advmod
to PART aux
spell VERB xcomp
Babbage PROPN dobj
. PUNCT punct
They PRON nsubj
just ADV advmod
need VERB ROOT
practice NOUN dobj
. PUNCT punct
You PRON nsubj
ca AUX aux
n't ADV neg
abstract VERB ROOT
it PRON dobj
. PUNCT punct

Practicing with Lost in the City

For the rest of this lesson, we’re going to explore Edward P. Jones’s short story collection Lost in the City.

If you’re using this Jupyter notebook in Binder (in the cloud), please uncomment the cell below and work with only the first story from Lost in the City. The Binder notebook is currently having issues loading the entire collection.

#file = "../texts/literature/Lost-in-the-City_Stories/01-The-Girl-Who-Raised-Pigeons.txt"
#document = nlp(open(file).read())
filepath = "../texts/literature/Jones-Lost-in-The-City.txt"
document = nlp(open(filepath, encoding="utf-8").read())

Get Adjectives

POS

Description

Examples

ADJ

adjective

big, old, green, incomprehensible, first

To extract and count the adjectives in Lost in the City, we will follow the same model as above, except we’ll add an if statement that will pull out words only if their POS label matches “ADJ.”

Python Review!

While we demonstrate how to extract parts of speech in the sections below, we’re also going to reinforce some integral Python skills. Notice how we use for loops and if statements to .append() specific words to a list. Then we count the words in the list and make a pandas dataframe from the list.

Here we make a list of the adjectives identified in Lost in the City:

adjs = []
for token in document:
    if token.pos_ == 'ADJ':
        adjs.append(token.text)
adjs
['many',
 'vivid',
 'simple',
 'front',
 'few',
 'most',
 'least',
 'wooden',
 'few',
 'brown',
 'old',
 'few',
 'old',
 'eager',
 'same',
 'frightened',
 'furious',
 'colorful',
 'thick',
 'extended',
 'old',
 'same',
 'gray',
 'dead',
 'different',
 'radiant',
 'black',
 'impossible',
 'old',
 'most',
 'more',
 'own',
 'great',
 'such',
 'such',
 'difficult',
 'old',
 'few',
 'good',
 'obedient',
 'good',
 'most',
 'dead',
 'first',
 'long',
 'hopeful',
 'long',
 'half',
 'asleep',
 'most',
 'more',
 'welcome',
 'dead',
 'dead',
 'such',
 'little',
 'more',
 'good',
 'young',
 'blue',
 'crazy',
 'young',
 'wet',
 'own',
 'close',
 'certain',
 'clear',
 'uttered',
 'double',
 'next',
 'vacant',
 'front',
 'last',
 'rainy',
 'bigger',
 'much',
 'glad',
 'small',
 'top',
 'other',
 'most',
 'first',
 'pretty',
 'last',
 'third',
 'other',
 'other',
 'other',
 'other',
 'small',
 'old',
 'last',
 'old',
 'few',
 'right',
 'young',
 'whole',
 'lighter',
 'fluttering',
 'good',
 'good',
 'good',
 'young',
 'first',
 'front',
 'old',
 'next',
 'yellow',
 'yellow',
 'next',
 'light',
 'green',
 'enough',
 'blue',
 'only',
 'uncomfortable',
 'helpless',
 'few',
 '1st',
 'particular',
 'damn',
 '1st',
 'warm',
 'narrow',
 'oblivious',
 'own',
 'easy',
 'soundless',
 'pink',
 'much',
 'clear',
 'least',
 'few',
 'More',
 'best',
 'nice',
 'haired',
 'old',
 'easy',
 'younger',
 'top',
 'dull',
 'white',
 'black',
 'red',
 'several',
 'forlorn',
 'old',
 'first',
 'second',
 'same',
 'first',
 'other',
 'sure',
 'inseparable',
 'new',
 'gotdamn',
 'puppy',
 'puppy',
 'gotdamn',
 'puppy',
 'gettin',
 'older',
 'taller',
 'big',
 'big',
 'oldest',
 'very',
 'overwhelming',
 'fresh',
 'open',
 'such',
 'contented',
 'unhappy',
 'ready',
 'strutted',
 'oblivious',
 'right',
 'aware',
 'stupid',
 'long',
 'high',
 'medieval',
 'like',
 '1st',
 'high',
 'white',
 'white',
 'colored',
 'high',
 'same',
 'same',
 'late',
 'entire',
 'special',
 'crude',
 'dead',
 'other',
 'uncertain',
 'hapless',
 'long',
 'first',
 'solid',
 'about',
 'weekly',
 'entire',
 'small',
 'perfect',
 'male',
 'few',
 'bored',
 'female',
 'male',
 'dead',
 'hideous',
 'eyed',
 'dirty',
 'responsible',
 'first',
 'unsettling',
 'little',
 'predictable',
 'afraid',
 'first',
 'dead',
 'less',
 'various',
 'lower',
 'few',
 'submissive',
 'entire',
 'little',
 'ragged',
 'left',
 'additional',
 'small',
 'dead',
 'full',
 'dead',
 'stupid',
 'few',
 'various',
 'dead',
 'grassy',
 'medieval',
 'large',
 'preacher',
 'dead',
 'good',
 'sleepin',
 'cold',
 'cold',
 'open',
 'cold',
 'funny',
 'late',
 'mere',
 'akin',
 'double',
 'new',
 'thick',
 'old',
 'first',
 'few',
 'pleasurable',
 'less',
 'rainy',
 'early',
 'long',
 'loose',
 'lighthearted',
 'standard',
 'other',
 'true',
 'true',
 'other',
 'comfortable',
 'other',
 'secret',
 'hard',
 'second',
 'long',
 'other',
 'small',
 'other',
 'seventh',
 'secure',
 'first',
 'colored',
 'own',
 'asthmatic',
 'more',
 'back',
 'large',
 'enormous',
 'much',
 'electric',
 'more',
 'square',
 'marvelous',
 'complete',
 'simple',
 'such',
 'floppy',
 'real',
 'microscopic',
 'several',
 'absolute',
 'small',
 'other',
 'other',
 'first',
 'populated',
 'large',
 'light',
 'blue',
 'several',
 'tiny',
 'many',
 'invisible',
 'enormous',
 'puppy',
 'little',
 'puppy',
 'part',
 'more',
 'attractive',
 'difficult',
 'mean',
 'impatient',
 'other',
 'stupid',
 'stupidest',
 'whole',
 'wide',
 'necessary',
 'clean',
 'bad',
 'best',
 'farthest',
 'other',
 'fast',
 'last',
 'assistant',
 'last',
 'same',
 'little',
 'hussy',
 'assistant',
 'about',
 'oldest',
 'old',
 'assistant',
 'able',
 'higher',
 'better',
 'old',
 'assistant',
 'assistant',
 'old',
 'evil',
 'next',
 'good',
 'little',
 'same',
 'damn',
 'first',
 'few',
 'several',
 'back',
 'most',
 'good',
 'old',
 'blue',
 'most',
 'deliberate',
 'old',
 'back',
 'Little',
 'little',
 'other',
 'free',
 'clean',
 'other',
 'old',
 'screenless',
 'naked',
 'clean',
 'first',
 'small',
 'alien',
 'tall',
 'wild',
 'good',
 'normal',
 'possible',
 'last',
 'light',
 'nuisance',
 'thick',
 'beautiful',
 'enough',
 'first',
 'tiny',
 'intermittent',
 'next',
 'empty',
 'more',
 'more',
 'various',
 'dead',
 'last',
 'lower',
 'own',
 'white',
 'more',
 'brighter',
 'alone',
 'dead',
 'left',
 'able',
 'farthest',
 'logical',
 'surprised',
 'ragged',
 'amazed',
 'Most',
 'front',
 'awake',
 'burlap',
 'new',
 'few',
 'only',
 'unharmed',
 'upper',
 'old',
 'barefoot',
 'more',
 'many',
 'nasty',
 'favorite',
 'same',
 'warm',
 'many',
 'back',
 'typical',
 'empty',
 'few',
 'very',
 'much',
 'next',
 'last',
 'cold',
 'last',
 'final',
 'high',
 'empty',
 'FIRST',
 'unremarkable',
 'ashamed',
 'first',
 'blue',
 'green',
 'yellow',
 'white',
 'brown',
 'faint',
 'blue',
 'uncommon',
 'full',
 'brown',
 'pale',
 'green',
 'new',
 'little',
 'stingiest',
 'gardenia',
 'last',
 'yellow',
 'thin',
 'black',
 'white',
 'greatest',
 'black',
 'small',
 'black',
 'white',
 'speckled',
 'such',
 'younger',
 'precious',
 'older',
 'new',
 'many',
 'sad',
 'faced',
 'front',
 'other',
 'thick',
 'jumbo',
 'gardenia',
 'wrong',
 'several',
 'more',
 'many',
 'whole',
 'higher',
 '—',
 'high',
 'less',
 'liable',
 'larger',
 'newer',
 'crowded',
 'gray',
 'left',
 'other',
 'buzz',
 'many',
 'white',
 'afraid',
 'tiny',
 'small',
 'happy',
 'large',
 'worthy',
 'high',
 'other',
 'several',
 'thick',
 'additional',
 'few',
 'new',
 'sure',
 'happier',
 'satisfied',
 'better',
 'last',
 'first',
 'middle',
 'more',
 'other',
 'baptismal',
 'public',
 'single',
 'old',
 'Few',
 'long',
 'last',
 'white',
 'old',
 'old',
 'loud',
 'loud',
 'low',
 'old',
 'seventh',
 'bored',
 'small',
 'other',
 'second',
 'second',
 'English',
 'more',
 'goddamn',
 'second',
 'green',
 'drunken',
 'old',
 'last',
 'first',
 'same',
 'wrong',
 'old',
 'cold',
 'seventh',
 'final',
 'second',
 'fine',
 'other',
 'small',
 'several',
 'brown',
 'thicker',
 'real',
 'more',
 'real',
 'personal',
 'personal',
 'enough',
 'other',
 'potential',
 'blah',
 'blah',
 'able',
 'last',
 'large',
 'stayin',
 'willya',
 'youngest',
 'only',
 'other',
 'older',
 'thick',
 'more',
 'right',
 'enthusiastic',
 'small',
 'late',
 'full',
 'nice',
 'several',
 'static',
 'friendly',
 'new',
 'new',
 'truest',
 'other',
 'good',
 'long',
 'second',
 'other',
 'quiet',
 'grandfather',
 'final',
 'chump',
 'real',
 'cheaper',
 'sendin',
 'alone',
 'few',
 'small',
 'strange',
 'best',
 'few',
 'possible',
 'more',
 'short',
 'immaculate',
 'Such',
 'nervous',
 'other',
 'surprised',
 'pleasant',
 'early',
 'last',
 'good',
 'full',
 'little',
 'usual',
 'small',
 'larger',
 'last',
 'decent',
 'alone',
 'own',
 'damn',
 'hearse',
 'bad',
 'first',
 'right',
 'sorry',
 'other',
 'good',
 'clear',
 'famous',
 'able',
 'same',
 'round',
 'white',
 'right',
 'sorry',
 'quiet',
 'right',
 'front',
 'second',
 'empty',
 'naked',
 'light',
 'pregnant',
 'best',
 'damn',
 'other',
 'empty',
 'empty',
 'gettin',
 'feelin',
 'better',
 'full',
 'other',
 'curtainless',
 'right',
 'sorry',
 'other',
 'slow',
 'young',
 'sooo',
 'cute',
 'cute',
 'good',
 'right',
 'heavy',
 'much',
 'small',
 'giant',
 'good',
 'close',
 'small',
 'larger',
 'front',
 'other',
 'young',
 'first',
 'much',
 'best',
 'whole',
 'many',
 'next',
 'only',
 'other',
 'weaker',
 'less',
 'more',
 'few',
 'close',
 'next',
 'second',
 'glad',
 'outta',
 'nice',
 'cute',
 'worse',
 'other',
 'little',
 'next',
 'many',
 'whole',
 'little',
 'sure',
 'sure',
 'flicted',
 'damn',
 'better',
 'damn',
 'good',
 '8th',
 'few',
 'different',
 'same',
 'quiet',
 'few',
 'older',
 'Most',
 'hi',
 'hi',
 'small',
 'back',
 'little',
 'young',
 'more',
 'dramatic',
 'oily',
 'small',
 'interested',
 'hi',
 'funny',
 'unreal',
 'little',
 'Several',
 'much',
 'tiny',
 'comfortable',
 'endearing',
 'hard',
 'other',
 'little',
 'right',
 'certain',
 'beautiful',
 'much',
 'poor',
 'little',
 'music—“my',
 'few',
 'back',
 'asleep',
 'front',
 'back',
 'rough',
 'young',
 'able',
 'drinkin',
 'rough',
 'rough',
 'back',
 'back',
 'little',
 'first',
 'good',
 'close',
 'few',
 'real',
 'nice',
 'rough',
 'cute',
 'tired',
 'cute',
 'damn',
 'next',
 'many',
 'cute',
 'shitty',
 'good',
 'best',
 'whole',
 'damn',
 'more',
 'dumb',
 'cute',
 'sick',
 'tired',
 'dumbest',
 'sooooo',
 'cute',
 'whole',
 'bad',
 'whole',
 'open',
 'other',
 '8th',
 'Little',
 'little',
 '8th',
 'about',
 'little',
 'drunk',
 'other',
 'first',
 'thinkin',
 'left',
 'good',
 'same',
 'blew',
 'plastic',
 'few',
 'bout',
 'only',
 'sure',
 'few',
 '13th',
 'little',
 'particular',
 'biggest',
 'worse',
 '13th',
 'low',
 'few',
 'important',
 'entire',
 'middle',
 'strange',
 'first',
 'real',
 'true',
 'middle',
 'angry',
 'best',
 'thrifty',
 'alive',
 'other',
 'old',
 'loudest',
 'third',
 'fourth',
 'goin',
 'foreign',
 'sleep',
 'Sometime',
 'alone',
 'naked',
 'open',
 'last',
 'half',
 'such',
 'asleep',
 'undisturbed',
 'gold',
 'key',
 'expensive',
 'more',
 'last',
 'high',
 'anxious',
 'empty',
 'fourth',
 'retarded',
 'such',
 'good',
 'only',
 'splintered',
 'foreign',
 'retarded',
 'tall',
 'feeble',
 'minded',
 'naked',
 'about',
 'obscene',
 'other',
 'bad',
 'little',
 'less',
 '“',
 'Good',
 'sure',
 'available',
 'inappropriate',
 'whole',
 'funny',
 'last',
 'retarded',
 'first',
 'old',
 'good',
 'second',
 'better',
 'more',
 'free',
 'few',
 'brown',
 'colder',
 'opulent',
 'certain',
 'much',
 'few',
 'male',
 'last',
 'few',
 'proper',
 'cheap',
 'proper',
 'nice',
 'retarded',
 'quick',
 'only',
 'first',
 'first',
 'cheap',
 'impressive',
 'locked',
 'most',
 'Hawaiian',
 'special',
 'wild',
 'eyed',
 'very',
 'irredeemable',
 'Most',
 'drunk',
 'enough',
 'goddamn',
 'alone',
 'unseen',
 'alone',
 'happy',
 'whole',
 'dark',
 'tiny',
 'small',
 'wrong',
 'little',
 'Hawaiian',
 'expensive',
 'loud',
 'tiny',
 'green',
 'thin',
 'more',
 'retarded',
 'little',
 'more',
 'closer',
 'Manny',
 'more',
 'little',
 'last',
 'small',
 'functional',
 'special',
 'off',
 'small',
 'few',
 'fine',
 'more',
 'more',
 'far',
 'dead',
 'dead',
 'satisfied',
 'interested',
 ...]

Then we count the unique adjectives in this list with the Counter() module:

adjs_tally = Counter(adjs)
adjs_tally.most_common()
[('other', 150),
 ('old', 142),
 ('more', 131),
 ('first', 106),
 ('good', 103),
 ('few', 101),
 ('little', 94),
 ('next', 66),
 ('small', 65),
 ('last', 64),
 ('white', 57),
 ('same', 52),
 ('second', 50),
 ('long', 49),
 ('much', 49),
 ('own', 48),
 ('new', 46),
 ('right', 44),
 ('whole', 42),
 ('dead', 41),
 ('most', 36),
 ('young', 34),
 ('only', 34),
 ('many', 32),
 ('several', 31),
 ('big', 31),
 ('bad', 30),
 ('such', 28),
 ('front', 26),
 ('black', 26),
 ('retarded', 26),
 ('dark', 25),
 ('late', 22),
 ('blue', 21),
 ('nice', 21),
 ('older', 21),
 ('high', 21),
 ('alone', 21),
 ('sorry', 21),
 ('best', 20),
 ('large', 20),
 ('back', 20),
 ('real', 20),
 ('able', 20),
 ('empty', 19),
 ('enough', 18),
 ('ready', 18),
 ('less', 18),
 ('surprised', 18),
 ('asleep', 17),
 ('green', 17),
 ('better', 17),
 ('loud', 17),
 ('damn', 16),
 ('afraid', 16),
 ('full', 16),
 ('open', 15),
 ('tiny', 15),
 ('happy', 15),
 ('5th', 15),
 ('least', 14),
 ('close', 14),
 ('easy', 14),
 ('quiet', 14),
 ('thick', 13),
 ('different', 13),
 ('wrong', 13),
 ('about', 12),
 ('left', 12),
 ('cold', 12),
 ('naked', 12),
 ('orange', 12),
 ('half', 11),
 ('glad', 11),
 ('light', 11),
 ('sure', 11),
 ('Most', 11),
 ('tired', 11),
 ('third', 10),
 ('red', 10),
 ('oldest', 10),
 ('early', 10),
 ('true', 10),
 ('tall', 10),
 ('entire', 9),
 ('assistant', 9),
 ('thin', 9),
 ('foreign', 9),
 ('skinned', 9),
 ('sittin', 9),
 ('clean', 8),
 ('beautiful', 8),
 ('final', 8),
 ('cute', 8),
 ('silent', 8),
 ('wooden', 7),
 ('brown', 7),
 ('certain', 7),
 ('clear', 7),
 ('younger', 7),
 ('gettin', 7),
 ('very', 7),
 ('stupid', 7),
 ('colored', 7),
 ('special', 7),
 ('perfect', 7),
 ('possible', 7),
 ('youngest', 7),
 ('usual', 7),
 ('pregnant', 7),
 ('giant', 7),
 ('interested', 7),
 ('drunk', 7),
 ('important', 7),
 ('cheap', 7),
 ('hungry', 7),
 ('10th', 7),
 ('crazy', 6),
 ('dirty', 6),
 ('wild', 6),
 ('awake', 6),
 ('larger', 6),
 ('low', 6),
 ('fine', 6),
 ('sick', 6),
 ('13th', 6),
 ('alive', 6),
 ('expensive', 6),
 ('“', 6),
 ('bright', 6),
 ('huge', 6),
 ('married', 6),
 ('gray', 5),
 ('difficult', 5),
 ('double', 5),
 ('top', 5),
 ('pretty', 5),
 ('yellow', 5),
 ('particular', 5),
 ('puppy', 5),
 ('aware', 5),
 ('various', 5),
 ('funny', 5),
 ('comfortable', 5),
 ('hard', 5),
 ('favorite', 5),
 ('—', 5),
 ('poor', 5),
 ('plastic', 5),
 ('biggest', 5),
 ('Good', 5),
 ('proper', 5),
 ('closer', 5),
 ('safe', 5),
 ('bald', 5),
 ('proud', 5),
 ('sweet', 5),
 ('past', 5),
 ('familiar', 5),
 ('blind', 5),
 ('simple', 4),
 ('eager', 4),
 ('great', 4),
 ('warm', 4),
 ('gotdamn', 4),
 ('fresh', 4),
 ('like', 4),
 ('male', 4),
 ('sleepin', 4),
 ('seventh', 4),
 ('brighter', 4),
 ('greatest', 4),
 ('precious', 4),
 ('crowded', 4),
 ('satisfied', 4),
 ('personal', 4),
 ('short', 4),
 ('heavy', 4),
 ('worse', 4),
 ('Several', 4),
 ('rough', 4),
 ('drinkin', 4),
 ('fourth', 4),
 ('colder', 4),
 ('quick', 4),
 ('federal', 4),
 ('capable', 4),
 ('worth', 4),
 ('regular', 4),
 ('fewer', 4),
 ('14th', 4),
 ('wonderful', 4),
 ('upset', 4),
 ('talkin', 4),
 ('lucky', 4),
 ('somethin', 4),
 ('common', 4),
 ('steady', 4),
 ('wet', 3),
 ('1st', 3),
 ('dull', 3),
 ('loose', 3),
 ('standard', 3),
 ('complete', 3),
 ('invisible', 3),
 ('wide', 3),
 ('necessary', 3),
 ('higher', 3),
 ('free', 3),
 ('middle', 3),
 ('public', 3),
 ('single', 3),
 ('goddamn', 3),
 ('thicker', 3),
 ('strange', 3),
 ('pleasant', 3),
 ('decent', 3),
 ('slow', 3),
 ('8th', 3),
 ('hi', 3),
 ('angry', 3),
 ('loudest', 3),
 ('Hawaiian', 3),
 ('far', 3),
 ('shiny', 3),
 ('okay', 3),
 ('Next', 3),
 ('bottom', 3),
 ('smaller', 3),
 ('longest', 3),
 ('human', 3),
 ('dressed', 3),
 ('horrible', 3),
 ('peculiar', 3),
 ('strong', 3),
 ('painful', 3),
 ('playful', 3),
 ('sized', 3),
 ('ill', 3),
 ('similar', 3),
 ('infirmary', 3),
 ('closest', 3),
 ('wobbly', 3),
 ('sickly', 3),
 ('unable', 3),
 ('kind', 3),
 ('handsome', 3),
 ('greasy', 3),
 ('senior', 3),
 ('utter', 3),
 ('general', 3),
 ('soft', 3),
 ('furious', 2),
 ('radiant', 2),
 ('welcome', 2),
 ('rainy', 2),
 ('bigger', 2),
 ('lighter', 2),
 ('uncomfortable', 2),
 ('narrow', 2),
 ('oblivious', 2),
 ('pink', 2),
 ('forlorn', 2),
 ('taller', 2),
 ('overwhelming', 2),
 ('medieval', 2),
 ('bored', 2),
 ('hideous', 2),
 ('eyed', 2),
 ('responsible', 2),
 ('lower', 2),
 ('ragged', 2),
 ('additional', 2),
 ('secret', 2),
 ('enormous', 2),
 ('square', 2),
 ('part', 2),
 ('attractive', 2),
 ('mean', 2),
 ('farthest', 2),
 ('fast', 2),
 ('evil', 2),
 ('Little', 2),
 ('normal', 2),
 ('logical', 2),
 ('nasty', 2),
 ('gardenia', 2),
 ('sad', 2),
 ('Few', 2),
 ('English', 2),
 ('blah', 2),
 ('friendly', 2),
 ('grandfather', 2),
 ('Such', 2),
 ('nervous', 2),
 ('outta', 2),
 ('oily', 2),
 ('dumb', 2),
 ('goin', 2),
 ('gold', 2),
 ('key', 2),
 ('feeble', 2),
 ('Manny', 2),
 ('curious', 2),
 ('smart', 2),
 ('Indian', 2),
 ('overweight', 2),
 ('funeral', 2),
 ('upstairs', 2),
 ('cool', 2),
 ('9th', 2),
 ('closed', 2),
 ('15th', 2),
 ('natural', 2),
 ('longtime', 2),
 ('calm', 2),
 ('grown', 2),
 ('flat', 2),
 ('chief', 2),
 ('worst', 2),
 ('skeptical', 2),
 ('lazy', 2),
 ('mad', 2),
 ('current', 2),
 ('longer', 2),
 ('fond', 2),
 ('passionate', 2),
 ('easier', 2),
 ('junior', 2),
 ('fair', 2),
 ('Many', 2),
 ('sunny', 2),
 ('hot', 2),
 ('legal', 2),
 ('bruised', 2),
 ('separate', 2),
 ('permanent', 2),
 ('shorter', 2),
 ('scary', 2),
 ('12th', 2),
 ('19th', 2),
 ('fit', 2),
 ('tabloid', 2),
 ('moody', 2),
 ('brotha', 2),
 ('nonanswer', 2),
 ('21st', 2),
 ('careful', 2),
 ('fifth', 2),
 ('tenth', 2),
 ('wealthy', 2),
 ('wondrous', 2),
 ('medical', 2),
 ('scared', 2),
 ('false', 2),
 ('careless', 2),
 ('kinda', 2),
 ('fashioned', 2),
 ('disabled', 2),
 ('sportin', 2),
 ('plain', 2),
 ('original', 2),
 ('willing', 2),
 ('sighted', 2),
 ('busy', 2),
 ('legitimate', 2),
 ('unfamiliar', 2),
 ('actin', 2),
 ('greater', 2),
 ('fellow', 2),
 ('frozen', 2),
 ('anough', 2),
 ('rainin', 2),
 ('darkest', 2),
 ('dear', 2),
 ('vivid', 1),
 ('frightened', 1),
 ('colorful', 1),
 ('extended', 1),
 ('impossible', 1),
 ('obedient', 1),
 ('hopeful', 1),
 ('uttered', 1),
 ('vacant', 1),
 ('fluttering', 1),
 ('helpless', 1),
 ('soundless', 1),
 ('More', 1),
 ('haired', 1),
 ('inseparable', 1),
 ('contented', 1),
 ('unhappy', 1),
 ('strutted', 1),
 ('crude', 1),
 ('uncertain', 1),
 ('hapless', 1),
 ('solid', 1),
 ('weekly', 1),
 ('female', 1),
 ('unsettling', 1),
 ('predictable', 1),
 ('submissive', 1),
 ('grassy', 1),
 ('preacher', 1),
 ('mere', 1),
 ('akin', 1),
 ('pleasurable', 1),
 ('lighthearted', 1),
 ('secure', 1),
 ('asthmatic', 1),
 ('electric', 1),
 ('marvelous', 1),
 ('floppy', 1),
 ('microscopic', 1),
 ('absolute', 1),
 ('populated', 1),
 ('impatient', 1),
 ('stupidest', 1),
 ('hussy', 1),
 ('deliberate', 1),
 ('screenless', 1),
 ('alien', 1),
 ('nuisance', 1),
 ('intermittent', 1),
 ('amazed', 1),
 ('burlap', 1),
 ('unharmed', 1),
 ('upper', 1),
 ('barefoot', 1),
 ('typical', 1),
 ('FIRST', 1),
 ('unremarkable', 1),
 ('ashamed', 1),
 ('faint', 1),
 ('uncommon', 1),
 ('pale', 1),
 ('stingiest', 1),
 ('speckled', 1),
 ('faced', 1),
 ('jumbo', 1),
 ('liable', 1),
 ('newer', 1),
 ('buzz', 1),
 ('worthy', 1),
 ('happier', 1),
 ('baptismal', 1),
 ('drunken', 1),
 ('potential', 1),
 ('stayin', 1),
 ('willya', 1),
 ('enthusiastic', 1),
 ('static', 1),
 ('truest', 1),
 ('chump', 1),
 ('cheaper', 1),
 ('sendin', 1),
 ('immaculate', 1),
 ('hearse', 1),
 ('famous', 1),
 ('round', 1),
 ('feelin', 1),
 ('curtainless', 1),
 ('sooo', 1),
 ('weaker', 1),
 ('flicted', 1),
 ('dramatic', 1),
 ('unreal', 1),
 ('endearing', 1),
 ('music—“my', 1),
 ('shitty', 1),
 ('dumbest', 1),
 ('sooooo', 1),
 ('thinkin', 1),
 ('blew', 1),
 ('bout', 1),
 ('thrifty', 1),
 ('sleep', 1),
 ('Sometime', 1),
 ('undisturbed', 1),
 ('anxious', 1),
 ('splintered', 1),
 ('minded', 1),
 ('obscene', 1),
 ('available', 1),
 ('inappropriate', 1),
 ('opulent', 1),
 ('impressive', 1),
 ('locked', 1),
 ('irredeemable', 1),
 ('unseen', 1),
 ('functional', 1),
 ('off', 1),
 ('average', 1),
 ('mellowest', 1),
 ('Real', 1),
 ('movin', 1),
 ('weakest', 1),
 ('easiest', 1),
 ('French', 1),
 ('scuffed', 1),
 ('copier', 1),
 ('skittish', 1),
 ('16th', 1),
 ('darkened', 1),
 ('inner', 1),
 ('outer', 1),
 ('joint', 1),
 ('outdoor', 1),
 ('disappointed', 1),
 ('bare', 1),
 ('due', 1),
 ('cooler', 1),
 ('soothin', 1),
 ('pro', 1),
 ('Satisfied', 1),
 ('roomful', 1),
 ('18th', 1),
 ('bench', 1),
 ('ohh', 1),
 ('17th', 1),
 ('excellent', 1),
 ('silly', 1),
 ('shot', 1),
 ('total', 1),
 ('straight', 1),
 ('chubby', 1),
 ('someplace', 1),
 ('interviewin', 1),
 ('skinny', 1),
 ('spiteful', 1),
 ('sore', 1),
 ('depressed', 1),
 ('prepared', 1),
 ('Wrong', 1),
 ('irresistible', 1),
 ('fucking', 1),
 ('nonmeat', 1),
 ('reeaal', 1),
 ('reckless', 1),
 ('terrible', 1),
 ('unusual', 1),
 ('mischievous', 1),
 ('hogshead', 1),
 ('Jewish', 1),
 ('main', 1),
 ('tiredness', 1),
 ('humid', 1),
 ('homemade', 1),
 ('hysterical', 1),
 ('rolling', 1),
 ('fuckin', 1),
 ('simplest', 1),
 ('me—“my', 1),
 ('talkative', 1),
 ('healthy', 1),
 ('exhausted', 1),
 ('hooed', 1),
 ('trustworthy', 1),
 ('executive', 1),
 ('halfway', 1),
 ('Clean', 1),
 ('cocky', 1),
 ('Alone', 1),
 ('rare', 1),
 ('flush', 1),
 ('imaginary', 1),
 ('West', 1),
 ('granny', 1),
 ('neva', 1),
 ('impressed', 1),
 ('Smithsonian', 1),
 ('inane', 1),
 ('compatible', 1),
 ('following', 1),
 ('Orange', 1),
 ('deeper', 1),
 ('fat', 1),
 ('conscious', 1),
 ('obvious', 1),
 ('near', 1),
 ('preoccupied', 1),
 ('horrendous', 1),
 ('towering', 1),
 ('judicial', 1),
 ('drafty', 1),
 ('affectionate', 1),
 ('loyal', 1),
 ('smallest', 1),
 ('crossed', 1),
 ('culpable', 1),
 ('writin', 1),
 ('precocious', 1),
 ('adorable', 1),
 ('energetic', 1),
 ('African', 1),
 ('unsuccessful', 1),
 ('superstitious', 1),
 ('relevant', 1),
 ('25th', 1),
 ('trophy', 1),
 ('Fine', 1),
 ('orangeicing', 1),
 ('plural', 1),
 ('luxurious', 1),
 ('keloid', 1),
 ('fireproof', 1),
 ('dutiful', 1),
 ('confused', 1),
 ('spected', 1),
 ('commencin', 1),
 ('stylish', 1),
 ('smoky', 1),
 ('ancient', 1),
 ('unoccupied', 1),
 ('makeshift', 1),
 ('paved', 1),
 ('lye', 1),
 ('eternal', 1),
 ('unreliable', 1),
 ('White', 1),
 ('upright', 1),
 ('fifteenth', 1),
 ('exemplary', 1),
 ('bulb', 1),
 ('lowest', 1),
 ('popular', 1),
 ('conveyed', 1),
 ('22nd', 1),
 ('religious', 1),
 ('preachin', 1),
 ('grateful', 1),
 ('audible', 1),
 ('exquisite', 1),
 ('m’appelle', 1),
 ('deceptive', 1),
 ('sixth', 1),
 ('uncomplaining', 1),
 ('cavernous', 1),
 ('Egyptian', 1),
 ('louder', 1),
 ('lost', 1),
 ('grammatical', 1),
 ('farther', 1),
 ('downstairs', 1),
 ('reddest', 1),
 ('twentieth', 1),
 ('scolding', 1),
 ('oaken', 1),
 ('safer', 1),
 ('midway', 1),
 ('convinced', 1),
 ('splendid', 1),
 ('vital', 1),
 ('later', 1),
 ('unobtrusive', 1),
 ('unguarded', 1),
 ('talking', 1),
 ('dopey', 1),
 ('invitin', 1),
 ('lighted', 1),
 ('countin', 1),
 ('synthetic', 1),
 ('patient', 1),
 ('righthand', 1),
 ('practicin', 1),
 ('High', 1),
 ('purple', 1),
 ('cheapest', 1),
 ('celebratory', 1),
 ('tight', 1),
 ('initial', 1),
 ('musky', 1),
 ('blonde', 1),
 ('generous', 1),
 ('untied', 1),
 ('whininess', 1),
 ('indestructible', 1),
 ('prehistoric', 1),
 ('virtual', 1),
 ('furlike', 1),
 ('cruel', 1),
 ('golden', 1),
 ('lifeless', 1),
 ('fragile', 1),
 ('Swiss', 1),
 ('neat', 1),
 ('downtown', 1),
 ('darker', 1),
 ('definite', 1),
 ('pronounced', 1),
 ('well', 1),
 ('overhead', 1),
 ('fragrant', 1),
 ('extreme', 1),
 ('magnificent', 1),
 ('positive', 1),
 ('elementary', 1),
 ('combative', 1),
 ('uncontrollable', 1),
 ('tolerable', 1),
 ('charming', 1),
 ('ugly', 1),
 ('respectable', 1),
 ('bold', 1),
 ('unflinching', 1),
 ('weary', 1),
 ('ivory', 1),
 ('Vivian', 1),
 ('thankful', 1),
 ('understandin', 1),
 ('concerned', 1),
 ('droopy', 1),
 ('newest', 1),
 ('gentle', 1),
 ('vague', 1),
 ('undeniable', 1),
 ('boastful', 1),
 ('spiritual', 1),
 ('goose', 1),
 ('recent', 1),
 ('pasture', 1),
 ('solo', 1),
 ('absurd', 1),
 ('elusive', 1),
 ('Higher', 1),
 ('Amazing', 1),
 ('satisfying', 1),
 ('lean', 1),
 ('unkind', 1),
 ('playin', 1),
 ('dotted', 1),
 ('unblemished', 1),
 ('occasional', 1),
 ('fullest', 1),
 ('respectful', 1),
 ('shadiest', 1),
 ('unpalatable', 1),
 ('childlike', 1),
 ('misspelled', 1),
 ('liquid', 1),
 ('garble', 1),
 ('NEW', 1),
 ('conscientious', 1),
 ('upside', 1),
 ('deceitful', 1),
 ('stale', 1),
 ('fallen', 1),
 ('civilized', 1),
 ('relaxed', 1),
 ('unnatural', 1),
 ('unmarried', 1),
 ('mild', 1),
 ('babble', 1),
 ('unpronounceable', 1),
 ('runaway', 1),
 ('confident', 1),
 ('eerie', 1),
 ('hospitable', 1),
 ('bedtime', 1),
 ('countless', 1),
 ('perfunctory', 1),
 ('sour', 1),
 ('hardest', 1),
 ('unrelated', 1),
 ('distinct', 1),
 ('heaviest', 1),
 ('scarred', 1),
 ('passin', 1),
 ('silklike', 1),
 ('everyday', 1),
 ('huh', 1),
 ('unnoticed', 1),
 ('havin', 1),
 ('babiest', 1),
 ('varied', 1),
 ('sympathetic', 1),
 ('stickin', 1),
 ('burnin', 1),
 ('barbed', 1),
 ('rockin', 1),
 ('everlastin', 1),
 ('sudden', 1),
 ('weak', 1),
 ('mysterious', 1),
 ('itinerant', 1),
 ('docile', 1),
 ('outside', 1),
 ('windowless', 1),
 ('quickest', 1),
 ('monotonous', 1),
 ('stampless', 1),
 ('deceased', 1),
 ('unpleasant', 1),
 ('innocent', 1),
 ('bloody', 1),
 ('oatmeal', 1),
 ('dripping', 1),
 ('tighter', 1),
 ('awful', 1),
 ('here', 1),
 ('True', 1),
 ('risky', 1),
 ('unforgiving', 1),
 ('unfair', 1),
 ('polite', 1),
 ('satisfactory', 1),
 ('mechanic', 1),
 ('holstered', 1),
 ('flustered', 1),
 ('purplish', 1),
 ('delicate', 1),
 ('stunning', 1),
 ('serrated', 1),
 ('alert', 1),
 ('unforced', 1),
 ('embarrassing', 1),
 ('understandable', 1),
 ('venetian', 1),
 ('lovesick', 1),
 ('thirsty', 1),
 ('humiliating', 1),
 ('valuable', 1)]

Then we make a dataframe from this list:

df = pd.DataFrame(adjs_tally.most_common(), columns=['character', 'count'])
df[:100]
character count
0 other 150
1 old 142
2 more 131
3 first 106
4 good 103
5 few 101
6 little 94
7 next 66
8 small 65
9 last 64
10 white 57
11 same 52
12 second 50
13 long 49
14 much 49
15 own 48
16 new 46
17 right 44
18 whole 42
19 dead 41
20 most 36
21 young 34
22 only 34
23 many 32
24 several 31
25 big 31
26 bad 30
27 such 28
28 front 26
29 black 26
30 retarded 26
31 dark 25
32 late 22
33 blue 21
34 nice 21
35 older 21
36 high 21
37 alone 21
38 sorry 21
39 best 20
40 large 20
41 back 20
42 real 20
43 able 20
44 empty 19
45 enough 18
46 ready 18
47 less 18
48 surprised 18
49 asleep 17
50 green 17
51 better 17
52 loud 17
53 damn 16
54 afraid 16
55 full 16
56 open 15
57 tiny 15
58 happy 15
59 5th 15
60 least 14
61 close 14
62 easy 14
63 quiet 14
64 thick 13
65 different 13
66 wrong 13
67 about 12
68 left 12
69 cold 12
70 naked 12
71 orange 12
72 half 11
73 glad 11
74 light 11
75 sure 11
76 Most 11
77 tired 11
78 third 10
79 red 10
80 oldest 10
81 early 10
82 true 10
83 tall 10
84 entire 9
85 assistant 9
86 thin 9
87 foreign 9
88 skinned 9
89 sittin 9
90 clean 8
91 beautiful 8
92 final 8
93 cute 8
94 silent 8
95 wooden 7
96 brown 7
97 certain 7
98 clear 7
99 younger 7

Get Nouns

POS

Description

Examples

NOUN

noun

girl, cat, tree, air, beauty

To extract and count nouns, we can follow the same model as above, except we will change our if statement to check for POS labels that match “NOUN”.

nouns = []
for token in document:
    if token.pos_ == 'NOUN':
        nouns.append(token.text)

nouns_tally = Counter(nouns)

df = pd.DataFrame(nouns_tally.most_common(), columns=['noun', 'count'])
df[:100]
noun count
0 man 297
1 mother 258
2 woman 251
3 time 246
4 father 182
5 day 182
6 way 163
7 people 156
8 door 146
9 145
10 boy 129
11 house 124
12 world 122
13 car 117
14 night 111
15 hand 110
16 girl 106
17 room 106
18 place 102
19 children 101
20 morning 100
21 years 98
22 something 87
23 head 86
24 life 84
25 table 80
26 street 77
27 store 75
28 child 69
29 week 69
30 things 68
31 women 68
32 apartment 68
33 thing 66
34 window 62
35 one 60
36 bed 58
37 nothing 57
38 front 57
39 money 57
40 everything 56
41 anything 55
42 words 53
43 light 51
44 eyes 51
45 days 51
46 son 51
47 daughter 50
48 side 50
49 floor 50
50 family 50
51 school 50
52 work 49
53 chair 49
54 men 48
55 face 48
56 name 48
57 baby 47
58 times 47
59 corner 46
60 arm 46
61 evening 46
62 home 46
63 church 45
64 months 44
65 line 44
66 year 44
67 pictures 44
68 kitchen 42
69 word 42
70 hands 41
71 heart 41
72 weeks 41
73 arms 40
74 brother 40
75 picture 40
76 back 39
77 boys 38
78 telephone 38
79 end 36
80 mind 35
81 friends 35
82 husband 35
83 hair 35
84 birds 34
85 group 34
86 someone 33
87 train 32
88 wife 31
89 feet 31
90 stuff 31
91 girls 31
92 hall 31
93 pigeons 30
94 job 30
95 daddy 29
96 month 29
97 city 29
98 coop 28
99 sound 26

Get Verbs

POS

Description

Examples

VERB

verb

run, runs, running, eat, ate, eating

To extract and count works of art, we can follow a similar-ish model to the examples above. This time, however, we’re going to make our code even more economical and efficient (while still changing our if statement to match the POS label “VERB”).

Python Review!

We can use a list comprehension to get our list of verbs in a single line of code! Closely examine the first line of code below:

verbs = [token.text for token in document if token.pos_ == 'VERB']

verbs_tally = Counter(verbs)

df = pd.DataFrame(verbs_tally.most_common(), columns=['verb', 'count'])
df[:100]
verb count
0 was 1377
1 had 1119
2 said 822
3 would 511
4 did 306
5 be 288
6 were 281
7 do 273
8 could 255
9 been 252
10 have 219
11 told 192
12 know 190
13 see 186
14 got 177
15 get 172
16 took 165
17 come 153
18 went 148
19 go 142
20 came 140
21 is 138
22 put 116
23 say 110
24 began 105
25 looked 105
26 knew 101
27 made 100
28 tell 100
29 thought 96
30 take 90
31 asked 90
32 wanted 85
33 can 83
34 saw 80
35 stood 79
36 make 78
37 turned 78
38 want 77
39 ’m 76
40 ai 73
41 think 70
42 called 69
43 look 67
44 found 64
45 left 64
46 seemed 62
47 sat 62
48 gone 59
49 done 57
50 going 57
51 give 54
52 standing 53
53 gave 52
54 lived 51
55 gon 51
56 heard 51
57 kept 50
58 looking 50
59 let 49
60 are 49
61 ’s 48
62 set 43
63 sitting 43
64 hear 43
65 find 43
66 being 42
67 call 40
68 opened 40
69 seen 40
70 taken 39
71 remember 39
72 watched 39
73 walked 38
74 might 38
75 moved 38
76 should 37
77 continued 36
78 work 35
79 held 35
80 stopped 35
81 used 34
82 pulled 34
83 waited 34
84 felt 33
85 happened 32
86 saying 32
87 live 32
88 waiting 31
89 leave 31
90 coming 31
91 read 30
92 Do 29
93 became 29
94 reached 29
95 returned 28
96 ask 28
97 sing 28
98 walk 27
99 crossed 27

Keyword Extraction

Get Sentences with Keyword

spaCy can also identify sentences in a document. To access sentences, we can iterate through document.sents and pull out the .text of each sentence.

We can use spaCy’s sentence-parsing capabilities to extract sentences that contain particular keywords, such as in the function below.

With the function find_sentences_with_keyword(), we will iterate through document.sents and pull out any sentence that contains a particular “keyword.” Then we will display these sentence with the keywords bolded.

import regex
from IPython.display import Markdown, display
def find_sentences_with_keyword(keyword, document):
    
    #Iterate through all the sentences in the document and pull out the text of each sentence
    for sentence in document.sents:
        sentence = sentence.text
        
        #Check to see if the keyword is in the sentence (and ignore capitalization by making both lowercase)
        if keyword.lower() in sentence.lower():
            
            #Use the regex library to replace linebreaks and to make the keyword bolded, again ignoring capitalization
            sentence = re.sub('\n', ' ', sentence)
            sentence = re.sub(f"{keyword}", f"**{keyword}**", sentence, flags=re.IGNORECASE)
            
            display(Markdown(sentence))
find_sentences_with_keyword(keyword="mother", document=document)

Her father had known Miles long before the girl was born, before the thought to marry her mother had even crossed his mind.

The men found her in the kitchen, her head buried in the lap of Miles’s mother, her arms tight around the waist of the old woman, who had been sitting at the table having Sunday lunch with her husband.

Buster,” Miles’s mother said to him,

One morning, toward four o’clock, they cut open her stomach and pulled out the child only moments after Clara died, mother and daughter passing each other as if along a corridor, one into death, the other into life.

There was a song about trains that kept rumbling in his head as the night wore on, a song his mother would sing when he was a boy.

The girl had never had an overwhelming curiosity about her mother, but it fascinated her to see the face of the lady in all the pictures on a woman who moved and laughed and did mother things.

On an otherwise unremarkable September morning, long before I learned to be ashamed of my mother, she takes my hand and we set off down New Jersey Avenue to begin my very first day of school.

My mother has uncharacteristically spent nearly an hour on my hair that morning, plaiting and replaiting so that now my scalp tingles.

Behind my ears, my mother, to stop my whining, has dabbed the stingiest bit of her gardenia perfume, the last present my father gave her before he disappeared into memory.

My mother does not believe that a girl in kindergarten needs such things, so I am taking them only because of my insistent whining and because they are presents from our neighbors, Mary Keith and Blondelle Harris.

Miss Mary and Miss Blondelle are watching my two younger sisters until my mother returns.

The women are as precious to me as my mother and sisters.

This is my mother:

When I say the word in fun to one of my sisters, my mother slaps me across the mouth and the word is lost for years and years.

In my neighborhood, I have many friends, but I see none of them as my mother

At I Street, between New Jersey Avenue and Third Street, we enter Seaton Elementary School, a timeworn, sad-faced building across the street from my mother

When, in answer to her question, my mother tells her that we live at 1227 New Jersey Avenue, the woman first seems to be picturing in her head where we live.

My mother shakes her head vigorously.

I want her to go here,” my mother says.

The woman continues to act as if she has known me all my life, but she tells my mother that we live beyond the area that Seaton serves.

My mother is not convinced and for several more minutes she questions the woman about why I cannot attend Seaton.

For as many Sundays as I can remember, perhaps even Sundays when I was in her womb, my mother has pointed across I Street to Seaton as we come and go to Mt. Carmel.

I am learning this about my mother: The higher up on the scale of respectability a person is—and teachers are rather high up in her eyes

This is my mother: She says, “

But it is not across the street from my mother’s church, her rock, one of her connections to God, and I sense her doubts as she absently rubs her thumb over the back of her hand.

” my mother asks a woman at one of the tables.

” my mother asks the woman, picking up a few pieces of the paper from the table.

I see,” my mother says, looking about the room.

The woman asks my mother what she means.

My mother looks at me, then looks away.

She finishes the form for her daughter and my mother

My mother is now diseased, according to the girl

’s eyes, and until the moment her mother takes her and the form to the front of the auditorium

, the girl never stops looking at my mother.

Don’t stare,” my mother says to me.

My mother answers the questions the woman reads off the form.

My mother slowly enunciates each word of my name.

This is my mother: As the questions go on, she takes from her pocketbook document after document, as if they will support my right to attend school, as if she has been saving them up for just this moment.

Few of the papers are needed here, but it does not matter and my mother continues to pull out the documents with the purposefulness of a magician pulling out a long string of scarves.

She has learned that money is the beginning and end of everything in this world, and when the woman finishes, my mother offers her fifty cents, and the woman accepts it without hesitation.

My mother and I are just about the last parent and child in the room.

My mother presents the form to a woman sitting in front of the stage, and the woman looks at it and writes something on a white card, which she gives to my mother.

Before long, the woman who has taken the girl with the drooping curls appears from behind us, speaks to the sitting woman, and introduces herself to my mother and me.

She’s to be my teacher, she tells my mother.

My mother stares.

We go into the hall, where my mother kneels down to me.

And even when the teacher turns me toward the classrooms and I hear what must be the singing and talking of all the children in the world, I can still hear my mother

But lately, for no reason at all, the teacher seemed to be in her face all the time, and that morning the teacher had made the death mistake of bringing up Cassandra’s father and mother.

On the other side of Rhonda was her mother, and behind Rhonda were two older brothers and her father, who peered out through thick glasses.

mother’ll pay twenty dollars.

Gladys explained that her mother wanted a few small boxes of stuff taken to Gladys’s father in Anacostia.

Taking boxes from the mother to the father all the way in Anacostia seemed strange to Cassandra, but being an orphan set adrift in the world she had learned that unless it could involve a death mistake, it was best to ask as few questions as possible.

Anita and Melanie must have heard it before, because they didn’t seem surprised at what the mother was saying.

: What does this little bitch have that my mother doesn’t have?

Besides, we still drivin on gas my mother paid for.”

The boys’ mother, waiting for a bus, ignored the drunk.

But Cassandra did not leave, and when the bus came and took the mother and her boys away, she began to worry that the drunk would spot her and come over and dance and make her the center of attention.

He was followed by his mother, who carried a switch as long as the boy.

” the mother hollered.

Anita’s mother was standing a few feet from the front of Rhonda’s house where two policemen, unsmiling, arms folded, stood as if they would never again do anything as important.

“Jeffrey shot Rhonda,” her mother said.

“Honey, it’s true,” Anita’s mother said to Cassandra and took the girl by the arm.

Anita’s mother said, “

Anita and her mother took Cassandra into the girl’s room.

Anita’s mother came in and gave Cassandra a cup of cocoa sitting in a saucer.

Anita watched as her mother helped Cassandra off with her clothes and into one of her mother’s nightgowns.

These are the times,” Carol would have joked, “when we miss our mothers most.” Manny, on the telephone, asked if he wanted to tend bar that evening and make some change.

The old man took the family’s picture, and when the mother raised one finger, the old man advanced the film and snapped again.

She smelled of garlic and, beneath that, of a soap that reminded him of the halls in the hospital where his mother had died.

His mother and father would bring everyone down to see the Washington they put on postcards and in the pages of expensive coffee-table books.

Never mind, they said, that he loved Caesar’s mother as much as he loved anyone and that stealing the flowers was his way of showing that love.

Caesar, though, was moved, and they grew closer after his mother’s funeral.

—a lace piece made by his grandmother that was on the back of the easy chair in the living room; a drawing of the house signed and dated by his sister

taped to the refrigerator; the kitchen curtains he had helped his mother put up.

Must be your mother, too.

The boy in the picture, dressed in green swimming trunks with his back to some ocean, could well be a grown man by now, or he could be in his grave, Caesar thought, but today, on that street, his mother was saying he needed five thousand dollars for an operation or he would die as sure as anything.

I lived with my mother, so rent and food weren’t a problem, though my brother, when he came around with that family of his, liked to get in my shit and tell me I should be looking for another job.

Usually, my mother was okay, but I could tell when my brother and his flat-butt wife had been around when I wasn’t there, because for days after that my mother would talk that same shit about me getting a job, like I’d never slaved a day before in my life.

My mother always said that every bullfrog praises his own poem, but I know for a natural fact that I was an excellent worker.

When my brother and I were in our early teens, my mother said this to us with the most seriousness she had ever said anything:

The worst that my mother feared didn’t happen to her baby boy that day.

Had he told me to deny my mother and father, I think I would have done that too.

My mother believed that a day had the best foundation if you had breakfast, so after she fixed our breakfast, and went off to work, I went back to bed and slept to about noon.

One night, all of us—me and Lonney and his mother and Brenda and her parents

Lonney’s mother stood up, in that eager, happy way women do when they want to greet their husbands home from a foreign land.

I think that he wanted to make things work with Brenda, if only to push the whole thing in his father’s face, but the foundation, as my mother would have said, was built on shifting sand.

In about a year or so he had separated from her, though he continued to be a good father to the child, a chubby little girl they named after his mother.

The next morning I forced myself to stay awake after my mother had left, then went off about eight o’clock to see what the place was about.

And so for a change I was up when my mother rose and I fixed our breakfast.

My mother would have sold me back into slavery for a good cup of coffee, and no one made it to her satisfaction the way she did.

My mother would fix me dinner

Then, before I could fall asleep in the chair, my mother would run water for me to take a bath, the same thing I had seen her do for my father so often when I was a child that I didn’t notice it anymore.

I just said yeah and good night and thanks, because my mother had always taught me and my brother that the currency of manners didn’t cost anything.

In the dark apartment, I could see that slice of light along the bottom of my mother’s closed door.

I learned in bits and pieces over time that she lived in an apartment on Neal Place a few doors from 5th Street, was a year out of Dunbar, was a secretary with the government people, that her family lived in a house on N Street that her mother’s parents had bought….

The mother seemed to live and die for her stories on television, and I rarely saw her either.

Gimme” meant the mother wanted it on credit.

I came back to Washington and dropped his mother off at her house and then went back to work, though Penny had said I could take the day off.

I waited until we were across O Street and asked with words that would have done my mother proud if I could take her to Howard Theater to see Dinah Washington that Saturday night.

Kentucky’s mother was the first Negro supervisor at some office in the Department of Commerce.

The second time I realized my allegiances were expanding, that I was making room in my soul for more than one neighborhood, was when I was asked to be godfather to two babies within one month; Penny got to be the godmother

One picture showed Joy Lambert, the mother of Patricia and Tommy Turner.

More and more I got to staying at her place, sleeping at my mother’s only a few times a month.

I hope you know what you doin,” my mother would say sometimes.

In fact, my mother said those very words that August Thursday night when I went to get clean clothes from her place.

my mother said her parents always made when she was a little girl.

’s mother, was there.

Then the ambulance arrived and in little or no time they took the girl and her mother away, the flashing light on the roof shining on all the houses as it moved down 5th Street.

Her mother,” Penny said, “wouldn’t let me come by to apologize.

All that month, and for some months after, that was the heart of the phone conversation, that the mother wouldn’t allow her to come to see her and the family.

Tommy came with her, and all the time they were in the store, the boy held his mother’s hand.

The boy never said a word the whole time, just stood there close to his mother, with his thumb in his mouth and blinking very, very slowly as if he were about to fall asleep on his feet.

Since Penny had told me to wipe the slate clean for Patricia’s mother, Old Lady Baxter became the biggest deadbeat.

My mother once told me that in place of muscles God gave women the ability to cry on a moment’s notice.

I could see that she did not approve, but she only had this look that my mother had the day my brother came home with the first piece of clothing my parents allowed him to buy on his own.

After a while I went to my mother’s place.

I found my mother at the kitchen table, listening to gospel on the radio.

I was twenty-seven years old, the age my mother was when she married.

On the crowded Stadium-Armory subway platform, she held Avis, the baby, by the hand, lest the girl wander off, and Marvin, the oldest, stood on his mother’s other side.

” he asked his mother.

My son the engineer, his mother thought.

Marcus and Avis managed to find seats just in front of their mother, and she was surprised when the dreadlock man sat down beside her.

Beside the boy was a woman as old as Marvella’s mother, asleep, her head leaning against the window.

I should tell him to turn around, his mother thought, but this might be one time when he’s justified.

She wrapped the strap of her pocketbook around her shoulder and held her daughter’s hand as they made their way two blocks up 12th Street to Horizon House, where Marvella’s mother lived.

They took the elevator up to her mother

” Avis, interested, leaned across her mother’s lap.

Marcus…” his mother said.

“Marcus,” his mother said.

My son the old man, his mother thought.

Well, it doesn’t matter,” his mother said.

She was in her mother

She closed her eyes and leaned back against her mother.

The boy looked at the man as if for help from his mother’s order.

By keeping the boys at Thompson School in Northwest two blocks from her mother, she worried less when the school day ended.

” his mother said just as Avis was about to take up the chant.

THE SUNDAY FOLLOWING mother’S DAY

When Madeleine Williams was four years old and her brother Sam was ten, their father killed their mother one night in early April.

If their mother sent forth to her children a cry of help, or of good-bye, they did not hear it, at least not on any conscious level, and they slept clear through to morning.

Judge John Tellingford chose to sentence Samuel Williams to twenty-five years at Lorton and, Madeleine found on page 164 of the transcript, the judge noted that for whatever demented reason the father had stabbed to death the mother of his children, he had seen fit to spare his children an equally horrendous fate.

In all the times she and Curtis took the girl home, she had seen the mother only a few times, and each time the mother had stood in her bathrobe in the entrance to the kitchen smoking a cigarette and watching as Madeleine said good-bye to the girl.

Whatcha takin up?” the mother asked each time.

In Madeleine’s junior year, Clarine’s father killed the mother with four pistol shots into the heart.

Madeleine could not remember what the mother looked like, and even when she looked down at the woman in her coffin, she could not connect the dead woman with the woman standing in her bathrobe.

The girl, along with the other children, had seen her father shoot her mother.

He named the place after his mother and aunt.

20 years of happy living isn’t long enough for a motherfucker like you,” Sam finally wrote his father.

A reconciliation was arranged by Hazel and Maddie on the Sunday before mother’s Day, which happened to be Maddie’s birthday.

Why would anyone in the world write to the man who killed their own mother?

She hugged him, clung to him, and for those moments they were no more or less than a boy and girl without a mother and father.

Madeleine and Curtis had planned to spend mother’s Day at the Children’s Center with their child, but Curtis’s father became ill during the week and Curtis went to Norfolk.

Madeleine would have waited until he returned to visit the center, but she did not want too much time to pass after mother’s Day before she saw her son.

On the Sunday morning following mother’s Day, she was ready to leave her apartment when the chime on her door rang.

I wanted to wish you a happy mother’s Day.

mother’s Day was last Sunday.

Happy mother’s Day.

On the lawn there was a family fussing over a fellow who could have been five or in his teens or a man of thirty or so, and who was sitting in the lap of a woman who may have been his mother.

the woman’s daughter, who stood beside an older brother, just behind their mother.

All right now,” their mother said, “Thas anough a that.”

If it was true what her mother had once told her, then nothing rang the telephone like death in the middle of the night.

We are very sorry to call at such a time, but your mother died twenty minutes ago.

Your mother was an exemplary patient,” the woman said.

Try to remember, she told herself, for all of it must go in the diary: On the night of June 29th, I was dreaming such-and-such and such-and-such when my mother passed away.

She thought she had been dreaming of Antibes and the naked Texan on the beach when her mother died, but the more she thought the more it seemed that Antibes had actually been on her mind when she came the first time hours ago.

her mother had said of the first boy she had ever slept with, though all her mother knew at that time was that the boy had taken her to a movie and Mile Long for a steak sandwich.

her mother had asked.

My mother has died and I have to go to her.”

He does not even know my name on the very night my mother has died, she thought, wanting him out of her bed and out of her house.

I can’t believe I’m walkin the same paths that my Lord walked,” her mother had said that second day in Israel, standing at the Church of Gethsemane.

Lydia had presented the trip to her mother on her sixty-fifth birthday and had even paid the way for Georgia Evans, her mother’s best friend.

Lydia had not wanted to go, but her mother had insisted.

Oh, Lydia,” her mother had said, “what would the Holy Land mean to me without you bein there with me?” Georgia, lazy, far less religious that her mother, had complained about all the hills.

Georgia would drink all the water she carried, then want more from Lydia and her mother.

My mother lies moldering

Why, Ms. Walsh, your mother lies moldering and you’ve been fucking.

She called a Capitol cab, because that was the company her mother had always used.

Trust Capitol to get you there and back in one piece,” her mother had said.

Once, as a girl, she had traveled with her mother

On that sixth morning in Israel, Georgia had gone back to her bed after breakfast and seemed unable to move, but she had insisted that Lydia and her mother go on without her.

When mother and daughter returned from floating on the Dead Sea, they found Georgia in a lounge chair at the swimming pool, descending into drunkenness, not quite certain who they were or where she was.

When Lydia’s mother told her that she was in the Holy Land, the land of Jesus, Georgia said, “Yes.

Then, after Lydia’s mother slapped her, Georgia asked her friend for forgiveness.

Her mother refused to say anything more to Georgia.

They left her and had dinner in the hotel dining room, but her mother only picked at her food.

Each place along the way, in Jericho, in Bethany, in Nazareth, she stayed close to Lydia’s mother, but Cornelia paid her no mind, and it showed on Georgia’s face.

On the living-room wall in Lydia’s town house, among the photographs of places she had visited around the world, between the pictures of her standing before the Kremlin on a winter day and of her in a cavernous room in a Danish castle the guide said was haunted, there was the picture of Lydia and Cornelia and Georgia standing where the tourist bureau said Joseph’s carpentry shop had been, Georgia in her hideous wig standing on one side and her mother in the middle.

My mother has died.

I know what it’s like to lose a mother.

She had wanted to buy a second town house in the area as another investment and have her mother and Georgia live there, but her mother told her that she did not know if she could live among so many white people.

I knew folks who lived in Southwest before they threw the colored out and made it for the wealthy,” her mother had said when Lydia told her she had bought a town house there.

You know, your mother….

Had she remembered to finally write down her mother’s recipe for that wondrous beef stew somewhere in one of the appointment books?

I’m sorry for all this,” her father had said on his death bed to her mother.

” her mother said to her father.

In one of the houses on that street her mother lived until Lydia’s last year of law school.

her mother had asked while the man was in the bathroom.

” John Brown lies molderin where my mother lies molderin….

The cab driver thought that her crying meant that maybe it had finally hit her that her mother had died and that soon his passenger would be coming to herself.

,” her mother told Lydia years later when she thought her daughter was old enough to understand, “she just became his slave.

” HIS mother’S HOUSE

When his mother moved into that 10th Street house Santiago Moses bought for her, one of the few things she brought from the place she left behind was the coffee table made of see-through glass and false wood.

Santiago Moses watched his mother, then he gave the table another playful tap.

My mother’s old anough to know what she want.

She and Humphrey’s mother, Pearl Malone, had been like sisters since they were ten years old.

She was on the second floor, in the room she had furnished with such care and set aside for her mother, who would not ever step foot in her daughter’s house.

The pictures of her mother started in the center of the coffee table display.

Her mother rarely smiled in the photographs and she often looked into the camera as if the eye had caught her doing something God would not approve of.

I just don’t take the good pictures,” her mother had said when she saw the display at the apartment Joyce had left behind.

The next time Joyce saw the boy and his mother, it was the day after they had delivered the pool table for the basement in the new house.

” he yelled to his mother as she turned the corner, and then he threw the car in her direction.

When Santiago called that afternoon, he told his mother it would be all right, that the boy would be gone by the evening.

But the boy, Adam, stayed three weeks, and no one ever again saw his mother.

The next day was Friday, and she put off going to see her mother because it was too nice an April day not to begin preparations for her backyard garden.

Three weeks after she had promised to come by the next day, Joyce finally got up the courage to visit her mother.

Her mother had not called back to remind her of the promise, which was her mother’s way.

As always, Joyce tried to anticipate all that her mother would throw at her and spent the walk down to her mother

Her mother lived in the Judiciary House, an H Street apartment building for senior citizens and the disabled.

Joyce let herself into the apartment with her own key and found her mother in the living room.

The small black-and-white television Joyce’s father had bought on credit twelve years ago was on, but her mother was reading a paperback book and paid no attention to the television.

Her mother, who did not look up when her daughter entered, was wearing, as usual when she was inside, one of her dead husband’s shirts and a pair of his checkered socks.

And as Joyce bent down to give her mother the kind of kiss on the cheek that Santiago would give Joyce, she smelled her father’s cheap musky aftershave.

I want you to give Sandy a message,” her mother said, taking off the eyeglasses she kept on a string around her neck.

In that second-floor bedroom in her 10th Street house she had set aside for her mother, there was a telephone with giant numbers so her mother would not have trouble dialing her friends.

“Santiago been comin by some nights and sleepin sida my bed like he did when he was young,” her mother said.

She sat across from her mother in a cheap metal chair that was part of the dinette set.

Well, maybe he’ll listen if the words come from his own mother.

I just this moment changed my mind,” her mother said.

Her mother put on her glasses and opened the book.

Joyce stood and went to the door, and her mother told her not to forget the money.

At a shoe store near 13th Street, she abandoned once and for all any hope she ever had that her mother would come to live with her and spend the rest of her life in the room on the second floor.

In one, taken months after his mother married, Humphrey was sleeping in a chair by a window.

Everyone said good morning except for the little boy, who sat beside his mother at the table, playing with a truck that was as big as his head.

She was, like her mother, a very plain and charming woman who could immediately put the world at ease.

And usually after the worst fights, she ran away to stay with Diane, until her husband came and knocked contritely at his mother-in-law

So it was often difficult for him to collect himself on Sunday morning, to pull himself away from a willing woman and clear his head of alcohol, and to go off to be with four other women, one of whom was old enough to be his grandmother, two of whom were old enough to be his mother, and one of whom had a ring through her nose about another man.

’s grandmother was Maude Townsend, a blind woman who lived at Claridge Towers on M Street Northwest, an apartment house for old people and the disabled.

Then he said, “mother,” which was what he called his wife.

He said “mother” once more, took up the Bible, walked to the edge of the pulpit, and extended his hand, as if for help, to his wife.

Only those near him could hear him say “mother” yet again.

The old man’s son had come to help his father as well, but his mother shook her head and pointed to the pulpit.

Once, as she was singing at the Virgin mother Baptist Church on Kentucky Avenue, Jesus had come down the aisle and sat down in a pew near the front.

I don’t wanna hear another word outta your mouth till your mother get home.

Then he would pass the newspaper among his children and tell them to read him the headlines, and each one would hold it uncomfortably and repeat what their mother had said.

they did not know about, that the friend’s mother, like any good mother, would soon send their daughter home.

Of the eight children he had had with Woodrow’s mother and the five he had had with other women, only Woodrow, Alice, and a half-brother who lived down the road from the old man came to the funeral.

In a wooden trunk in one of the back rooms, Woodrow found several pictures of his mother.

He had not seen his mother’s face in more than forty years, had thought his father had destroyed all the pictures of her.

Woodrow’s mother sat at the far end of the table, between two of her daughters.

Maybe a half hour or more into that storm, I started seein this figure, this thing, that kinda just stood in that corn patch in my mother’s garden.

My Uncle Joe, his wife Ebbie, her mother, and my Uncle Ray.

You could see where Aunt Ebbie’s mother had been rockin the crib with Aunt Ebbie’s baby in it.

A wind was coming down the chimney and through the door, and it was rockin the crib, rockin Aunt Ebbie’s mother’s hand

I put Aunt Ebbie’s mother’s hand in her lap.

They do the job your mother and father used to do, huh?

But when the other receptionist said it was the old lady in the blue coat, the guard held back for the longest time, as if to grab her would be like arresting his own grandmother.

But I was born right here in D.C. Born, bred, and buttered, my mother used to say.

Maybe thas what I got from listenin to my father readin and to my mother

My mother had this idea that everything could be done in Washington, that a human bein could take all they troubles to Washington and things would be set right.

When I went back home to visit that first time and told my mother all about my livin in Washington, she fell into such a cry, like maybe I had managed to make it to heaven without dyin.

She had not played another tape since the one about her mother

This white family my mother worked for, they had a son married and gone to live in Baltimore.

So he wrote to his mother and she asked my mother and my mother asked me about goin to live in Baltimore.

Hadn’t ever been kissed, hadn’t ever been anything, but here I was goin farther from home than my mother and father put together….

Get Keyword in Context

We can also find out about a keyword’s more immediate context — its neighboring words to the left and right — and we can fine-tune our search with POS tagging.

To do so, we will first create a list of what’s called ngrams. “Ngrams” are any sequence of n tokens in a text. They’re an important concept in computational linguistics and NLP. (Have you ever played with Google’s Ngram Viewer?)

Below we’re going to make a list of bigrams, that is, all the two-word combinations from Lost in the City. We’re going to use these bigrams to find the neighboring words that appear alongside particular keywords.

#Make a list of tokens and POS labels from document if the token is a word 
tokens_and_labels = [(token.text, token.pos_) for token in document if token.is_alpha]
#Make a function to get all two-word combinations
def get_bigrams(word_list, number_consecutive_words=2):
    
    ngrams = []
    adj_length_of_word_list = len(word_list) - (number_consecutive_words - 1)
    
    #Loop through numbers from 0 to the (slightly adjusted) length of your word list
    for word_index in range(adj_length_of_word_list):
        
        #Index the list at each number, grabbing the word at that number index as well as N number of words after it
        ngram = word_list[word_index : word_index + number_consecutive_words]
        
        #Append this word combo to the master list "ngrams"
        ngrams.append(ngram)
        
    return ngrams
bigrams = get_bigrams(tokens_and_labels)

Let’s take a peek at the bigrams:

bigrams[5:20]
[[('Her', 'DET'), ('father', 'NOUN')],
 [('father', 'NOUN'), ('would', 'VERB')],
 [('would', 'VERB'), ('say', 'VERB')],
 [('say', 'VERB'), ('years', 'NOUN')],
 [('years', 'NOUN'), ('later', 'ADV')],
 [('later', 'ADV'), ('that', 'ADP')],
 [('that', 'ADP'), ('she', 'PRON')],
 [('she', 'PRON'), ('had', 'VERB')],
 [('had', 'VERB'), ('dreamed', 'VERB')],
 [('dreamed', 'VERB'), ('that', 'DET')],
 [('that', 'DET'), ('part', 'NOUN')],
 [('part', 'NOUN'), ('of', 'ADP')],
 [('of', 'ADP'), ('it', 'PRON')],
 [('it', 'PRON'), ('that', 'ADP')],
 [('that', 'ADP'), ('she', 'PRON')]]

Now that we have our list of bigrams, we’re going to make a function get_neighbor_words(). This function will return the most frequent words that appear next to a particular keyword. The function can also be fine-tuned to return neighbor words that match a certain part of speech by changing the pos_label parameter.

def get_neighbor_words(keyword, bigrams, pos_label = None):
    
    neighbor_words = []
    keyword = keyword.lower()
    
    for bigram in bigrams:
        
        #Extract just the lowercased words (not the labels) for each bigram
        words = [word.lower() for word, label in bigram]        
        
        #Check to see if keyword is in the bigram
        if keyword in words:
            
            for word, label in bigram:
                
                #Now focus on the neighbor word, not the keyword
                if word.lower() != keyword:
                    #If the neighbor word matches the right pos_label, append it to the master list
                    if label == pos_label or pos_label == None:
                        neighbor_words.append(word.lower())
    
    return Counter(neighbor_words).most_common()
get_neighbor_words("mother", bigrams)
[('my', 86),
 ('her', 75),
 ('his', 39),
 ('and', 27),
 ('had', 24),
 ('said', 18),
 ('the', 18),
 ('would', 12),
 ('was', 9),
 ('day', 8),
 ('your', 7),
 ('she', 6),
 ('their', 6),
 ('has', 5),
 ('anita', 4),
 ('told', 4),
 ('in', 4),
 ('died', 3),
 ('church', 3),
 ('says', 3),
 ('i', 3),
 ('as', 3),
 ('who', 3),
 ('only', 3),
 ('hand', 3),
 ('thought', 3),
 ('lived', 3),
 ('he', 3),
 ('a', 3),
 ('lies', 3),
 ('lydia', 3),
 ('ebbie', 3),
 ('miles', 2),
 ('but', 2),
 ('to', 2),
 ('does', 2),
 ('when', 2),
 ('is', 2),
 ('asks', 2),
 ('before', 2),
 ('wanted', 2),
 ('put', 2),
 ('of', 2),
 ('with', 2),
 ('old', 2),
 ('once', 2),
 ('at', 2),
 ('marvella', 2),
 ('lap', 2),
 ('following', 2),
 ('asked', 2),
 ('own', 2),
 ('happy', 2),
 ('it', 2),
 ('woodrow', 2),
 ('did', 1),
 ('things', 1),
 ('returns', 1),
 ('slaps', 1),
 ('tells', 1),
 ('shakes', 1),
 ('that', 1),
 ('what', 1),
 ('looks', 1),
 ('takes', 1),
 ('answers', 1),
 ('slowly', 1),
 ('continues', 1),
 ('offers', 1),
 ('presents', 1),
 ('stares', 1),
 ('kneels', 1),
 ('footsteps', 1),
 ('paid', 1),
 ('boys', 1),
 ('waiting', 1),
 ('hollered', 1),
 ('took', 1),
 ('came', 1),
 ('helped', 1),
 ('nightgowns', 1),
 ('raised', 1),
 ('caesar', 1),
 ('funeral', 1),
 ('too', 1),
 ('so', 1),
 ('always', 1),
 ('feared', 1),
 ('believed', 1),
 ('lonney', 1),
 ('stood', 1),
 ('rose', 1),
 ('closed', 1),
 ('parents', 1),
 ('seemed', 1),
 ('off', 1),
 ('proud', 1),
 ('kentucky', 1),
 ('girl', 1),
 ('away', 1),
 ('penny', 1),
 ('patricia', 1),
 ('place', 1),
 ('other', 1),
 ('this', 1),
 ('asleep', 1),
 ('apartment', 1),
 ('kinda', 1),
 ('order', 1),
 ('one', 1),
 ('sent', 1),
 ('looked', 1),
 ('spend', 1),
 ('after', 1),
 ('such', 1),
 ('two', 1),
 ('passed', 1),
 ('knew', 1),
 ('on', 1),
 ('best', 1),
 ('go', 1),
 ('slapped', 1),
 ('refused', 1),
 ('recipe', 1),
 ('house', 1),
 ('moved', 1),
 ('then', 1),
 ('humphrey', 1),
 ('pearl', 1),
 ('started', 1),
 ('rarely', 1),
 ('because', 1),
 ('way', 1),
 ('thinking', 1),
 ('there', 1),
 ('married', 1),
 ('which', 1),
 ('say', 1),
 ('yet', 1),
 ('shook', 1),
 ('virgin', 1),
 ('baptist', 1),
 ('get', 1),
 ('friend', 1),
 ('like', 1),
 ('good', 1),
 ('face', 1),
 ('sat', 1),
 ('garden', 1),
 ('used', 1),
 ('askin', 1),
 ('all', 1),
 ('idea', 1),
 ('worked', 1)]
get_neighbor_words("mother", bigrams, pos_label='ADJ')
[('own', 2),
 ('closed', 1),
 ('proud', 1),
 ('other', 1),
 ('asleep', 1),
 ('happy', 1),
 ('such', 1),
 ('best', 1),
 ('old', 1),
 ('good', 1)]

Your Turn!

Try out find_sentences_with_keyword() and get_neighbor_words with your own keywords of interest.

find_sentences_with_keyword(keyword="YOUR KEY WORD", document=document)
get_neighbor_words(keyword="YOUR KEY WORD", bigrams, pos_label=None)