Skip to content
  • Stop Words · Data Science

Stop Words

LearnViaNotes @learnvianotes· 1 min read

Stop words are very common words (“is”, “the”, “a”) that carry little meaning on their own. Removing them shrinks the text and highlights the words that matter.

Python
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

stop_words = set(stopwords.words("english"))
print(len(stop_words), "is" in stop_words, "good" in stop_words)   # 198 True False

txt = "This is not a good time to talk."
kept = [w for w in word_tokenize(txt) if w.lower() not in stop_words and len(w) >= 2]
print(kept)                                  # ['good', 'time', 'talk']

Stop words in spaCy

Python
import spacy
nlp = spacy.load("en_core_web_sm")          # python -m spacy download en_core_web_sm

print(len(nlp.Defaults.stop_words) > 300)   # True
print(nlp.vocab["is"].is_stop, nlp.vocab["GFG"].is_stop)   # True False

nlp.Defaults.stop_words.add("i.e")          # add your own
nlp.vocab["i.e"].is_stop = True
nlp.Defaults.stop_words.discard("done")     # or remove one
nlp.vocab["done"].is_stop = False
print(nlp.vocab["i.e"].is_stop, nlp.vocab["done"].is_stop)  # True False

Free for students and teachers

Write notes like this one

Cards for every idea, quizzes and flashcards, read aloud — on the web and your phone.

Something wrong with this note? Open it to report