Before a computer can work with text, it has to be cut into pieces (tokens) and words have to be brought to a common form.
Why not just split()?
txt = "Hello Geeks. We're hoping you guys are doing great."
print(len(txt.split("."))) # 3 ← counts an empty piece after the last "."
print(txt.split()[:3]) # ['Hello', 'Geeks.', "We're"] ← punctuation stuck to wordsTokenizing with NLTK
from nltk.tokenize import sent_tokenize, word_tokenize
txt = "Hello Geeks. We're hoping you guys are doing great."
print(len(sent_tokenize(txt))) # 2
print(word_tokenize(txt)) # ['Hello', 'Geeks', '.', 'We', "'re", 'hoping', 'you', 'guys', 'are', 'doing', 'great', '.']
print([w for w in word_tokenize(txt) if w.isalnum()]) # ['Hello', 'Geeks', 'We', 'hoping', 'you', 'guys', 'are', 'doing', 'great']pip install nltk
python -m nltk.downloader punkt punkt_tab stopwords wordnet omw-1.4Stemming vs lemmatization
from nltk.stem import PorterStemmer, WordNetLemmatizer
stem, lem = PorterStemmer(), WordNetLemmatizer()
words = ["change", "changes", "changed", "studies", "better"]
print([stem.stem(w) for w in words]) # ['chang', 'chang', 'chang', 'studi', 'better']
print([lem.lemmatize(w) for w in words]) # ['change', 'change', 'changed', 'study', 'better']
print(lem.lemmatize("changed", pos="v"), lem.lemmatize("better", pos="a")) # change good| Stemming | Lemmatization | |
|---|---|---|
| How | Chops suffixes by rules | Looks words up in a dictionary (WordNet) |
| Output | May not be a real word (chang) | Always a real word (change) |
| Speed | Fast | Slower |
| Needs part of speech | No | For best results (pos="v" verb, "a" adjective) |