Skip to content
  • Encoding Text as Numbers · Data Science

Encoding Text as Numbers

LearnViaNotes @learnvianotes· 3 min read

Models work with numbers, not words. The simplest encoding gives every word in the vocabulary an ID — then each sentence becomes a list of IDs, and can be turned back.

Build a vocabulary

Python
corpus = ("India, officially the Republic of India, is a country in South Asia. "
          "It is the seventh-largest country by area, the second-most populous country, "
          "and the most populous democracy in the world. "
          "In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives.")

from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords

stop = set(stopwords.words("english"))
def keep(word):
    return word.lower() not in stop and len(word) >= 2      # drop stop words and punctuation

words = [w.lower() for w in word_tokenize(corpus) if keep(w)]
vocab = sorted(set(words))                                   # sorted → same IDs every run
word_to_num = {w: i for i, w in enumerate(vocab, start=1)}
num_to_word = {i: w for w, i in word_to_num.items()}
print(len(vocab), vocab[:4])                                 # 18 ['area', 'asia', 'country', 'democracy']

Encode and decode

Python
corpus = ("India, officially the Republic of India, is a country in South Asia. "
          "It is the seventh-largest country by area, the second-most populous country, "
          "and the most populous democracy in the world. "
          "In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives.")

from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords

stop = set(stopwords.words("english"))
keep = lambda w: w.lower() not in stop and len(w) >= 2
vocab = sorted({w.lower() for w in word_tokenize(corpus) if keep(w)})
word_to_num = {w: i for i, w in enumerate(vocab, start=1)}
num_to_word = {i: w for w, i in word_to_num.items()}

encoded = [[word_to_num[w.lower()] for w in word_tokenize(s) if keep(w)] for s in sent_tokenize(corpus)]
print(encoded[0])                                            # [5, 10, 12, 5, 3, 15, 2]
print(" ".join(num_to_word[n] for n in encoded[0]))          # india officially republic india country south asia

Keeping every word

To decode back to the full sentence, keep stop words and drop only punctuation. The vocabulary grows, but nothing is lost except capitalisation and punctuation:

Python
corpus = ("India, officially the Republic of India, is a country in South Asia. "
          "It is the seventh-largest country by area, the second-most populous country, "
          "and the most populous democracy in the world. "
          "In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives.")

from nltk.tokenize import word_tokenize, sent_tokenize
is_word = lambda w: any(c.isalpha() for c in w)
vocab = sorted({w.lower() for w in word_tokenize(corpus) if is_word(w)})
word_to_num = {w: i for i, w in enumerate(vocab, start=1)}
num_to_word = {i: w for w, i in word_to_num.items()}
first = sent_tokenize(corpus)[0]
ids = [word_to_num[w.lower()] for w in word_tokenize(first) if is_word(w)]
print(" ".join(num_to_word[i] for i in ids))   # india officially the republic of india is a country in south asia

Free for students and teachers

Write notes like this one

Cards for every idea, quizzes and flashcards, read aloud — on the web and your phone.

Something wrong with this note? Open it to report