Text Representation

One-Hot Encoding in NLP

One-Hot Encoding is the most fundamental way to convert categorical data, such as strings of words, into numerical vectors that machine learning algorithms can understand. It represents every word in a vocabulary as a binary vector with all 0s and a single 1.

from sklearn.preprocessing import OneHotEncoder
import numpy as np

# 1. Define our vocabulary documents
words = np.array([['cat'], ['dog'], ['barks'], ['meows']])

# 2. Initialize the encoder (sparse=False returns a dense array for readability)
encoder = OneHotEncoder(sparse_output=False)

# 3. Fit and transform the words
one_hot_vectors = encoder.fit_transform(words)

# Display the categories and their vectors
print("Vocabulary Categories:", encoder.categories_[0])
print("\nVectors:")
for i, word in enumerate(words):
    print(f"{word[0]:<6} -> {one_hot_vectors[i]}")

'''
Output:
Vocabulary Categories: ['barks' 'cat' 'dog' 'meows']

Vectors:
cat    -> [0. 1. 0. 0.]
dog    -> [0. 0. 1. 0.]
barks  -> [1. 0. 0. 0.]
meows  -> [0. 0. 0. 1.]
'''

N-grams

Bag of Words (BoW) completely destroys sentence structure because it treats every word as an independent token (unigram). N-grams solve this by taking contiguous sequences of n words from a given text. This allows the model to capture local context and short grammatical structures.

Why are N-grams crucial for NLP?

from sklearn.feature_extraction.text import CountVectorizer

docs = ["The food is not good but the service is very good"]

# ngram_range=(min_n, max_n). 
# Example (2,2) means ONLY Bigrams. (1,2) means Unigrams AND Bigrams.
vectorizer = CountVectorizer(ngram_range=(2, 3))

X = vectorizer.fit_transform(docs)

# Print all extracted N-gram features
features = vectorizer.get_feature_names_out()
print("Extracted N-grams:")
for f in features:
    print(f"- '{f}'")

'''
Output:
Extracted N-grams:
- 'but the'
- 'but the service'
- 'food is'
- 'food is not'
...
- 'is not good'   <-- Trigram successfully captured the true sentiment
- 'not good'      <-- Bigram captured negation
- 'very good'
'''

TF-IDF (Term Frequency - Inverse Document Frequency)

Bag of Words evaluates the frequency of words. However, frequency isn't always the best indicator of importance. If you analyze 1,000 news articles, the word "the" will appear 10,000 times. A standard BoW model will wrongly assume "the" is the most important topic of the articles!

TF-IDF solves this. It relies on a simple premise: a word is highly important if it appears frequently in a specific document, but rarely across the entire corpus (all documents).

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

# Notice that the word 'this' and 'document' appear in multiple documents!
docs = [
    "This is a document about machine learning.",
    "This document covers natural language processing.",
    "I love baking sweet pastries, a completely unrelated topic."
]

# Initialize TF-IDF
vectorizer = TfidfVectorizer(stop_words='english')
tfidf_matrix = vectorizer.fit_transform(docs)

# Get terms
terms = vectorizer.get_feature_names_out()

df = pd.DataFrame(tfidf_matrix.toarray(), columns=terms, index=['Doc 1 (ML)', 'Doc 2 (NLP)', 'Doc 3 (Baking)'])

# Format for rounded float readability
pd.set_option('display.float_format', lambda x: '%.2f' % x)
print(df[['document', 'learning', 'machine', 'baking']])

'''
Output:
                document  learning  machine  baking
Doc 1 (ML)          0.47      0.62     0.62    0.00
Doc 2 (NLP)         0.47      0.00     0.00    0.00
Doc 3 (Baking)      0.00      0.00     0.00    0.45

Notice: 'document' has a LOWER score (0.47) than 'machine' (0.62) in Doc 1. 
Why? Because 'document' appears in both Doc 1 and Doc 2, so the IDF penalty lowered its uniqueness score!
'''

Co-occurrence Matrix

Bag-of-Words and TF-IDF create Document-Term matrices (Documents as rows, Words as columns). In contrast, a Co-occurrence Matrix creates a Word-Word matrix (Words as rows, Words as columns). It captures how often two different words appear together within a specific "window" distance in a sentence.

This follows the distribution hypothesis by famous linguist J.R. Firth: "You shall know a word by the company it keeps." Words that appear in similar contexts usually share semantic meaning.