Word Embeddings

Introduction to Word Embeddings

We've looked at One-Hot, BoW, and TF-IDF encoding. All of these generate Sparse Vectors (mostly zeros) where the length of the vector is equal to the massive size of the vocabulary (50k+ dimensions). Word Embeddings represented a paradigm shift in 2013: migrating from Sparse Vectors to Dense Vectors.

The State of the Art: The "Big 3" Static Embeddings

Word2Vec Formulations

Introduced by Tomas Mikolov at Google in 2013, Word2Vec is not a single algorithm, but a 2-layer Neural Network framework containing two distinct architectural flavors: CBOW and Skip-Gram. The secret? It learns embeddings as a byproduct of a "fake" classification task.

from gensim.models import Word2Vec
from nltk.tokenize import word_tokenize

# Sample corpus (should be massive in reality!)
corpus = [
    "Machine learning is the study of computer algorithms.",
    "Deep learning is a subset of machine learning using neural networks.",
    "Artificial intelligence creates smart machines."
]

# Tokenize sentences into lists of words
tokenized_corpus = [word_tokenize(sentence.lower()) for sentence in corpus]

# 1. Train Word2Vec Model
# vector_size: dimension of embedding (usually 100-300)
# window: context window size
# sg: 1 for Skip-Gram, 0 for CBOW
model = Word2Vec(sentences=tokenized_corpus, vector_size=10, window=3, min_count=1, sg=1)

# 2. Extract the learned vector for a word
print("Vector for 'learning':")
print(model.wv['learning'])

# 3. Find most similar semantic words!
print("\nMost similar to 'learning':")
print(model.wv.most_similar('learning', topn=3))

GloVe (Global Vectors)

Created by Stanford researchers in 2014, GloVe (Global Vectors for Word Representation) sought to combine the best of both previous NLP worlds. It argues that while predictive models like Word2Vec are great, they only look at a small window of local context. They miss out on the global statistics of the entire document corpus.

The Ratio of Probabilities

The mathematical genius of GloVe lies in looking at ratios of co-occurrence probabilities rather than pure probabilities.

Let's look at the words Ice and Steam. We want to see how they interact with probe words like Solid and Gas.

Takeaway: The ratio clearly discriminates the relevant thermodynamic properties (Solid vs Gas) while canceling out words that appear frequently with both (Water) or rarely with both (Fashion). GloVe forces the model to learn these ratios to encode meaning.

# Rather than training GloVe from scratch (which takes massive resources),
# industry standard is to download pre-trained vectors. 
# spaCy's 'en_core_web_md' or 'lg' come bundled with GloVe vectors.

import spacy

# Load Medium model (contains 20k GloVe vectors of 300 dimensions)
nlp = spacy.load("en_core_web_md")

word1 = nlp("king")
word2 = nlp("queen")
word3 = nlp("apple")

# Extract the dense 300D GloVe numpy array
print(f"Vector shape for 'king': {word1.vector.shape}")
# >>> (300,)

# spaCy automatically calculates Cosine Similarity using these GloVe vectors!
print(f"Similarity (King vs Queen): {word1.similarity(word2):.3f}")
# >>> 0.725

print(f"Similarity (King vs Apple): {word1.similarity(word3):.3f}")
# >>> 0.204

FastText: Subword Embeddings

Created by Facebook's AI Research (FAIR) lab in 2016, FastText is an extension of the Word2Vec model. While Word2Vec and GloVe treat every word as a distinct, atomic entity, FastText breaks words down into smaller pieces called character n-grams.

from gensim.models import FastText

corpus = [["hello", "world", "this", "is", "nlp"], 
          ["machine", "learning", "is", "awesome"]]

# Train FastText
# min_n and max_n control the character n-gram sizes
model = FastText(sentences=corpus, vector_size=10, 
                 window=3, min_count=1, min_n=3, max_n=6)

# The model has never seen "learnings", but it can 
# calculate a vector anyway based on "learn" + "ing" + "s"!
oov_word = "learnings"

# This works perfectly, unlike Word2Vec!
vector = model.wv[oov_word] 
print(f"Vector for {oov_word} generated successfully!")

ELMo: Contextual Embeddings

ELMo (Embeddings from Language Models), introduced in 2018 by AllenNLP, marked the critical turning point in NLP history: the shift from entirely Static Embeddings (Word2Vec/GloVe) to fully Contextual Embeddings.

How ELMo Solves This

ELMo does not use a fixed dictionary lookup. Instead, ELMo calculates the embedding for a word on-the-fly by looking at the entire sentence it lives in.