LSTM Networks

Long Short-Term Memory (LSTM) Networks

LSTMs are a highly modified version of RNNs specifically engineered to solve the Vanishing Gradient problem and remember long-term dependencies.

The Architecture of an LSTM Cell

While an RNN has a simple single neural net layer (like tanh) inside its repeating module, an LSTM cell contains four interacting layers wrapped into "gates" that control the flow of information.

• 1. Cell State (The Conveyor Belt): The core memory straight down the middle of the cell. Information flows along it smoothly with minor linear interactions.
• 2. Forget Gate: Decides what information from the past memory we should throw away or forget (outputs between 0 and 1).
• 3. Input Gate: Decides what new information from the current word we should add to our memory.
• 4. Output Gate: Decides what part of the memory we should output as our hidden state for this time step.

Level 1 — Implementing LSTMs for NLP

Because Keras and PyTorch hide the complex gate math, implementing an LSTM is as simple as importing the layer.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Embedding, Dropout

model = Sequential([
    Embedding(input_dim=20000, output_dim=128),
    # The LSTM layer replaces the SimpleRNN layer. 
    # It has 128 internal units managing the Cell State and Gates.
    LSTM(128, return_sequences=True), 
    Dropout(0.2), # Dropout helps prevent overfitting
    
    # We can stack LSTMs by returning sequences from the first one
    LSTM(64),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

Level 2 — Uses of LSTMs

For roughly 5 years (2013-2018), LSTMs were the undisputed kings of NLP before Transformers took over. They were used for: