A modern approach to AI · Chapter 24

Deep Learning for Natural Language Processing

Neural representations and sequence architectures learn useful language patterns from examples, with transformers enabling parallel attention across context.

Chapter guide

  1. 24.1 Word Embeddings — Dense vectors encode learned similarities and relations, though similarity does not ensure identical meaning.
  2. 24.2 Recurrent Neural Networks for NLP — Recurrent models update a hidden state as tokens arrive, representing sequence context step by step.
  3. 24.3 Sequence-to-Sequence Models — An encoder-decoder maps an input sequence to an output sequence, often with attention.
  4. 24.4 The Transformer Architecture — Self-attention mixes information across token positions; feed-forward layers and residual paths refine representations.
  5. 24.5 Pretraining and Transfer Learning — Pretraining learns broad representations from large data; adaptation targets a task or domain.
  6. 24.6 State of the art — Capabilities evolve rapidly; evaluation, data quality, compute, and limitations remain central.

Interactive: self-attention map

Select a query token to see illustrative attention weights over a short sentence. Each row is normalized; this is a teaching visualization, not a trained model.

How to read it. A brighter cell means more weight assigned to that key position for the selected query; attention weights alone do not explain a model's full reasoning.

Query, key, and normalized weights

In self-attention, a token representation supplies a query that is compared with each token's key; the resulting scores are normalized into weights and used to combine value vectors. The demo omits learned vectors and uses a hand-designed distance rule: for selected index i and key index j, its score is exp(−0.9|i−j|), plus 0.4 when i=j. Dividing by the sum makes the row total 100%. For query “learn,” neighboring words receive more weight than distant words; selecting another query recomputes the row.

A word embedding is a dense learned vector; an RNN carries a state token by token, whereas self-attention directly mixes information across positions. This five-token picture illustrates only a single query row, not a trained transformer.

Complexity and scope

Let L be the number of tokens. This demo computes and displays one row of L weights in O(L) time and O(L) space (including the visible cells); its fixed sentence has L=5. For all queries, forming the full dot-product score map costs O(L²dₖ) arithmetic when each key has dimension dₖ and materializing it takes O(L²) score storage. A transformer layer performs additional projection and feed-forward work; this visualization is not a runtime model and does not measure those costs.