A modern approach to AI · Chapter 23

Natural Language Processing

Language technologies connect symbolic structure and statistical evidence to interpret and produce human language.

Chapter guide

  1. 23.1 Language Models — Estimate how probable a word sequence is, often by predicting the next token from context.
  2. 23.2 Grammar — A grammar specifies structural patterns; formal rules can describe infinitely many sentences compactly.
  3. 23.3 Parsing — A parser builds candidate structures for a sentence, sometimes preserving multiple interpretations.
  4. 23.4 Augmented Grammars — Feature constraints and semantic actions enrich grammar rules with agreement or meaning-related information.
  5. 23.5 Complications of Real Natural Language — Ambiguity, idioms, context, variation, and noisy input challenge clean formal models.
  6. 23.6 Natural Language Tasks — Translation, information extraction, dialogue, question answering, and generation combine language components.

Structure can constrain interpretation, while probabilistic models rank alternatives using data and context.

Interactive: next-token context

A tiny illustrative language model changes its candidate distribution with the preceding words. Probabilities are pedagogical, not corpus estimates.

Current context

Choose a context by predicting.

Reading a model

The bars sum to 100%. A high probability is a relative prediction, not a guarantee of truth, grammaticality, or appropriate use. Parsing asks a different question: what structures could explain the observed sequence?

Terms and a short trace

A token is a model's unit of text; a context is preceding input; a language model assigns scores or probabilities to continuations. A grammar describes structural possibilities, while a parse is one candidate structure. The demo fixes its first context to “The astronomer observed” and displays candidate probabilities 52%, 31%, and 17%. Pressing Predict advances to a predefined context; it does not sample a token or infer a new distribution.

Reading a value such as 0.52 means “52% of this illustrative next-token distribution,” not “the statement is 52% true.” Real systems may tokenize words into subword units, use much larger learned models, and condition on broader context.

Demo complexity

Let C be the number of stored contexts, K the maximum candidates in one context, and P the current prefix length in characters. Rendering/advancing takes O(K+P) time to update the candidate bars and context, with O(K+P) temporary/DOM space for those visible values; the stored fixture uses O(CK + total prefix characters). Here C=3 and K=3 and all prefixes are short, so each update is constant-sized in practice. These bounds describe this visualization—not training or running a real language model, whose cost depends on model parameters and context length.