AI · DEEP LEARNING

21 · Deep Learning

Deep learning composes layers of differentiable computation to learn useful representations. Backpropagation efficiently computes how each parameter contributes to prediction error.

21.1 Simple Feedforward Networks

Feedforward networks pass inputs through weighted layers and nonlinear activations. Hidden units allow them to represent patterns beyond a single linear decision boundary.

21.2 Computation Graphs for Deep Learning

A computation graph represents a model as operations on values. The chain rule propagates derivatives backward to calculate gradients for learning.

21.3 Convolutional Networks

Convolutions reuse local filters across an input, exploiting spatial structure in images and reducing the number of independent parameters.

21.4 Learning Algorithms

Gradient-based optimizers update parameters to reduce a loss. Mini-batches, learning rates, initialization, and normalization affect training behavior.

21.5 Generalization

Training accuracy alone does not ensure performance on new data. Validation, regularization, and appropriate data splits help measure and improve generalization.

21.6 Recurrent Neural Networks

Recurrent networks reuse a state update across sequences, allowing context from earlier inputs to influence later predictions.

21.7 Unsupervised Learning and Transfer Learning

Unsupervised objectives learn from structure without labels; transfer learning adapts representations learned on one task to another.

21.8 Applications

Deep networks support perception, language, and control, while their data needs, compute costs, and failure modes call for careful evaluation.

Key terms and a backpropagation trace

Feedforward network: A directed computation from inputs through hidden layers to an output. Here there are two input values, two tanh hidden units, and one sigmoid output.
Loss and gradient: Binary cross-entropy measures prediction error; a gradient gives the direction and sensitivity of loss to each weight.
Backpropagation: The chain rule propagates the output error backward through the network. Gradient descent adjusts weights in the direction that locally lowers loss.
Generalization: Performance on new data. This demo trains and reports loss on only four XOR examples, so it demonstrates fitting—not a reliable estimate of generalization.

Single-example trace: with input (1,0), compute h₁=tanh(a₁+b₁₀), h₂=tanh(a₂+b₂₀), then o=sigmoid(v₁h₁+v₂h₂+c). Compare o with target 1; propagate error d=o−1 backward through tanh derivatives (1−h²), then update weights. Training repeats this online update over the four examples for each of 500 epochs. The order and initial weights are fixed; 500 epochs do not guarantee perfect XOR classification.

Train a tiny neural network

Learn the XOR pattern

A 2–2–1 tanh/sigmoid network learns four labeled examples with backpropagation and gradient descent.

Input pair

inputs

hidden tanh

sigmoid

Choose an input; train to update network weights.

Demo performance

Let E be epochs (500), m the training examples (4), and P the network's trainable parameters (9). One forward/backward online update costs O(P) for this fully connected network, so training takes O(EmP) time; rendering the four-example mean loss is O(mP). Here E, m, and P are fixed, so the click is bounded constant work in the specific page. The model parameters occupy O(P) space; extra per-example activations and working values are O(U), where U is hidden-unit count (2 here). Thus auxiliary space excluding model and data is O(U)=O(1) for this fixed network; visualization adds O(1) elements. This describes arithmetic operation growth, not wall-clock timing or a convergence guarantee.