20.1 Statistical Learning
A statistical model describes how data could have been generated. Learning estimates its parameters from examples, often by maximizing likelihood or a posterior probability.
AI · STATISTICAL LEARNING
20 · Learning Probabilistic Models
Statistical learning fits probability models to data. Complete observations permit direct likelihood optimization; hidden variables require reasoning about their uncertainty.
20.2 Learning with Complete Data
When all relevant variables are observed, likelihood decomposes into contributions from examples. Maximum-likelihood estimates choose parameters that make observed data probable.
20.3 Learning with Hidden Variables: The EM Algorithm
Expectation-maximization alternates an E-step, estimating latent-variable responsibilities using current parameters, with an M-step, updating parameters to maximize expected complete-data likelihood.
Key terms and an EM trace
- Likelihood / maximum likelihood
- Likelihood scores parameters by how probable they make the observed data; maximum likelihood chooses parameters with the largest score.
- Latent variable / responsibility
- A latent assignment is unobserved. A responsibility rᵢₖ is the posterior probability that observation i came from component k under current parameters.
- E-step and M-step
- The E-step computes responsibilities. The M-step uses their weighted sufficient statistics to update mixture weights, means, and variances.
- Initialization and local optima
- EM is sensitive to initialization and generally converges to a local optimum, not necessarily the globally best mixture. This demo uses one fixed initialization and does not check convergence.
One-iteration trace: each of the 14 observed x values gets two soft membership weights (which sum to 1); the weighted sums yield a mean and variance for each Gaussian. The variance is floored at 0.08 to avoid collapse. “Run 10 steps” means exactly ten iterations, not “run until converged.”
Latent-variable model
Separate two hidden clusters with EM
Fit a two-Gaussian mixture to fixed 1D observations; component labels are hidden.
Ready. Each iteration re-estimates memberships and component parameters.
Demo performance
Let n be the number of observations and K the number of Gaussian components. Each iteration computes K densities/responsibilities per observation and accumulates weighted moments: O(nK) time. Temporary accumulators and one observation's responsibilities need O(K) auxiliary space; the input data take O(n), and the rendered observations/means take O(n+K) UI space. Here n=14 and K=2, so the algorithmic work per iteration is O(n) only because K is fixed at two. Ten iterations multiply time by 10. A general dense covariance Gaussian mixture has different per-component costs; this claim is for the univariate scalar-variance implementation shown.