Humans have extraordinarily large brains, which tripled in size in the last few million years. Other animals also experienced a significant, though smaller, increase in brain size. These increases are puzzling, because brain tissue is energetically expensive—a smaller brain is easier to maintain in terms of calories. Here we present a theory, captured in an analytic and computational model, that explains these increases in brain size: The Cultural Brain Hypothesis. The theory relies on the idea that brains expand to store and manage more information. Brains expand in response to the availability of information and calories. Information availability is affected by learning strategies (e.g. learning from others or learning by yourself), group size, mating structure, and the length of the juvenile period, which co-evolve with brain size. The model captures this co-evolution under different conditions and describes the specific and narrow conditions that can lead to a take-off in brain size—a possible pathway that led to the extraordinary expansion in our own species. We call these conditions the Cumulative Cultural Brain Hypothesis. These theories are supported by our tests using existing empirical data.