Pretraining Sets a Ceiling Post-Training Can't Break

Every few months someone relitigates the same question: does post-training actually teach models anything, or does it just polish what pretraining already put there? A new controlled study answers it about as cleanly as anyone has — by raising a language model on nothing but elementary-school material and then trying, with every standard trick, to teach it more.

The LittleLearner project, from a team including Ryan Cotterell (ETH Zürich) and Wieland Brendel (Max Planck Institute for Intelligent Systems), built an 88-billion-token corpus called LittleCurriculum by filtering FineWeb-Edu down to U.S. K–5 curriculum content — five filtering stages spanning age-of-acquisition word statistics, LLM-as-judge annotation against Common Core standards, FastText and ModernBERT classifiers, and regex sweeps for algebraic notation. Then they trained models from scratch at 0.6B, 1.3B, and 5B parameters, each paired with an unfiltered control on the identical architecture and recipe. That control is what makes this more than a stunt: every gap you see is attributable to the data, not the model.

The headline finding, per the paper: nothing downstream recovers what the pretraining filter removed. Scaling from 0.6B to 5B improves in-scope performance but barely moves beyond-grade-5 problems. GRPO post-training on math boosts K–5 arithmetic — and does essentially nothing for grade 8 material, where LittleLearner solves fewer than half as many MathCAMPS problems as its unfiltered twin even at pass@1024. Few-shot prompting with worked explanations? The authors report the model's answers don't change based on the explanation given. Most damning: within their post-training budgets, RL on beyond-K–5 data performed no better than RL on K–5 data. You can't fine-tune in what pretraining never built.

The inverse of the phi bet

This is worth situating, because the field has spent three years running the opposite experiment. Microsoft's TinyStories showed in 2023 that tiny models trained on toddler-vocabulary text produce fluent, coherent English — fluency is cheap. Then phi-1's "Textbooks Are All You Need" and its descendants turned aggressive quality filtering into a strategy: filter up toward textbook-grade data and small models punch far above their weight. FineWeb-Edu itself — the very corpus LittleLearner filters from — exists because classifier-scored educational filtering reliably boosts benchmarks per FLOP.

LittleLearner is the control condition that whole research program never ran. Filtering up buys you efficiency; here's the invoice for what filtering takes away: a capability ceiling that scaling, RL, and in-context learning all bounce off. It also lands squarely on one side of the ongoing debate about whether RLVR-style post-training creates reasoning capacity or merely elicits it. In the cleanest data-controlled setting to date, the answer is elicit. Pretraining writes the ceiling; everything after decorates the room.

Two texture details deserve attention. First, the ceiling isn't a cliff. Perplexity and accuracy degrade gradually as material moves past grade 5 — "growing unfamiliarity," not a wall — which suggests concepts partially reassemble through composition even when they're never stated directly. Second, the model has no idea the ceiling exists. Ask it about quantum entanglement and it confidently explains that entanglement is "when a person gets caught in two or more ropes that are connected in a special way" — an answer that lit up the Hacker News thread. A knowledge boundary, even a surgically clean one, produces confabulation rather than "I don't know." Calibration doesn't come free with curation.

What this means if you curate training data

The practical audience here isn't frontier labs — it's every team doing continued pretraining or from-scratch training on a domain corpus: code models, medical, legal, finance. The uncomfortable implication is that your data-mixing decision is the one you cannot fix later. If your filtered corpus lacks the conceptual substrate for a capability — cross-file reasoning for a code model, mechanism-of-action knowledge for a biomedical one — no SFT set, no RL budget, no clever few-shot scaffold will conjure it. Within tested budgets, post-training on exactly the missing material didn't help. Plan breadth into pretraining and spend post-training on elicitation, formatting, and preference — not on capability injection.

It also puts a number-shaped warning on precision-obsessed filtering. LittleCurriculum's pipeline achieved 0% retention of beyond-grade-5 content but kept only ~35% of legitimate K–5 material. That's the trade every dedup-and-classifier pipeline makes quietly: the stricter your filter, the more in-scope diversity you shred. If you're tuning a quality classifier threshold this quarter, this paper is the argument for measuring recall, not just contamination.

And it's an evaluation warning. The confident-nonsense failure mode means you can't probe a domain model's boundary by asking it where the boundary is. You need out-of-scope evals designed against the training distribution — which is exactly what LittleLearner's released models and corpus make possible for the first time: a sandbox where "the model never saw this" is a verifiable claim rather than a guess about what leaked into Common Crawl.

Caveats, then a verdict

The honest limits: 5B parameters is small enough that in-context learning is weak generally, so the ICL null result is the shakiest of the three. Post-training budgets were finite — someone will argue a 100× larger RL run breaks through, and this paper can't rule that out. The filter-purity claims rest on the authors' own benchmark, and HN commenters found answers (a garbled but recognizable Rayleigh-scattering explanation) that make you wonder how much physics hides in children's science books. And it's days old and not yet peer-reviewed.

But the design is strong precisely where most data-quality research is weak: matched controls, interpretable scope, released artifacts. My read is that this holds up, because it agrees with everything adjacent — RL-elicitation studies, the BabyLM results, the phi trajectory read in reverse. Treat "we'll fix the data gap in post-training" as a red flag in any training plan you review. The ceiling goes in first, and it doesn't move.