Microsoft's Phi models were trained in large part on synthetic 'textbook-quality' data generated by other language models, a deliberate bet that small models fed curated synthetic examples could match the reasoning of much larger ones trained on raw web text. It worked well enough that the approach spread — Nvidia's Nemotron datasets and a growing list of labs now generate training examples with an existing model rather than sourcing them from humans. This has become routine enough to barely register as a decision anymore.
It is, underneath the routine, an editorial act. Someone at Microsoft or Nvidia is choosing what a representative example of a domain looks like, using a system whose own patterns and blind spots get folded into the definition of representative before a single human reader ever checks it.
The appeal is straightforward. Human-generated text in specialized domains — rare languages, niche technical fields, certain kinds of structured reasoning — is genuinely scarce, and synthetic generation produces volume where volume was the bottleneck. It also sidesteps legal friction around scraping human-authored content without consent, since the model generates fresh text rather than reproducing someone else's.
The risk is compounding rather than acute. A model trained partly on a prior model's output inherits that model's stylistic habits, characteristic errors, and gaps — not as isolated mistakes but as patterns reinforced through another round of training. Do this across several generations, each partly trained on the last one's output, and the population of 'normal' text a model has seen can drift from how humans actually write and reason, without any single step looking obviously wrong.
Researchers describe this with the shorthand of model collapse, though the reality on the ground is messier and slower: not systems suddenly failing, but a gradual narrowing of stylistic and conceptual diversity, a smoothing-over of edge cases that synthetic generation tends to underrepresent because it draws, by construction, on patterns the prior model already found probable.
Microsoft and Nvidia are aware of this and mitigate it — mixing synthetic data with curated human sources, filtering for diversity rather than volume, using multiple different generator models so no single one's blind spots dominate. These are real engineering responses. None fully resolves the underlying issue: synthetic data generation is a decision about which version of a domain gets amplified into the training set that shapes the next generation of models.
The stakes are highest in domains with thin human data to begin with — underrepresented languages, minority perspectives within a broader culture — precisely because synthetic data there makes up a larger share of what a model ultimately learns. If Phi's or Nemotron's generator underrepresented those areas in the first place, the gap does not get filled by more synthetic volume. It gets encoded more deeply, dressed up as scale.