As one X user bluntly summarized: “Claude is winning because rich people are providing the training data. Poor Meta has to train AI with the peasants of the internet.”
Crude, and not quite right about the mechanism. But it points at something real.
A new survey by Epoch AI and Ipsos finds that 80 percent of Claude’s weekly users in the United States live in households earning 100,000 dollars or more. For Meta AI, the figure is 37 percent. ChatGPT, Gemini, Grok, and Copilot cluster in between.
The standard AI bias conversation is rather predictable: biased datasets, misclassification, discriminatory decisions. Real problems, worth the attention. But the Epoch numbers point at something the dataset framing misses.
Frontier models are not primarily trained on their users. Pretraining corpora, curated fine-tuning data, and paid annotators do most of the work. What users shape is what happens after. Which tasks get measured, which failures get diagnosed, which complaints trigger an engineering sprint, which workflows justify a new evaluation suite. The deployed system gets better where its core users demand it gets better.
When 80 percent of a tool’s users are high-income professionals, the gravity pulls in one direction. Contracts, code, research synthesis, legal reasoning, structured argument. These are the tasks where failure is visible, reported, and fixed. When the user base sits inside a messaging app, the gravity pulls toward conversational fluency and low-friction replies. The underlying models may start similar, but the products diverge over time.
The cohort does not directly train the model, but user signals, filtered through product decisions, strongly shape which failures are treated as worth fixing, and which capabilities are pushed to production.
No protected class needs to be misclassified for this to matter. The risk sits upstream, in whose problems are treated as worth solving first and whose failure modes get fixed fastest. The pipeline is mediated, providers curate, safety training cuts across cohorts, annotators impose standards. Those layers may dampen the effect, but they do not entirely remove it.
This reframes what alignment means. The word is usually offered as a technical property, a model aligned with human values, with humanity as the implied reference point. In deployment, alignment reflects a weighted mix of provider priorities, annotator judgments, and the user populations that generate the strongest signals. The weights are not neutral, and every alignment claim has a hidden subject.
The fairness toolkit was built for a different problem. This one lives in the allocation of attention and iteration, and it will be shaped by whoever the product managed to acquire first.
(Hat tip to Luukas Ilves for the pointer.)