If a language model has never encountered your brand, it cannot recommend you. It guesses, and a guess usually means it names a competitor. That is the mechanic behind every conversation about AI visibility, and it starts long before anyone types a prompt. It starts with model training data.

Understanding how that data gets gathered, cleaned and compressed tells you which levers you actually control, and why most of what is being sold as “LLM optimization” is just good marketing wearing a new label.

What training data actually is

Training data is the corpus a model learns from in order to predict the next token in a sequence. It can be labeled, where each input is paired with a correct answer, or unlabeled, where the model infers structure on its own.

The scope is close to unlimited: web pages, books, code, forum threads, product reviews, image libraries, video, audio. Speech models need recordings across accents and emotional registers. Multimodal models need all of it at once. The only real constraint is what can be obtained, and that constraint is tightening fast.

One correction worth making early. A model is not a database of facts. It stores billions of numerical weights encoding statistical relationships between tokens, with no record it can look up and no memory of any individual document. That distinction explains most of what people find confusing about AI answers.

Compression, not memorization

During training, a model processes enormous volumes of examples and adjusts its weights through backpropagation. Predict the next token correctly and the weights hold. Predict wrongly and the error propagates back through the network, nudging the weights toward a better answer. Repeat across trillions of tokens.

What emerges is a vector space: text converted into numerical representations, with word and sentence embeddings that preserve semantic meaning and context. Relationships between concepts get encoded structurally. This is parametric memory, knowledge baked into the architecture itself.

Parametric memory is fast, because nothing has to be fetched. It is also frozen and lossy. The model knows the world as it existed when training stopped, and it will confidently misremember details it compressed too aggressively. That is why retrieval augmented generation and live web search exist: non-parametric memory, slower but current, and essential wherever an answer needs grounding. We unpacked that mechanism in our piece on grounding and RAG, and the commercial fallout of a frozen knowledge base in when the training data cutoff becomes a ranking factor.

The pipeline nobody talks about

Raw data is not training data. Between the two sits a process that is expensive, slow and stubbornly manual at the edges: collect a corpus wide enough to cover the problem space, clean it into consistent formats, label a portion of it so supervised learning has a reference for correctness, pre-process it so no category is wildly over-represented, then partition off a slice the model never sees so you can validate that it generalized rather than memorized.

Labeling is the bottleneck. Annotating a single hour of video can take up to 800 hours of human work. Expert annotation in fields like medicine or law is harder still, because the people qualified to do it have better things to do.

The workaround is bootstrapping. Annotators label a small seed set, a compact model trains on it, and that model proposes labels for everything else while humans validate rather than create. It works, and it concentrates risk: a bias in the seed set gets amplified across the whole corpus.

The spectrum of supervision

• Supervised: every input carries the correct answer.
• Unsupervised: no labels; the model finds its own patterns. Good at surfacing structure humans miss, unreliable when you need a specific outcome.
• Semi-supervised: a small labeled core teaches the rules, the rest is inferred.
• RLHF: reinforcement learning from human feedback, where people choose between outputs (preference data) or demonstrate a task for the model to imitate (demonstration data).
• Pre-training and fine-tuning: broad datasets build general capability, narrow ones create domain specialists.
• Multimodal and edge case data: mixed media, plus deliberately adversarial examples designed to break the model so it becomes more robust.

Licensing sits awkwardly over all of it. Research has found that 23% of supervised training datasets are published under research or non-commercial licenses, which is a live legal problem for anyone shipping a commercial product on top of them.

Where the data comes from

Common Crawl is the workhorse: an open repository of web crawl data, free to anyone. The full web graph holds roughly 607 million domain records across all datasets, with monthly releases covering between 94 and 163 million domains. The Mozilla Foundation’s 2024 report Training Data for the Price of a Sandwich found that 64% of the 47 LLMs it analyzed used at least one filtered version of it. You can check your own presence against the Common Crawl index server by URL pattern.

Wikipedia and Wikidata punch above their weight. The default English Wikipedia dataset runs to about 19.88 GB of complete articles, a small share of total tokens and an outsized share of influence, because it is where entity resolution and factual consensus get settled. Wikidata layers a vast structured knowledge graph on top, and several major labs have signed agreements covering this content.

Licensed content fills the gaps. Multi-million dollar publisher deals were the story of 2023 and 2024, though the pace has slowed considerably. On the multimodal side, Shutterstock supplies images and video, Getty Images has an arrangement with Perplexity, and Disney is a 2026 partner for the Sora video platform. Under that three-year agreement, Sora can generate short user-prompted social videos using Disney characters, Disney makes a $1 billion equity investment in OpenAI, and it receives warrants to purchase additional equity.

Books, code and the open web round it out. BookCorpus turned roughly 11,000 unpublished books into a 985 million-word dataset. GitHub and Stack Overflow underpin the coding capability that has become one of the most commercially valuable features of modern LLMs. The open web supplies scale, freshness and genuine opinion data, at the cost of being noisy and inconsistent.

The commons is closing

Three pressures are squeezing supply at once.

Access is being revoked. Eight in ten of the world’s largest news websites now block AI training crawlers. Robots.txt is only a directive, but CDN-level blocking and updated terms of service have teeth.

Consumption outpaces production. Humans do not write books, code or articles fast enough to feed the next training run, and as quality publishing moves behind paywalls the freely available pool skews further toward low-value material.

Synthetic data compounds it. Generated content is fine for targeted fine-tuning and poor as a primary diet. Models trained on model output drift, and that drift is the mechanism behind model collapse.

Bias runs through every phase. Origin bias sits in the dataset itself, in whether it fairly represents the population it claims to. Development bias enters through the features and weighting chosen during training. Deployment bias appears when flawed evaluation creates feedback loops that reinforce the original skew. In domains like healthcare, that cycle can encode historical inequality into a system people treat as neutral.

How to get into model training data

There are two schools. The first tries to reverse-engineer specific seed datasets and engineer entry into them. The second ignores the plumbing and focuses on being genuinely well covered and consistently described across the open web.

For almost every brand, the second is correct. Chasing individual corpora is fragile, borderline manipulative, and it collides with the fact that governs everything here: training data has a cutoff, so you cannot retroactively insert yourself into a model that has already shipped. You build presence now for the models trained next year.

The practical checklist:

• Manage the bot ecosystem deliberately. Training crawlers, indexing crawlers and live browsing agents are different things with different consequences. Decide which you allow rather than defaulting to whatever your CDN does.
• Serve content server-side. Google renders JavaScript well. Most AI crawlers see the raw HTML response and nothing else. If your substance loads client-side, it does not exist to them.
• Do entity optimization properly. Consistent naming and contact details, sameAs schema properties, presence in Google’s Knowledge Graph and in Wikidata, internal linking that connects your topics into a coherent structure.
• Write machine-readable content. Semantic HTML, real headings, tables for comparisons, lists for sequences. Extractable passages get reused; walls of prose do not.
• Be unambiguous about who you are. The more consistently your brand is described across independent sources, the less room a model has to conflate you with something else. That argument is the whole subject of part one of this series on disambiguation.
• Get mentioned by other people. Podcasts, industry events, guest contributions, conference talks, trade coverage. Direct associations are what you say about yourself; semantic associations are what everyone else says. Models weigh the second more heavily.

Two structural realities are worth holding in mind. Google has a partnership with Reddit and access to an enormous volume of YouTube transcripts, giving it better-structured proprietary data than almost anyone. And most labs buy from third-party vendors, with Scale AI building supervised datasets and Bright Data handling web collection, which means the pipeline is far more industrialized than the “scraped the internet” shorthand suggests.

The endgame is not appearing in a corpus for its own sake. It is being recognizable enough that when a model reaches for retrieval, you are the obvious candidate. Our analysis of how ChatGPT already knows who it will recommend before it searches shows how much gets decided by parametric memory before any live lookup happens.

Frequently asked questions

Can I get my site added to a model that has already launched?

No. A trained model’s parametric memory is fixed at its cutoff. You can influence what it retrieves live through RAG and web search, and you can build presence for the next training run, but you cannot edit a model that already exists.

Does being in Common Crawl guarantee I am in the training data?

No. Labs apply heavy filtering, and low-quality or duplicated pages get dropped. Presence in the crawl is a necessary condition, not a sufficient one.

Is blocking AI crawlers a mistake?

It depends on your business model. Publishers monetizing attention have a real case for blocking training bots while allowing retrieval agents that cite them. Brands that want to be recommended usually do not, because they are trading away the visibility they are trying to buy.

Will synthetic data solve the shortage?

Only in the right place. It is useful for fine-tuning, augmentation and covering rare cases. As a substitute for real human-generated material at pre-training scale, it degrades quality over successive model generations.

The takeaway

Model training data is a supply chain, not a magic box, and the supply is getting scarcer and more expensive. You cannot buy your way in and you cannot hack your way in after the fact. What you can do is make your brand unambiguous, machine-readable, server-rendered and widely discussed, so that whichever corpus gets assembled next contains you, described correctly, in more than one place.