Where Do LLMs Get Their Data? Sources, Datasets and Licensing

Where do large language models get their training data?

Large language models get training data from large collections of text, code and other content used to learn language patterns. The main sources are scraped web pages, open datasets, platform data, synthetic data and directly licensed material.

The source mix matters to people who publish, buy or govern AI systems. A model’s training corpus affects its vocabulary, technical competence, historical knowledge and exposure to material carrying copyright, privacy or licensing obligations.

Web crawls provide broad coverage. Common Crawl is a public snapshot of the internet, and OpenAI models draw on Common Crawl data. Other sources include open-source code repositories, books and academic text. Models also draw from Wikipedia, blogs, forums, Q&A sites such as Stack Overflow, and open government documents, per linkedin.com.

A crawler collects public pages, a dataset builder extracts readable text, filters low-quality material and removes duplicates, then a model trains on the remaining tokens. Direct licences can add material beyond public crawls, while synthetic data expands examples generated from existing models.

Training data is not a live search index or an answer citation. A page may have influenced training without appearing in a current answer, and a current answer may retrieve a source that was never part of the original training corpus.

Which public datasets do LLMs train on, and how large are they?

Public LLM datasets range from raw web archives to curated corpora selected for quality, education or domain diversity. Their sizes vary because some preserve broad web coverage while others apply extensive filtering before training.

Common Crawl is the largest named web source in this group. It contains more than 250 billion web pages and over 1 petabyte of raw data, per github.com, 2026-05-22. FineWeb is a processed subset designed for model-ready use.

Dataset	Scale or representative-mix share	What the content is	Who built or uses it
Common Crawl	More than 250 billion pages and over 1 petabyte of raw data, per github.com, 2026-05-22	Public web snapshot	Primary source for LLM training data collection; OpenAI models draw on it
C4	Size not supplied; about 10–15% of one representative open-source training mix, per github.com, 2026-05-22	Cleaned Common Crawl derivative	Dataset builder not stated in the supplied reference
The Pile	About 825 GB across 22 subsets, per llmbook.icsgen-ai.org, 2026-10-10	Diverse text subsets	Dataset builder and model use not stated in the supplied reference
RedPajama v2	About 30 trillion raw tokens, per llmbook.icsgen-ai.org, 2026-10-10	Raw corpus for LLM data preparation	Dataset builder and model use not stated in the supplied reference
FineWeb	About 15 trillion tokens, per llmbook.icsgen-ai.org, 2026-10-10	Processed Common Crawl data	Dataset builder and model use not stated in the supplied reference
FineWeb-Edu	About 1.3 trillion tokens, per llmbook.icsgen-ai.org, 2026-10-10	Content selected for educational quality	Dataset builder and model use not stated in the supplied reference
Dolma	About 3 trillion tokens, per llmbook.icsgen-ai.org, 2026-10-10	Open training corpus	Supports the OLMo model family, per llmbook.icsgen-ai.org, 2026-10-10

A representative open-source mix assigns about 60–70% to Common Crawl, 10–15% to C4, 4–5% to GitHub code and 3–5% to Wikipedia, per github.com, 2026-05-22. Those shares describe one mix, not a universal recipe for every model.

Which websites and platforms commonly appear in LLM data?

Common platforms in LLM data include Reddit, Wikipedia, Stack Overflow, GitHub, YouTube, Amazon, blogs, forums and open government documents. These sources contribute different forms of language and knowledge, but no single source explains every model answer.

Wikipedia contributes encyclopedic reference material. Stack Overflow contributes technical questions and answers. GitHub provides code and programming documentation. Reddit, Twitter and other forums contribute conversational language, opinions and problem discussions. LLMs also train heavily on Reddit, YouTube and Amazon data, per martech.org.

News sites and scientific databases can add reporting, research language and specialist terminology, but the supplied references do not identify named outlets or databases. Open government documents add public records and institutional material. Each category shapes a corpus differently because editorial practices, formats and update cycles differ.

Training weight and answer citations measure separate things. Reddit and LinkedIn together account for 22% of LLM citations, per reddit.com. ChatGPT draws nearly half of its citations from Wikipedia. Neither figure proves that those sources carry the same share of historical training data.

A citation reflects what an answer system surfaces at generation time. Training reflects material that influenced the model’s learned parameters earlier. Treating citations as a training-data inventory confuses distinct stages of AI answer production.

How is raw web data cleaned, filtered and deduplicated before training?

Raw web data passes through crawling, text extraction, filtering, deduplication and quality selection before LLM training begins. The process turns an internet-scale archive into a corpus of usable language rather than repeated, broken or low-value pages.

1. Web crawlers collect pages and associated metadata.
2. Extraction systems isolate readable text from navigation, boilerplate and markup.
3. Filters remove malformed, repetitive or unsuitable material.
4. Deduplication reduces copies of the same or near-identical text.
5. Quality selection retains material that meets a dataset’s intended standard.

FineWeb illustrates the difference between processing and selection. FineWeb contains about 15 trillion tokens of processed Common Crawl data, while FineWeb-Edu contains about 1.3 trillion tokens selected for educational quality, per llmbook.icsgen-ai.org, 2026-10-10. The smaller corpus reflects a narrower quality decision, not merely a shorter crawl.

Dataset construction requires policy choices about language, domains, repetition and document quality. Filtering is not a neutral afterthought because the criteria determine which language patterns remain in training.

A cleaned corpus is a collection of documents, not an explanation for every model response. A later answer emerges from learned token relationships, fine-tuning and, in retrieval-enabled systems, current source material.

Does publicly available mean an LLM can legally train on it?

No. Public availability does not automatically license content for reproduction, dataset creation or LLM training. A page can be accessible online while copyright, contractual terms, database rights or personal-data obligations still apply.

Public-domain material has no active copyright restriction of the relevant kind, though buyers should verify its status. Openly licensed data carries a stated permission framework, and buyers need to examine the licence version, scope and conditions. Directly licensed data is supplied under negotiated terms that define permitted uses and restrictions.

Publicly accessible material without an explicit licence has a different status from public-domain, openly licensed or directly licensed material. Access alone is not evidence of training permission. User-submitted data also requires separate review because a forum post, profile, review or uploaded file can be public while raising privacy, notice, retention and secondary-use questions.

The unresolved issue is whether and when collection, copying, processing or training on material without explicit permission is lawful under the applicable jurisdiction and facts. A technical description of a dataset is not a legal clearance process.

For procurement, the operational question is narrower: what rights, controls and contractual commitments apply to the data a vendor uses? That question requires written answers, not assumptions based on online access.

What do undisclosed datasets mean for copyright claims and proof?

Undisclosed training datasets make it difficult to prove whether a specific work entered a model’s training corpus. A dataset name such as Common Crawl or FineWeb does not establish the origin, licence or processing history of every document.

Open corpora allow more inspection because their structure and stated scope can be examined. Dolma contains about 3 trillion tokens and supports the OLMo model family, per llmbook.icsgen-ai.org, 2026-10-10. That transparency does not create a document-by-document provenance record.

Proprietary models often disclose only partial, vendor-stated information. Without a complete dataset inventory, an author, publisher or rights holder must determine whether the relevant work was collected, retained, filtered, deduplicated or used during training.

Useful provenance records include source URLs or snapshots, licence versions, crawl manifests, document hashes, filtering logs and vendor attestations. These records can establish what was collected, under which stated terms and how a document moved through a dataset process.

Data provenance is a recordkeeping issue as well as a licensing issue. A model card, broad source category or public dataset label provides context, but does not prove the status of a particular document.

What is the difference between training data and live retrieval?

Training data teaches an LLM statistical patterns before deployment, while live retrieval supplies current documents when a user asks a question. The distinction explains why an answer can sound informed yet be outdated, incomplete or wrong.

Pretraining uses large corpora to learn relationships between tokens. Fine-tuning adjusts a model for narrower behavior or a domain. Human-feedback data shapes response behavior. Retrieval data is fetched at answer time, while user prompts define the task.

LLMs do not store training data like a database. They learn statistical patterns and relationships between tokens, although highly repeated data can be memorised, per emergingai.pro. A model therefore cannot reliably identify the exact historical source behind every generated sentence.

Hallucinations occur when generated token patterns produce a plausible statement unsupported by reliable current evidence. Learning from a page historically is different from retrieving its current version. Retrieval can add fresh context, but depends on the sources the system surfaces for that answer.

Synthetic data can expand a corpus and create additional examples. It can also amplify earlier model errors when generated material is reused without strong quality controls.

How can a brand check whether ChatGPT mentions it?

A brand cannot audit an LLM’s full historical training data. The practical method is to run real buyer prompts and record mentions, citations and AI-referred activity. Teams should measure what answer engines produce now rather than infer visibility from a dataset label.

Sona AI Visibility Growth adds Agent Analytics, company-level visitor identification and AI search attribution. Mention tracking is the baseline that most AI visibility tools provide. Peec AI and Otterly.ai also provide AI visibility tracking for prompt-based monitoring.

Running the same prompt list daily records changes after retrieval-index updates, model updates and new competitor content. Teams can compare repeated results over time rather than treat one answer as a durable visibility finding.

A single brand-level visibility score is a dated snapshot, not a work item. Sona AI Visibility uses AI Search Insights to show tracked prompts, cited sources and page-level performance, so a team can identify the prompts and sources behind the result rather than view only an aggregate number.

Prompt-level monitoring separates training-data speculation from observable outcomes. A brand can document whether it appears, which page receives citations and whether AI-referred visits connect to pipeline. That evidence supports content, technical and measurement decisions without claiming proof of training inclusion.

What should B2B buyers ask an AI vendor about data provenance?

B2B buyers should request written answers about training sources, customer-data use, retention, deletion, subprocessors, audit rights and intellectual-property indemnity. For regulated buyers, verbal assurances are insufficient because contractual terms govern accountability.

• Which source categories trained the model, and which categories were directly licensed?
• Is customer data used for model training, evaluation or product improvement?
• What retention and deletion terms apply to prompts, files and outputs?
• Which subprocessors handle customer data or model requests?
• What audit rights, security documentation and incident-notification terms are available?
• What intellectual-property indemnity applies to model use and generated outputs?

Procurement teams should separate model provenance from operational data handling. A vendor can describe public training sources while offering separate contractual commitments for enterprise prompts, uploaded documents and API traffic.

Sona Marketing Measurement can connect AI search activity to pipeline and revenue, but data provenance questions require direct vendor documentation. Measurement answers what happened after an AI interaction. Contractual terms answer how a vendor handles business data.

A useful review process records each answer, assigns an owner and identifies the contract clause that supports it. That method gives legal, security and procurement teams evidence for approval.

Frequently Asked Questions

Is Common Crawl the main source of LLM training data?

Common Crawl is the largest single web source discussed in many LLM data inventories. A 2026 technical overview estimates Common Crawl at about 60–70% of a representative open-source training mix, per github.com, 2026-05-22. OpenAI models are described as using it, but it is one input among several.

Do LLMs store their training data like a database?

No. LLMs learn statistical patterns between tokens rather than storing records for exact lookup, per emergingai.pro. Highly repeated content can be memorised, but a generated answer does not provide a reliable database-style record of every source that influenced it.

Do LLMs train on Reddit and Wikipedia?

Reddit, Wikipedia, Stack Overflow and GitHub are commonly named LLM data sources. Citation shares in answers are separate from training weight. Reddit and LinkedIn accounting for 22% of LLM citations, per reddit.com, does not establish their share of a model’s training corpus.

Is training data the same as what ChatGPT cites in its answers?

No. Training data shapes patterns learned before deployment. Citations come from sources surfaced when an answer is generated, often through live retrieval. A cited page can be current even when the model’s underlying training data is older.

What is the difference between FineWeb and FineWeb-Edu?

FineWeb contains about 15 trillion tokens of processed Common Crawl data. FineWeb-Edu is a roughly 1.3 trillion-token subset selected for educational quality, per llmbook.icsgen-ai.org, 2026-10-10. The distinction is quality selection, not only corpus size.

Can a company tell whether its content was used to train an LLM?

Often not with certainty. Dataset names do not establish the origin of individual documents, and undisclosed training sets make proof harder. A company can instead run real prompts daily to document current mentions and citations, then use that evidence to assess AI visibility.

Last updated: October 2026