Prepare web data for AI training without losing its provenance

01Choose the learning task first

A classifier, a retrieval system and a generative model need different datasets. For a classifier, the quality and consistency of labels may matter more than the number of pages. For retrieval, preserving current source passages and citations can matter more than turning the content into training examples.

Write down what behaviour the dataset is meant to improve and how that improvement will be measured. Collecting everything available makes later filtering expensive and can introduce irrelevant text, personal information and rights questions that the task never required.

02Make access and use review part of collection

Review the source terms, licences, relevant rights and intended use before ingestion. Material being publicly visible does not establish that it may be copied into a training dataset. Obtain qualified advice for the jurisdictions and data involved rather than treating a generic scraping guide as legal clearance.

Keep a source register that records the basis on which each source is included, any restrictions and who approved its use. Avoid collecting personal information that the task does not need. Retention and deletion requirements should be designed into the dataset, not added after copies have spread across multiple systems.

03Preserve lineage through every transformation

Assign document IDs before cleaning. Keep the source URL, observation time, content hash, language and transformation version with each derived example. If a paragraph is translated or summarised, label that transformation. A reviewer should be able to distinguish original source text from model-generated text.

Keep raw and processed stores separate with access controls and retention limits appropriate to each. Retaining a raw copy can aid debugging, but is not a reason to keep every source indefinitely. The important property is traceability within the permitted retention period.

document_id, source_url, observed_at, content_hash,
language, rights_review_id, transform_version,
quality_status, split, deletion_reference

04Deduplicate before splitting the evaluation set

Exact duplicates can be identified by content hashes after a defined normalisation step. Near-duplicates need additional review, especially for syndicated articles, templated pages and translated copies. Repeated boilerplate can dominate a corpus even when every URL is different.

Group related documents before assigning training and evaluation splits. If copies of the same article appear on both sides, the evaluation can overstate generalisation. Keep a held-out set tied to the actual task and inspect whether source families, companies or time periods leak across the boundary.

05Release a dataset version, not an unexplained folder

Publish an internal manifest with source coverage, exclusions, quality checks, label definitions and known gaps. Record counts at each filtering stage so a future reviewer can explain why records disappeared. When the corpus changes, version the release and rerun the relevant evaluation. More rows are useful only when they improve the intended task without undermining traceability or quality.