The data brief to write before collecting your first page

01Describe the output as a decision

A request to scrape a retailer leaves many important questions unanswered. A request to compare the delivered price of matched products in three regions is much more useful. It identifies the consumer, the comparison and the context the collection must preserve.

Write a short statement describing who will use the dataset, what they will do with it and how often the decision changes. This prevents collecting attractive but irrelevant fields while overlooking the one condition that makes the output usable.

02Specify the record and its context

Decide what one row represents: a product, an offer, a listing, a location or an observation. Give it a stable identifier and define its relationship to other entities. A price observation should not overwrite the product's identity, and a seller should not be duplicated as an unrelated entity on every row.

03Agree on missing and unavailable states

Define the difference between not published, not applicable, collection failed and not yet collected. These are operationally different outcomes. If every one becomes an empty cell, the receiving team cannot tell whether it needs a retry, a schema change or no action at all.

List excluded areas explicitly, including private account content, checkout-only values or sources whose access has not been agreed. A clear exclusion is more useful than a promise of complete data that nobody can measure.

04Make the pilot representative

Choose sample pages that exercise the difficult cases, not just the easiest URLs. Include multiple templates, a legitimate empty result, a missing optional field and any region-sensitive behaviour. Keep some pages out of development so the final check tests generalisation rather than memorised examples.

Define acceptance thresholds with the data consumer. Measure field correctness, usable coverage and delivery timeliness separately. A job that completes on time with incorrect prices is not a successful feed, while a slightly late delivery may still be useful for a non-urgent research task.

05Name the owners and recovery path

Assign responsibility for source changes, failed jobs, schema revisions and downstream integration. Agree how incidents are reported and whether backfills are required. Keep a sample output and a versioned data dictionary with the brief so discussions refer to the same contract.

Finally, define a stop condition for the pilot. It should end with a go, revise or stop decision based on the agreed criteria. A pilot that grows indefinitely without an acceptance decision is already a production project, just without production expectations.