Web data is a sample of a market, not the market
Build research around an explicit sampling frame, coverage gaps and observation context so a large dataset does not create false confidence.
In this article
THE KEY IDEA
A bigger dataset can still be biased. Describe what could be observed, what was collected and what remains outside the sample.
01Define the population and the observable sample
A set of public listings represents what those sources expose under particular conditions. It does not necessarily represent every seller, transaction or customer in the market. Begin by defining the population you want to understand and the narrower set you can actually observe.
For example, advertised property prices are not completed sale prices. A job board shows postings, not every open position or completed hire. Public reviews reflect people who chose to review and whose reviews were displayed. These distinctions should shape the analysis before data collection begins.
02Make the sampling frame explicit
Record sources, regions, categories, languages, time windows and selection rules. Keep search parameters and sort order when they influence which records are visible. If only the first few result pages are collected, state that boundary rather than calling the dataset complete.
Track collection failures separately from exclusions. A region omitted by design is different from a region missing because its source failed. Both affect interpretation, but they require different remedies.
| Record | So a reader can judge |
|---|---|
| Source and category scope | Which part of the market is represented |
| Observation window | Whether records are temporally comparable |
| Selection and pagination rules | How records entered the sample |
| Missing and failed coverage | Where absence may reflect collection limits |
| Entity matching policy | Whether duplicates inflate the counts |
03Avoid false precision in derived metrics
A percentage calculated from a large sample can look authoritative even when the sample is systematically incomplete. Show the denominator and relevant coverage caveats. Do not label a scraped listing count as market share unless you have a defensible method connecting that count to the market definition.
Keep observed and inferred fields separate. A displayed price is an observation; a revenue estimate derived from it is an inference with additional assumptions. Document those assumptions and test how the result changes when they change.
04Compare consistent cohorts over time
A trend can move because the source mix changed rather than because the market changed. Preserve a consistent cohort where possible and report additions, removals and collection gaps. Compare like-for-like regions, categories and record definitions before interpreting a difference as a market signal.
Use a manual audit to identify source-specific quirks. A source may cap visible results, duplicate listings across categories or change its default sort order. These are not merely scraper implementation details; they can alter the research conclusion.
05Deliver a methodology with the dataset
Include a short description of scope, collection dates, cleaning, deduplication and known limitations. Keep the data dictionary close to the output and version both when definitions change. A research dataset is more useful when another analyst can reproduce the interpretation, not just open the file.
References & further reading
Put it into practice.
Try a public-page workflow, or scope a maintained dataset with our team.