Measure scraping cost per usable record

01Use the right denominator

A request can return a page with no useful data. A page can contain many records, and one business record can require multiple pages. Pricing per request, page and record therefore describes different units. Define which unit matters to the receiving team before comparing costs.

A useful internal metric is total collection-and-processing cost divided by accepted records. Keep transport success and data acceptance separate. Otherwise a cheap stream of empty pages can look more efficient than a smaller, accurate dataset.

Workflow A: $12 total / 8,000 accepted records = $0.0015 each
Workflow B: $9 total / 4,000 accepted records = $0.00225 each

02Account for the work around the request

Include discovery pages, detail pages, permitted rendering, repeated attempts, parsing, validation and delivery. For AI analysis, include the content sent to the model and any repeated processing. A large page with irrelevant navigation can cost more to analyse without improving the answer.

Separate one-time setup from recurring operation. A source-specific parser may require initial work but be efficient for a stable repeated feed. A flexible analysis workflow may avoid that setup for many unrelated layouts. Compare them over a realistic operating period, not only the first run.

03Refresh according to the decision

A daily strategy report may not benefit from collecting the same unchanged page every minute. A short-lived quote may require a much tighter observation window. Ask how quickly a change would affect the consumer's decision, then choose a refresh policy that matches that need.

Reuse permitted captures or validated intermediate data when the task allows it. Do not repeat a paid analysis simply because the UI was refreshed. At the same time, do not reuse a stale result without disclosing its observation time. Cost control and freshness need an explicit trade-off.

• Measure accepted records and rejected records separately.
• Track cost by source and processing stage.
• Identify duplicate submissions before increasing capacity.
• Set a budget ceiling and preserve partial results when it is reached.

04Read the billing definition carefully

Check what the provider considers billable: accepted tasks, successful requests, rendered pages, data volume or another unit. Review how errors, retries and optional processing are charged. A provider's diagnostic price field is not necessarily its billing contract.

Use the current product documentation and account terms for actual rates. Do not use the hypothetical numbers in an educational article as a quote. For a managed project, ask which maintenance, backfills and delivery changes are included and which require a revised scope.

05Optimise the largest avoidable cost first

Profile a representative run before changing architecture. The bottleneck may be repeated discovery, unnecessary rendering, poor matching that discards records or a downstream cleanup task. Fixing the cheapest-looking line item may have little effect on the real cost. Optimise toward a dataset the business can use, with enough observability to explain the bill.