A 200 response is not a quality check
Validate page meaning, required fields, context and coverage before letting collected data reach a downstream decision.
In this article
THE KEY IDEA
Transport success tells you that a response arrived. Acceptance checks tell you whether the response contains usable evidence.
01Check the page before parsing its fields
A successful HTTP response can contain a login page, a challenge, an empty application shell or an error message rendered as ordinary HTML. Detect these states before interpreting the document as a product, job or company record. Otherwise a parser can return technically valid but meaningless output.
Keep collection status separate from data status. A task can complete at the provider while your application rejects the result for missing required information. That distinction makes incident reports clearer and prevents downstream consumers from treating rejected data as a genuine empty result.
02Define acceptance at several levels
Use a small set of checks tied to the business purpose. Validate types and required keys, then check context and relationships. A price can be a valid positive number and still belong to the wrong currency, variant or seller.
| Level | Illustrative check | Failure response |
|---|---|---|
| Page | Expected content type and no login wall | Classify collection failure |
| Record | Required identifier and valid field types | Reject or quarantine the record |
| Context | Currency, location and variant agree | Hold for investigation |
| Batch | Coverage and null rates within reviewed bounds | Review the run |
| History | Unexpected changes explained | Compare source evidence |
03Keep evidence for rejected records
A rejection should include the rule, input identity and a safe reference to supporting evidence. Preserve the original value alongside the normalised one where appropriate. Avoid logging credentials or unnecessary personal information while collecting debugging material.
Do not silently replace a rejected value with a convenient default. Zero is a real number, not a generic missing price. Today's date is not a substitute for an absent publication date. Keep unknown, unavailable and invalid states distinguishable in the output.
04Use baselines without assuming yesterday was correct
Track record counts, required-field completeness and value distributions by source and template. A sudden change can reveal a broken selector or a collection gap. But a genuine promotion or catalogue change can also move the metrics, so an anomaly should trigger review rather than automatic correction.
Maintain a small set of representative pages and saved fixtures. Fixtures test parser behaviour; live checks detect source changes. Periodically review the expected values, because a live page is not a permanent ground truth. Include legitimate empty states and missing optional fields in the test set.
05Measure usable output
Report accepted, rejected and unresolved records with their denominators. Separate discovery coverage from extraction accuracy. A parser that reads every available page correctly can still miss most of the intended dataset if discovery is incomplete.
Agree how partial deliveries are handled. A few missing optional descriptions may be acceptable, while a currency mismatch may require holding the feed. The acceptance policy should belong to the data contract, so both collection and consumption teams know when the output can be trusted.
References & further reading
Put it into practice.
Try a public-page workflow, or scope a maintained dataset with our team.