No public API? Inspect the page before choosing a browser
A practical sequence for checking source HTML, structured metadata and rendered content without assuming every page needs AI.
In this article
THE KEY IDEA
Inspect where the required fields actually live. Use the least complex permitted method that captures them reliably.
01Start with the data you need, not a favourite tool
Write down a short list of required fields and inspect a representative page. Are the values visible before interaction? Do they depend on a selected variant, destination or date? Does the page show an explicit empty state? These questions determine what you need to collect and which context must accompany it.
Check whether the publisher offers a documented API, feed or licensed export. Those can provide a clearer contract than a changing page. If access is restricted, resolve permission first. The absence of a public API is not permission to use an internal one or bypass an access control.
02Inspect the original document
The response HTML may already contain the relevant text or structured metadata. JSON-LD can describe products, articles or jobs, but coverage varies. Compare any embedded values with the visible page and selected context. Metadata may describe a parent product while the displayed offer belongs to a specific variant.
An HTML parser is useful when the data is already present. Cheerio parses markup but does not run a page's JavaScript. If a field exists only after a later application request, changing CSS selectors will not make it appear in the original document.
| Observation | Next check |
|---|---|
| Required values exist in source HTML | Parse a saved fixture and validate fields |
| JSON-LD contains the fields | Check entity type, arrays and visible consistency |
| Only a loading shell is returned | Inspect rendered content and a readiness signal |
| Login, challenge or restricted data | Review authorised access; do not treat it as content |
03Add rendering only for a demonstrated need
A browser is appropriate when permitted interaction or JavaScript execution is needed to produce the data. Choose a readiness condition tied to the output. Record the final URL, relevant selections and any failure. Loading the page is only one step; you still need to prove that the expected content was returned.
Browser developer tools can help explain how a page obtains information. They do not turn every observed network request into a supported integration. Undocumented endpoints can change without notice and may depend on permissions or session state. Keep that maintenance and access risk visible when evaluating an approach.
04Separate extraction from interpretation
Use deterministic extraction for stable fields where practical. Add AI when the task involves semantic interpretation, variable layouts or a question across passages. Keep the original evidence and validate the output. A model cannot reliably recover information that was never collected, and it should not manufacture a missing field.
For a broad list of unrelated company websites, semantic analysis may reduce the need for one parser per layout. For a repeated, tightly defined catalogue feed, a source-specific extraction contract may be easier to test and maintain. Volume alone does not decide the architecture.
05Save a regression fixture
Before scaling, keep permitted examples of normal pages, missing fields and empty results. Test the extraction against those fixtures after every change. Add live checks at an agreed cadence, because a saved fixture proves parser behaviour, not that the source still has the same structure today.
References & further reading
Put it into practice.
Try a public-page workflow, or scope a maintained dataset with our team.