Skip to main content
The journal
AI workflows3 min read

Deep website research needs evidence, not more pages

Plan a multi-page research task around the question, keep source relationships and stop when additional pages no longer improve the answer.

In this article
  1. The starting URL is a lead, not the whole answer
  2. Choose pages by expected usefulness
  3. Build an evidence record before a summary
  4. Do not let the source control the research agent
  5. Measure answer coverage, not crawl volume

THE KEY IDEA

A page budget is a ceiling, not a target. More collected text can increase cost and confusion without adding evidence.

01The starting URL is a lead, not the whole answer

A company's careers page can be a valid starting point for research even when the question concerns its customers. The right next step is not to summarise that page harder. Identify which pages might contain the missing evidence: case studies, customer stories, service pages or dated announcements.

Translate the question into evidence requirements. To identify customers, you need an explicit relationship claim and its context. A logo alone may represent a partner, integration or former customer. Keep those alternatives visible rather than turning every mentioned brand into a current client.

02Choose pages by expected usefulness

Discover links from navigation and page content, then rank them against the question. Deduplicate equivalent URLs, avoid repetitive tags and filter obvious distractions. Use public search where the research scope permits it, but open and inspect the underlying source rather than treating a search snippet as the final evidence.

Keep first-party statements separate from independent coverage. A company's own page can explain what it claims to sell. A third-party directory may add context, but can be old or copied from elsewhere. Multiple pages repeating one press release are not independent confirmation.

  • Record why a page was selected.
  • Prefer a specific dated relationship statement over an undated logo wall.
  • Track unavailable pages and unresolved claims.
  • Stop when the agreed evidence requirement or budget boundary is reached.

03Build an evidence record before a summary

Store the claim, source URL, relevant passage, observation time and any published date separately. The model can then synthesise those records instead of reading an unstructured pile of page text. Keep contradictory claims side by side until the conflict is resolved or disclosed.

Illustrative evidence record
{
  "claim": "Example Co published a customer case study",
  "source_url": "https://example.com/case-study",
  "relationship": "customer_at_publication",
  "current_status": "not_confirmed",
  "observed_at": "2026-10-11T09:00:00Z"
}

04Do not let the source control the research agent

A retrieved page is data, not an instruction source. Content that tells an agent to reveal credentials, change its task or visit an unrelated private endpoint must not become a tool command. Limit the destinations and actions available to the research workflow, and keep secrets outside the model's page context.

Distinguish evidence from inference in the final answer. State what was found, what remains uncertain and which pages support each material conclusion. Do not use a confident writing style to hide a missing source or a failed collection.

05Measure answer coverage, not crawl volume

Review whether the research answered each part of the question. A ten-page result can be stronger than a thirty-page result if its evidence is more relevant. Record pages attempted, usable sources and unresolved questions so the reader can understand the limits without scrolling through the entire crawl log. Make detailed evidence available on demand.

References & further reading

Put it into practice.

Try a public-page workflow, or scope a maintained dataset with our team.