Keep a web-backed knowledge base fresh and traceable
Design collection, chunking and refresh around source identity so generated answers can cite the right version of a page.
In this article
THE KEY IDEA
Fresh collection does not automatically make every answer current. Track the path from page version to indexed passage to cited answer.
01Treat retrieval as a maintained data pipeline
A web-backed retrieval system has several stages: discover pages, collect them, clean content, create searchable units, update the index and retrieve evidence for an answer. A failure in any stage can leave old information available even when the latest crawl appears successful.
Define which sources belong in the knowledge base and what questions they are meant to support. A bounded source set is easier to review than unrestricted discovery. Keep collection failures separate from intentional exclusions and genuine page removals.
02Give pages and versions stable identities
Use a document identifier that survives a routine recrawl, then create a version identifier from the content and transformation configuration. Store the source URL, final URL, observation time and any published or updated date. These fields answer different questions and should not be collapsed into one freshness timestamp.
Keep the relationship between a chunk and its parent document version. If the page changes, replace or retire the affected chunks rather than adding a second unexplained set. Otherwise a retrieval result can mix two contradictory versions of the same source.
{
"document_id": "company-about",
"version_id": "content-hash-and-parser-version",
"section": "Services",
"source_url": "https://example.com/about",
"observed_at": "2026-10-11T09:00:00Z"
}03Chunk around meaning, then test retrieval
Preserve headings, table labels and enough surrounding context for a passage to make sense. A price without its currency or a policy sentence without its exception can become misleading when retrieved alone. Remove repetitive navigation, but do not strip useful qualifiers simply to reduce token count.
Build a small evaluation set of real questions and expected evidence. Test whether the right passages are retrieved before judging the generated answer. If retrieval misses the relevant section, rewriting the answer prompt will not reliably solve the underlying problem.
04Refresh only what needs to change
Use permitted change signals and content comparisons to avoid reprocessing unchanged documents. Schedule refreshes according to the information's useful lifetime, not a single global interval. A pricing page and a historical methodology page usually need different treatment.
When a page is unavailable, do not immediately erase its history or silently treat the old version as current. Apply a documented stale-data policy. Deletions, rights changes and access revocation should propagate to derived stores and indexes when required, not stop at the raw-content folder.
05Require answers to acknowledge missing evidence
Citations should point to the source supporting a claim, not merely to a domain mentioned in the response. Preserve which version was used and let reviewers inspect it within your retention policy. Instruct the system to distinguish collected evidence from inference and to say when the available sources cannot answer a question.
Retrieved pages remain untrusted input. Do not allow their text to override tool permissions, reveal secrets or redirect the task. A useful knowledge base combines relevant evidence with controlled execution; it is not just a large context window.
References & further reading
Put it into practice.
Try a public-page workflow, or scope a maintained dataset with our team.