What is change detection?
Change detection compares the current state of every connected source with the state recorded at the previous ingestion. It identifies documents that are new or edited, and separately those that have been removed. Only those documents are passed back through conversion and embedding, which keeps the index current without rebuilding it from the beginning each time.
Also called: Incremental ingestion, Delta detection
Why a retrieval system goes stale without anyone noticing
An index is accurate on the day it is assembled and drifts from then on. Policies are revised and pages are withdrawn, while the system continues to answer from what it collected earlier. Full rebuilds are slow and expensive, so they happen too rarely. Change detection makes small, frequent updates practical. Its handling of deletions matters most, because a withdrawn policy that remains in the index is still being cited as current.
- Superseded documents stay retrievable until something removes them.
- Incremental updates make frequent refreshes affordable.
- Change records show when the index last reflected each source.
- Deletions are the change most often missed and the most consequential.
How change detection works
- A content hash and modification time are stored for every ingested document.
- Each source is listed on a schedule and compared with the stored record.
- New and changed documents are queued for full re-ingestion.
- Removed documents trigger deletion of every chunk and graph entry derived from them.
- Permission changes are detected even when the content itself is unchanged.
- Every detected change is logged with the time it was applied to the index.
How change detection is evaluated
Change detection is evaluated by making known changes to a test source and checking how the index responds. Measure the following after each test change:
- time from a source change to its reflection in answers
- share of edits detected on the next cycle
- share of deletions that removed every derived entry
- permission changes detected without a content change
- documents re-ingested unnecessarily because of cosmetic differences
Limitations, risks and common failure modes
- Modification dates that sources fail to update hide genuine edits.
- Deleted documents leave orphaned chunks that remain retrievable.
- Cosmetic page changes trigger costly re-ingestion of unchanged content.
- A source that becomes unreachable is mistaken for one whose documents were removed.