What is a document store?
A document store holds the original files that a retrieval system indexes, together with every version of each one. The index holds chunks and vectors derived from those files, while the store holds the source of truth they were derived from. When an answer cites a passage, the document store is where a reviewer goes to open the original and confirm it says what the answer claims.
Also called: Document repository, Source store
Why an index without its originals cannot be audited
A citation is only useful if the cited document can be produced again in the form the system read. Source systems change constantly, and a policy that was replaced last month may be the one an answer relied upon last week. Without a versioned store, the reviewer opens today's document and finds wording that does not match the answer. The document store preserves the version that was indexed, so the trail holds even after the source has moved on.
- Answers must cite the version that was indexed, which may differ from the current one.
- Re-chunking and re-embedding both start from the stored original.
- Retention rules can be applied to originals separately from derived data.
- Deleting a document from the store gives a clean signal to remove its chunks.
How a document store works
- Each ingested file is stored as received, with a content hash and a timestamp.
- A changed file creates a new version without overwriting the earlier one.
- Every chunk in the index records the document identifier and version it came from.
- Access permissions from the source system are stored with each version.
- Citations resolve to a specific version and an exact location inside it.
- Retention and deletion schedules are applied according to document classification.
How a document store is evaluated
A document store is evaluated on whether every answer can be traced to an exact original. Test the following on a sample of recent answers:
- share of citations that resolve to an openable original
- share of citations that resolve to the version actually indexed
- chunks in the index whose source version no longer exists
- time to retrieve an original for a reviewer
- agreement between stored permissions and those in the source system
Limitations, risks and common failure modes
- Only the latest version is kept, so older answers lose their evidence.
- Deleted documents remain in the index after leaving the store.
- Permissions are copied once at ingestion and never refreshed.
- Storage of every version grows without a retention policy to govern it.