What is a document connector?
A document connector is the authenticated link between an answering layer and a repository holding content. It retrieves documents for indexing and keeps the index aligned with the source as material changes. It also carries the repository's permission model across, so the index knows who may read what.
Also called: Content connector
Why a connector is a permissions problem before it is a data problem
Connecting the first repository is a week of work, and connecting the seventh is where the architecture is actually tested. Each system expresses identity in its own form and refreshes at its own rate, so a connector that assumes one model quietly degrades on the next. The cost of getting this wrong appears as answers citing a document that was withdrawn a month ago. A connector is therefore a continuing synchronisation obligation rather than an integration completed once.
- A connector that indexes content without permissions creates a disclosure route.
- Freshness is a property of the connector rather than of the index.
- Each repository has its own deletion semantics the connector has to honour.
- Connector coverage determines what the answering layer can be asked about at all.
How a document connector works
- The connector authenticates as a service identity with read access to the repository.
- Documents are enumerated with their modification time and their access control list.
- Content is extracted and indexed with that metadata attached to every chunk.
- Subsequent passes detect changed and deleted documents, then reconcile the index.
- Permission changes are re-read on a schedule agreed with the repository owner.
- Failures are surfaced per repository, with the time of the last successful pass.
How a document connector is evaluated
A document connector is assessed on freshness and on whether permissions arrive intact. Examine the following per connected repository:
- lag between a change in the source and its appearance in the index
- proportion of documents carrying complete access control metadata
- share of deletions reflected in the index within the agreed window
- extraction failure rate by document type and by size
- time since the last successful synchronisation pass
Limitations, risks and common failure modes
- Permissions are indexed once and never refreshed as groups change.
- Deleted documents remain answerable because deletion events are never emitted.
- A service identity is granted broad rights, so the index exceeds any user's access.
- Scanned attachments extract poorly and are reported as successfully indexed.
- Rate limits on the source silently extend the synchronisation interval.