Metadata extraction

Compass

Deriving the attributes of a document, such as its owner and its type, which is what makes filtering possible.

What is metadata extraction?

Metadata extraction is the derivation of a document's attributes from the document itself, rather than from whatever a user typed at upload. The attributes retrieval depends on most are the effective date and the entitlement class. Extraction makes those attributes available to a filter rather than to a display field alone.

Why filtering fails before retrieval does

A retrieval system that cannot distinguish a superseded circular from its replacement will cite whichever one embeds closer to the question. The usual response is better ranking, when the actual defect is that no usable date is attached to either document. Metadata extraction is unglamorous, and it determines whether a request for current policy only can be honoured at all. Most enterprise corpora arrive with metadata that is either absent or inherited from a template nobody updated.

  • A filter can only operate on attributes that were actually extracted.
  • Entitlement class has to be derived before permission trimming can work at all.
  • Effective dates let the system prefer a current version over a near-identical draft.
  • Extraction from content survives migrations that destroy manually entered fields.

How metadata extraction works

  • The attributes worth holding are specified per document type before ingestion.
  • Structural signals such as headers and document templates are parsed first.
  • A model extracts the remaining attributes from the body text with a confidence value.
  • Extracted values are validated against controlled vocabularies where one exists.
  • Low confidence extractions are escalated to the source owner rather than stored silently.
  • Attributes are written to the index as filterable fields and retained with the version.

How metadata extraction is evaluated

Extraction is judged per attribute, since an average across all of them hides the one that matters. Measure the following on documents already ingested:

  • precision and recall per attribute against a manually labelled sample
  • coverage, meaning the share of documents carrying each required attribute
  • proportion of extracted dates matching the document's stated effective date
  • rate of low confidence extractions escalated to a human
  • filter accuracy measured end to end on real queries

Limitations, risks and common failure modes

  • Scanned documents yield a date from the letterhead rather than from the revision block.
  • Inherited template metadata is extracted faithfully and is wrong for every document.
  • Entitlement class is guessed from folder location, which drifts after any migration.
  • Accuracy is measured once on a clean sample that excludes the awkward formats.

Related terms

Document understandingInterpreting a document's structure and meaning, which goes well beyond recognising its characters.Key-value extractionPulling named fields and their values out of a document, which covers most routine intake.Table extractionRecovering rows and columns from a document, which is markedly harder than extracting isolated fields.TaxonomyThe agreed structure of categories a body of material is organised by, which retrieval leans on more than most teams expect.ChunkingHow a long document is divided before indexing, which quietly determines whether retrieval ever finds the right passage.Entity resolutionRecognising that several records refer to the same person or organisation despite differing in form.

All 315 terms