What is a SetFit intent classifier?
A SetFit intent classifier is a compact model trained to recognise intents from only a few labelled examples of each. The method first fine-tunes a sentence embedding model so that examples of the same intent lie close together, then trains a simple classifier on top of those embeddings. The result is small enough to respond almost instantly and accurate enough for the common requests a voice line receives.
Also called: SetFit classifier, Few-shot intent classifier
Why a small classifier belongs in front of a large model
Sending every caller turn to a large language model to work out what the caller wants is slow and expensive, and most turns are routine. A compact classifier can settle the common cases in milliseconds and hand only the ambiguous ones to a larger model. It also behaves identically every time for the same input, which makes routing decisions easier to test and to defend. Training needs only a handful of examples, so new intents can be added as the call design evolves.
- Routine turns are classified far faster than a large model can respond.
- The same input always produces the same intent, which simplifies testing.
- Only a few labelled examples are needed to add a new intent.
- Ambiguous turns are still escalated to a larger model when confidence is low.
How a SetFit intent classifier works
- A small set of example utterances is written or collected for each intent.
- Pairs of examples are used to fine-tune a sentence embedding model.
- A lightweight classification layer is trained on the resulting embeddings.
- At call time, each transcribed turn is embedded and classified.
- Predictions below a confidence threshold are passed to a larger model.
- Misclassified turns from live calls are reviewed and added to the training set.
How a SetFit intent classifier is evaluated
A SetFit classifier is evaluated on held-out utterances from real calls, kept separate from the training examples. Measure the following on that held-out set:
- accuracy per intent on the held-out set
- share of turns resolved without escalation to a larger model
- accuracy on turns the classifier marked as confident
- classification latency per turn
- drift in accuracy as caller language changes over time
Limitations, risks and common failure modes
- Training examples written by staff sound nothing like real callers.
- Two intents with overlapping wording are confused consistently.
- The confidence threshold is set once and never revisited.
- New caller requests are forced into an existing intent that does not fit.