LLM Cost Optimisation: The Levers Operators Underuse

If you operate LLM systems, the cost is shaped by more levers than you are probably using, and defaulting to one or two while leaving the rest unused pays a cost the unused levers would have reduced. The levers that shape LLM cost, the model choice, the inference, the context, the caching, the architecture, each affect the cost, and using the full set rather than one or two is what achieves the cost the levers can reach. Understanding the levers operators underuse is what lets you optimise the cost across the full set rather than the one or two you default to.

Why operators underuse the levers

Operators tend to reach for the obvious cost lever, often the model choice, and leave the less obvious levers unused, because the obvious lever is the one that comes to mind and the others require knowing they exist and how to use them. The model choice is the obvious lever, using a cheaper model to reduce cost, and operators reach for it while leaving the inference optimisation, the context management, the caching, and the architecture, the less obvious levers, unused, paying the cost those levers would have reduced. The underuse happens because the obvious lever is reached for and the others are not known or not considered, so the operator optimises the one lever and leaves the rest, achieving a fraction of the cost reduction the full set would have.

This underuse costs, because each unused lever is cost reduction not captured, and the full cost reduction requires using the full set of levers rather than one or two. An operator who optimises only the model choice captures the model-choice savings and leaves the inference, context, caching, and architecture savings uncaptured, paying a cost that using those levers would have reduced, and the sum of the unused levers is a significant cost reduction left on the table. This is the full set of optimisations that shape the cost, and using one or two while leaving the rest is underusing the levers, which pays more than the full set would. Operators underuse the levers because they reach for the obvious one and leave the less obvious ones unused, which pays the cost those levers would have reduced, and using the full set is what captures the cost reduction the levers can achieve.

The levers operators underuse

The levers that shape LLM cost span several dimensions, and operators underuse most of them. The model choice is the lever operators use most, and even it is often underused, defaulting to a frontier model everywhere rather than using the right model for each task, a smaller cheaper model where it suffices, which reduces the cost across the tasks that do not need the frontier model. The inference optimisation is a lever operators underuse, optimising how the inference operates, through batching and other techniques, to reduce the cost per call, which operators leave unused when they accept the inference cost as given.

The context is a lever, managing the context so each call processes the relevant content rather than an unmanaged full context, which reduces the cost per call by reducing the tokens processed, and the caching is a lever, reusing the repeated content so it is not reprocessed on every call, which reduces the cost of the repeated content. The architecture is a lever, designing the system to accomplish its tasks with fewer or cheaper calls, which reduces the cost by reducing the calls, connecting to the determinism-versus-autonomy balance where deterministic steps avoid the model calls autonomy would make. These are the levers that shape the cost, the model choice, inference, context, caching, and architecture, and operators underuse most of them, reaching for the model choice while leaving the inference, context, caching, and architecture unused. Using the full set of levers, each reducing the cost along its dimension, is what captures the cost reduction that using one or two leaves uncaptured.

The trade-off, and using the levers together

Using the cost levers involves trade-offs and effort, and naming these keeps the optimisation balanced. Each lever has a trade-off, the model choice trades capability, the context management risks omitting information, the architecture changes may constrain the system, and using the levers is engineering effort, so optimising the cost with the full set of levers trades the effort and the levers’ trade-offs for the cost reduction. The optimisation balances the cost reduction against the trade-offs, using each lever where its cost reduction is worth its trade-off, rather than pulling every lever to its extreme regardless of the cost to capability or correctness.

Using the levers together is what captures the full cost reduction, because the levers are complementary, each reducing the cost along a different dimension, so using them together captures more than using any one. The model choice reduces the per-call cost by using cheaper models, the inference optimisation reduces the per-call cost further, the context management and caching reduce the tokens per call, and the architecture reduces the calls, and together they reduce the cost more than any one alone, which is why using the full set captures the cost reduction that using one or two leaves uncaptured. The ROI measurement informs the balance, because the goal is the cost per resolved outcome, and the levers should reduce the cost without degrading the outcomes, which the ROI measurement reveals. The deeper point is that the cost is shaped by the full set of levers, and using them together, balanced against their trade-offs, is what achieves the cost the levers can reach, where using one or two leaves the rest of the reduction uncaptured. Using the levers together, balanced against their trade-offs, is what captures the cost reduction the full set can achieve.

Where it almost broke

We operated an LLM system and optimised its cost by reaching for the model choice, using a cheaper model, and left the other levers unused, accepting the inference cost as given, not managing the context, not using caching, and not optimising the architecture, so we captured the model-choice savings and left the inference, context, caching, and architecture savings uncaptured. The system’s cost was higher than it needed to be, because we had used one lever and left the rest, paying the cost the unused levers would have reduced, and the sum of the unused levers was a significant cost reduction we had left on the table by optimising only the obvious lever.

The correction used the full set of levers. We kept the model-choice optimisation and added the others, optimising the inference to reduce the per-call cost, managing the context so each call processed the relevant content, using caching to reuse the repeated content, and optimising the architecture to accomplish the tasks with fewer calls. We balanced each lever against its trade-off, using it where the cost reduction was worth the trade-off. The system’s cost dropped by more than the model choice alone had achieved, because we were now using the full set of levers rather than one, capturing the cost reduction the unused levers had been leaving on the table. The lesson was that the cost is shaped by a full set of levers operators underuse, and using them together, balanced against their trade-offs, is what captures the cost reduction that using one or two leaves uncaptured.

Use all the levers

The reframe that captures the full cost reduction is using the full set of cost levers rather than reaching for the obvious one and leaving the rest unused. LLM cost optimisation is about the full set of levers, the model choice, inference, context, caching, and architecture, that operators underuse, because using one or two leaves the reduction the other levers would have captured on the table. The cost is shaped by the full set, and using one lever achieves a fraction of the reduction the full set can.

The operators who operate LLM systems at the cost the levers can achieve are the ones that used the full set, optimised the model choice, the inference, the context, the caching, and the architecture, and balanced each against its trade-off. Reaching for the obvious lever and leaving the rest unused pays the cost the unused levers would have reduced, and the sum of the underused levers is a significant cost reduction left uncaptured. Using all the levers together, balanced against their trade-offs, is what captures the cost reduction the full set can achieve, which is the difference between the cost operators default to and the cost the levers can reach.