Inference Optimisation for LLM Workloads: Throughput, Batching, and Cost

If you serve a model at production volume, inference cost and throughput are among your largest operational concerns, because the cost per request multiplied by the volume is the operational cost of the workload, and the throughput determines whether you can serve the volume at all. Optimising the inference, operating it in the way that maximises throughput and minimises cost, is what makes the workload economical, and operating it naively leaves throughput and cost far from what optimisation would achieve. Understanding the levers of inference optimisation, batching, throughput management, and the cost trade-offs, is what lets you serve a model at scale affordably.

Why naive inference is expensive

Serving a model naively, processing each request independently as it arrives, uses the serving hardware inefficiently, because the hardware that operates the model is capable of processing multiple requests together, and processing them one at a time leaves that capacity unused. The throughput of naive serving is far below what the hardware can achieve, which means you need more hardware to serve a given volume, which raises the cost, and the cost per request is high because each request uses the hardware inefficiently. Naive inference is expensive because it does not use the serving hardware’s capacity for parallel processing, leaving throughput low and cost per request high.

The cost matters because it scales with volume, and at production volume the inefficiency of naive inference becomes a large cost, the cost that scales with the workload that determines the economics of serving the model. This is worse for agentic workloads, because an agent makes multiple model calls per task, multiplying the inference cost per task, so an agentic workload’s inference cost is the per-call cost multiplied by the calls per task multiplied by the volume, and naive inference inflates the per-call cost across all of it. The expense of naive inference is invisible at low volume, where the inefficiency is small in absolute terms, and very visible at production volume, where the inefficiency multiplied by the volume is a large cost, which is why inference optimisation matters for workloads at scale. Serving a model at volume without optimising the inference leaves throughput low and cost high, which is the expense optimisation exists to address.

The levers of inference optimisation

Inference optimisation raises throughput and lowers cost through several levers, the primary of which is batching. Batching processes multiple requests together, using the serving hardware’s capacity for parallel processing, which raises throughput substantially because the hardware processes many requests in the time it would have processed one, and lowers cost per request because each request uses the hardware more efficiently. Batching is the highest-leverage inference optimisation because it addresses the fundamental inefficiency of naive serving, using the parallel capacity that processing requests one at a time leaves idle, and it is what makes serving a model at volume economical by raising the throughput per unit of hardware.

Batching carries a trade-off with latency, which is the tension inference optimisation has to manage. Batching requests together means waiting to accumulate a batch, which adds latency to individual requests, so there is a trade-off between throughput, maximised by larger batches, and latency, minimised by smaller batches or no batching. For a latency-sensitive workload like conversational voice, the latency cost of batching matters and constrains how much you can batch, and for a throughput-oriented workload where latency is less critical, larger batches maximise throughput and minimise cost. Managing this trade-off, batching enough to achieve economical throughput while keeping latency within what the workload requires, is the core of inference optimisation. Other levers complement batching, but batching is the primary one, and the throughput-latency trade-off it carries is the central trade-off of serving a model at scale, which has to be managed for the workload’s specific latency requirements.

The trade-off, and matching optimisation to the workload

Inference optimisation trades engineering effort and latency for throughput and cost, and naming these trade-offs keeps the optimisation matched to the workload. The engineering to optimise inference, implementing batching and managing throughput, is real work, more than serving naively, and the latency cost of batching is a real trade-off against the throughput and cost gains, which means the optimisation has to be matched to what the workload needs. A high-volume workload benefits greatly from inference optimisation, because the throughput and cost gains multiplied by the volume are large, and a low-volume workload benefits less, because the gains are small in absolute terms, so the optimisation effort should match the volume.

The latency trade-off has to be matched to the workload’s latency requirements, batching aggressively for throughput where latency is not critical and batching less where latency matters, so the optimisation serves the workload’s actual balance of throughput and latency needs. This connects to the model choice that also affects inference cost, because using a smaller model where it suffices reduces the inference cost alongside the optimisation of how the inference is operated, and the two together, the right model operate the optimised way, determine the workload’s inference economics. Matching the inference optimisation to the workload, optimising where the volume justifies it and balancing throughput against latency for the workload’s requirements, is the judgment that makes the optimisation worthwhile. The optimisation is not maximising throughput unconditionally but achieving the throughput and cost the workload needs within its latency requirements, which is what makes serving the model economical for that specific workload.

Where it almost broke

We served a model for an agentic workload at production volume, and the inference cost was high enough to threaten the economics of the workload, because the serving was naive, processing requests independently and leaving the serving hardware’s parallel capacity unused, which kept throughput low and cost per request high. The agentic workload made multiple model calls per task, multiplying the high per-call cost across every call and every task, so the naive inference’s inefficiency was multiplied through the whole workload, producing a cost that made the workload barely economical. The model’s capability was fine, and the way the inference was operated was making it too expensive to serve at the volume required.

The correction optimised the inference. We implemented batching, processing multiple requests together to use the serving hardware’s parallel capacity, which raised throughput substantially and lowered cost per request, and we managed the throughput-latency trade-off for the workload’s requirements, batching enough to achieve economical throughput while keeping latency within what the workload needed. We also used a smaller model where it sufficed, reducing the per-call cost alongside the optimisation of how the inference was operated. The workload’s inference cost dropped to where the economics worked, because the inference was now using the hardware efficiently rather than leaving its capacity idle. The lesson was that naive inference is expensive at volume, and optimising it, primarily through batching managed against the latency requirements, is what makes serving a model at scale economical.

Operate the inference the way the workload can afford

The reframe that makes serving a model economical is optimising how the inference operates rather than accepting the cost of naive serving. Inference optimisation for LLM workloads raises throughput and lowers cost primarily through batching, managed against the latency the workload requires, which is what makes serving a model at production volume affordable. Naive inference leaves the serving hardware’s capacity idle, keeping throughput low and cost high, and the inefficiency multiplied by production volume is the expense that optimisation addresses.

The teams that serve models economically at scale are the ones that optimised the inference, batched to use the hardware’s parallel capacity, managed the throughput-latency trade-off for their workload, and used the right model operate the optimised way. The model’s capability is set by the model, and its cost at scale is set by how the inference is operated, and naive serving multiplied by production volume is an expense that optimisation can substantially reduce. Operating the inference the way the workload can afford, batching for throughput within the latency the workload needs, is what turns a model too expensive to serve naively into one economical to serve at scale.