What 'AI Inference Costs' Actually Means for a Business

Discussion of AI economics tends to focus on training: the expense of building a model. For a company deploying one rather than building it, training cost is largely irrelevant. What matters is inference — the cost of running the model each time it is used.

The distinction determines whether a product is viable, because the two costs behave in opposite ways.

Training is fixed, inference is variable

Training is a large one-off expenditure producing an artefact. It is a fixed cost, and like any fixed cost it is amortised over usage — the more it is used, the less each use bears.

Inference is the reverse. Every request consumes computation. Costs scale with usage, and unlike most software they do not approach zero at volume. Traditional software has near-zero marginal cost per user, which is why software margins are what they are. A product whose core feature calls a large model on every interaction does not have that property.

What drives the cost of a request

That last point is underappreciated. Providers achieve efficiency by batching requests together. A product requiring immediate responses forgoes some of that efficiency, and pays for it.

The unit economics question

The discipline is the same as any variable-cost business: what does one unit of use cost, and what is it worth?

For a feature inside a subscription product, the question is how many times a typical user invokes it per month, multiplied by cost per invocation, against what that user pays. A feature costing more per month than a meaningful share of the subscription is a problem regardless of how well it works.

This maths is why products that appeared viable during a free trial period become difficult when usage patterns settle — heavy users can cost more than they pay, and heavy users are usually the ones who like the product most.

The levers

Companies have more room than the framing suggests. Smaller models handle many tasks adequately at a fraction of the cost. Caching avoids recomputing identical requests. Routing sends straightforward queries to a cheap model and reserves the expensive one for hard cases. Trimming what gets sent as context reduces input cost directly.

Most production systems combine several of these, and the engineering effort involved is a real cost that should be counted alongside the compute it saves.

Falling prices, rising usage

Per-token prices have fallen substantially and may continue to. It is tempting to conclude that inference cost will cease to matter.

That may be optimistic. Falling unit costs have historically been met with rising consumption — cheaper inference makes it economic to run models on tasks that were not worth it before, and to run larger models on tasks that already were. Total spend can rise while unit cost falls.

Planning on the assumption that prices will fall enough to rescue unviable unit economics is a bet on a specific rate of decline, not a strategy.

Where inference runs changes the cost structure

Inference does not have to happen in a data centre. Small models can run on a phone or a laptop, which shifts the cost from the provider's bill to the user's hardware.

The trade-offs are specific. On-device inference has no marginal cost, works without connectivity, and keeps data local — a meaningful advantage where privacy or regulation is a concern. It is constrained by memory and battery, which caps model size, and it means the model cannot be updated as freely because it is distributed rather than served.

A number of production systems now split the difference: a small local model handles common cases and a larger hosted model handles the rest. This is the same routing logic as cost-based routing, applied across a hardware boundary.

Building the cost into the product

Once unit economics are understood, they become a product design constraint rather than an engineering afterthought.

None of these are exotic. They are the ordinary tools of any business with a meaningful cost of goods sold, applied to a category of software that historically had almost none — which is why they often arrive late, after the pricing page was already written.

The question to ask

For any proposed AI feature: what does it cost per use, how often will a typical user use it, and what is the revenue against that usage? If those numbers do not work at expected volume, the feature needs a cheaper approach or a different pricing model.

It is an ordinary question about variable costs. It is worth asking early, because the answer is much harder to change after the product has shipped and users have formed expectations about what is included.