AI model pricing is unusually easy to compare and unusually easy to misunderstand.
Providers publish a price for input tokens and a price for output tokens. Procurement teams can put those numbers into a spreadsheet, estimate monthly volume and identify the cheapest model in a few minutes. The result looks precise.
It may also be wrong.
The token invoice measures the cost of running the model. It does not measure the cost of getting work accepted by the business. Between those two points sit retries, human review, fact checking, correction, escalation, latency, failed tool calls and customer-facing risk.
A model that costs less per request can cost more per completed task if its output needs regular repair. A more expensive model can be cheaper when it produces a usable answer in one pass, calls the right tools, follows the required format and reduces review time. The reverse can also be true: a premium model is wasteful when a cheaper model handles a simple, low-risk task reliably.
This is why the useful metric is not cost per million tokens. It is cost per accepted outcome.
OpenAI uses similar language in its guidance on managing AI investments in the agentic era: measure useful work per dollar instead of optimising an isolated unit price. The principle applies no matter which provider or model you use.
What counts as an accepted outcome?
An accepted outcome is not merely an answer that the model produced. It is a result that passed the quality bar for the workflow and can move to the next stage without avoidable rework.
The definition changes by task.
For a product-description draft, acceptance may mean that the copy is accurate, follows the brand style, includes the required attributes and is ready for a quick editorial pass.
For a support reply, acceptance may mean that the message answers the customer’s question, uses only approved policy, avoids an unauthorised promise and is ready for an agent to send.
For code, acceptance may require passing tests, preserving existing behaviour, meeting security rules and surviving review.
For document extraction, acceptance may mean that every required field is present, tied to a source and stored in the correct schema.
For an autonomous workflow, acceptance may require more than the final text. The agent must choose the right tools, use the right accounts, stay inside permissions, stop at approval boundaries, produce an audit trail and recover safely when an external service fails.
Until the organisation defines acceptance, model comparison is mostly theatre. One team celebrates fluent output. Another discovers that a fluent mistake takes longer to inspect than a visibly incomplete answer.
The AI invoice has several missing lines
The full cost of a model-assisted workflow usually includes at least eight components.
1. Model usage
This is the visible part: input tokens, output tokens, cached context, reasoning settings, image or audio processing and any provider-specific charges. It matters, particularly at scale, but it is only the starting point.
2. Retry cost
Some outputs are rejected and generated again. The retry may use a revised prompt, more context or a stronger reasoning setting. Count the entire chain, not only the final successful call.
A workflow that succeeds after three attempts has a different economics profile from one that succeeds on the first attempt, even if the unit price is lower.
3. Human review
Review is often the largest hidden cost. A person has to read, compare, verify and decide. The work becomes expensive when the output is plausible enough to demand close attention but unreliable enough that attention cannot relax.
Review time should be measured in minutes per accepted outcome, separated by role. Five minutes from a junior editor and five minutes from a senior engineer are not the same cost.
4. Correction and reformatting
The model may have the right idea in the wrong shape. Staff then fix tone, remove unsupported claims, restore required fields, repair a spreadsheet, adjust code, correct citations or move content into the business system.
If people repeatedly perform the same correction, the problem may be the prompt, the model, the interface or the workflow design. Treat repeated cleanup as data, not as an unavoidable tax.
5. Tool and infrastructure cost
Agentic systems call search, databases, browsers, code execution, OCR, storage and third-party APIs. These services have their own usage cost and latency. A model that makes unnecessary calls or sends excessive context can inflate the bill outside the model provider.
6. Failure handling
What happens when a tool times out, authentication expires, a source changes format or the model cannot complete the task? Human intervention, queue recovery and duplicate prevention all cost money.
An agent that fails loudly before making a change can be cheaper than one that produces a partial result and leaves the team to discover the missing step later.
7. Delay
Latency is a business cost when the result blocks a customer, employee or downstream process. The fastest model is not automatically best, but time-to-accepted-result belongs in the evaluation.
A slower first response can still win if it eliminates retries. A fast cheap model can win when work can run in parallel and occasional rework does not block anything.
8. Error impact
Some mistakes are cheap. A weak internal summary can be discarded. Other mistakes create refunds, support incidents, compliance problems, broken deployments or damaged trust.
Expected error cost combines probability and impact. You do not need a perfect actuarial model. You do need to stop pretending that every wrong output costs the same.
A simple cost model
Start with one repeated workflow and a fixed evaluation set. For each model or configuration, calculate:
Model and tool cost for all attempts + review labour + correction labour + failure handling + expected error cost = total workflow cost
Then divide by the number of accepted outcomes.
This does not need to become a complex finance project. A spreadsheet is enough for the first pass. The important change is that the denominator is accepted work, not generated messages.
Imagine two models processing the same 1,000 tasks. Model A costs half as much per call but needs more retries and twice as much review. Model B has a higher provider invoice but passes the quality bar more often on the first attempt. Without measuring the surrounding labour, procurement chooses A and the operations team quietly pays the difference.
The example does not predict which model will win. That is exactly the point. The winner depends on the task, prompt, tools, quality bar and cost of errors.
Separate tasks before comparing models
“Which AI model should we use?” is too broad to answer well.
A business does not have one AI workload. It has many:
- short classification,
- extraction into a fixed schema,
- summarisation,
- customer-facing copy,
- research,
- code generation,
- code review,
- complex planning,
- document analysis,
- autonomous tool use.
The model that wins one category may be wasteful or unreliable in another.
Segment work by complexity, risk and volume.
High-volume, low-risk tasks reward low cost, speed and format reliability. Premium reasoning may add little value.
Medium-risk tasks often reward strong instruction following and predictable output. Review can remain human, but the goal is to make that review quick.
High-risk or hard tasks justify a stronger model when the extra capability reduces error, escalation or senior review. They may also require narrower permissions and mandatory approval regardless of model quality.
Do not route every task to the most capable model “to be safe”. That can consume budget without improving the acceptance rate. Do not route everything to the cheapest model “to scale”. That can move the cost from the provider invoice into payroll and incidents.
Build an evaluation from your real rejects
Public benchmarks are useful for understanding broad capability. They do not know your catalogue, policies, codebase, tone, data quality or operational constraints.
The best evaluation set comes from real work, especially work that previously failed.
Collect representative examples:
- routine easy cases,
- long or messy inputs,
- missing information,
- conflicting instructions,
- edge cases,
- content that should trigger refusal or escalation,
- tool failures,
- tasks where the correct answer is “I cannot complete this safely”.
Define the expected outcome and the acceptance criteria before running the models. If reviewers invent the standard after seeing the answers, preferences and brand familiarity will distort the result.
Use blind review where practical. Hide the model name and randomise the order. Record why an output failed: factual error, missing field, unsupported claim, poor tool choice, formatting problem, excessive latency, unsafe action or tone.
The reason matters. A prompt change may fix a formatting problem. Better retrieval may fix missing knowledge. A stronger model may fix planning. A permission boundary may be needed for an unsafe action. “Use a better model” is only one possible treatment.
Keep the evaluation small enough to run regularly. Models, prompts, tool integrations and business rules change. A one-time test becomes stale quickly.
Review time is a product metric
Many AI projects report output quality without measuring the reviewer.
That misses the human experience of the system.
Ask reviewers to record:
- time to accept or reject,
- number of corrections,
- severity of the worst problem,
- confidence after review,
- whether checking the AI was faster than doing the task directly.
That final question can be brutal. An AI draft that saves two minutes of writing but adds five minutes of verification has not increased productivity.
Review design also matters. Give the reviewer the source, the proposed output and the acceptance criteria in one place. Highlight uncertainty instead of hiding it. Preserve links to evidence. Make rejection and correction easy. Do not force staff to compare six tabs while the AI system claims to have saved time.
The model and the review interface form one product. Evaluate them together.
Accepted does not always mean correct
There is another uncomfortable issue: people can accept bad AI output.
Reviewers get tired. Fluent writing creates confidence. High volume encourages rubber-stamping. If the quality check depends entirely on human attention, acceptance rate may rise while actual quality falls.
Use deterministic checks where possible:
- validate required fields,
- test code,
- check totals,
- enforce schemas,
- compare identifiers,
- block forbidden phrases,
- verify that cited sources exist,
- limit actions by permission,
- detect duplicates.
These checks do not replace judgment. They protect judgment from spending time on errors a machine can catch reliably.
Audit a sample after acceptance. Compare downstream outcomes. If customer replies are accepted quickly but later reopened, the acceptance definition is incomplete. If generated product content passes editorial review but return reasons rise because specifications are unclear, the workflow is optimising the wrong thing.
An accepted outcome must remain useful after it leaves the AI interface.
Include quotas, availability and switching cost
The cheapest model on paper is not available if the team reaches its usage limit halfway through the week. A model with excellent quality can become an operational bottleneck when latency spikes or access changes.
Track:
- rate limits and weekly quotas,
- availability by plan and region,
- latency under real load,
- context limits,
- data-retention requirements,
- tool support,
- fallback behaviour,
- the effort required to move prompts and evaluations elsewhere.
Avoid designing a workflow that assumes one provider will always have the same price, model name or allowance. Keep prompts, acceptance tests and business rules outside the provider-specific interface where practical.
Portability does not mean pretending all models behave identically. It means the business can compare and switch without rediscovering its quality standard from scratch.
Use routing, not a single winner
The most cost-effective architecture is often a routing strategy.
Use a small or inexpensive model for predictable work. Escalate ambiguous cases. Reserve the strongest model for tasks where deeper reasoning changes the result. Require a human for actions with real external impact.
A useful route can follow this pattern:
- deterministic validation before the model;
- low-cost model for the standard case;
- confidence or rule check;
- stronger model for failed or complex cases;
- human approval for high-impact action;
- logging of the final outcome and reason for escalation.
Routing should not rely on the model declaring itself confident. Use observable signals: missing fields, failed tests, conflicting source data, task length, customer value, risk category or reviewer rejection.
The routing logic will evolve. Rejected outputs are the feedback loop.
Questions to ask before buying more AI
Before signing a larger contract or moving a workflow to a cheaper model, ask:
- What exact outcome are we buying?
- What makes that outcome acceptable?
- How often does the current process meet that bar?
- How much human time does review take?
- Which errors are common?
- Which errors are expensive?
- What happens when the model or a tool fails?
- Can simple cases use a cheaper route?
- When should the task escalate?
- Can we rerun the evaluation next month?
If the team cannot answer these questions, a lower token price will not create control.
Optimise the workflow, not the demo
AI demos usually show the successful final answer. Businesses live with the attempts that came before it, the reviewer who had to check it and the customer who experiences the mistake.
That is why model selection belongs inside workflow design.
Sometimes the cheapest model really is the cheapest. It handles the task reliably, at high volume, with minimal review. Use it.
Sometimes the expensive model earns its price by reducing retries, senior review and failure risk. Use it where the difference matters.
Sometimes neither model fixes the real problem because the instructions are unclear, the source data is poor or the workflow has no owner. Repair that first.
The objective is not to buy the most intelligent model or the lowest token rate. It is to produce accepted work at a sustainable total cost.
Token price is a line item.
The workday is the invoice.
