Entrar
AI inference is getting cheaper, but your agents are getting more expensive
The good news is that large language model (LLM) token costs are coming down. The conundrum: The overall cost of AI workloads is going up.
Gartner research predicts that, while token costs will fall by 95% by 2030, inference costs for agentic workflows will increase more than fivefold over the next two years. This is because AI app builders are using more, and often more expensive, tokens as LLMs get ever more complex. It is what Gartner calls the “inference paradox.”
In other words, “the rate of innovation is outpacing the cost curve,” Gartner analysts Will Sommer and Sabine Zimmerhansl noted in their report. “The market is captured by a token-deflation illusion.” Buyers dangerously assume that as AI providers improve token economics, these savings will be reflected in their roadmaps. But, simply put, “they will not.”
‘Swarms’ of agents eating up tokens
There is no doubt that AI delivers massive value, and is often better, and faster, than people at many routine tasks, the analysts pointed out. Agents can also more quickly identify patterns across siloed systems. For instance, Gartner has seen customer success agents reduce response times by 99%.
But as AI evolves, token usage increases, and token value is variable, Sommer and Zimmerhansl noted. Advanced AI agents that can reason already cost up to 150x more on a single task than basic AI chatbots.
A simple chatbot must read and interpret a request and quickly deliver a “probabilistically reasonable” answer, but agents, as they become more sophisticated, must think, question, and adapt when something goes wrong, and they increasingly run continuously, and often invisibly, in the background.
“They need to be able to validate their results for accuracy without necessarily having a human in the loop,” the analysts wrote. “They need to talk to other agents.”
All of this demands more resources, and token consumption increases exponentially as “swarms” of increasingly autonomous agents trigger and call each other. This incurs a “massive inference tax” before a user even gets their result, they noted.
The hardware costs to train medium-sized agentic models with advanced reasoning capabilities is 2.5x greater than training simple, similarly-sized chatbots, they reported. Further, agent inference costs are 5x greater, and agents require 5x to 30x more tokens than a chatbot to handle equivalent tasks.
“Now consider how costs will balloon when running hundreds of agents that can perform dozens or hundreds of tasks each hour,” the analysts said. Costs continue to skyrocket as agents break many problems into several small tasks, call higher-order models, and require multimodal data.
“The volume of compute required for these capabilities is mind-bending,” the analysts noted.
To analyze the impacts of agentic systems, Gartner built a Tokenomics Model based on various scenarios of training and inference. These included various designs (number of layers, LLM-as-a-judge or as mixture-of-experts), technology improvements, hardware specifications, and various other cost considerations (data, infrastructure, energy, labor).
The firm ran 12 types of AI model with various capabilities, and found that basic workflows cost around $0.05 per inference token; summarization and knowledge retrieval cost roughly $0.10; more complex workflows cost around $0.30; and planning and learning cost roughly $0.40 per token. This means provider cost per token for planning and learning tasks is 8x to 10x that of basic workflows.
“Costs will inevitably escalate, and as they do, there is no guarantee that value will grow commensurately,” Sommer and Zimmerhansl contended. “ROI from each new generation of technology will be hard-earned.”
Being economical in the age of AI
Enterprises can be diligent and keep token costs in check by developing and maintaining complex multimodal systems, Gartner said. They will also need ways to measure ROI and improvement across completed workflows.
Orchestration will be the differentiator, and “inference tiering” will improve cost and performance, so enterprises should develop systems that route queries to the most cost‐efficient model and block agents from invoking frontier models by default for simpler tasks, Gartner advised. Adopting usage-based pricing is another important step; move from flat compute fees to tiered plans that scale based on need.
Enterprises can consider mandating continuous refresh cycles, Sommer and Zimmerhansl added. “Treat each model release like a ‘new car’ losing value on day one,” they wrote, and build in data fine-tuning and self-learning feedback loops.
Builders should also set minimum standards for AI execution: Define success thresholds and risk mitigation and compliance overhead up front. “Refuse to greenlight deployments until scenarios are stress-tested against token-price swings and compliance expenses,” the analysts emphasized.
Further, Gartner also advises embedding value-per-outcome into product planning. This could mean requiring every AI feature to forecast and track its spend against a “clear outcome metric,” such as tasks automated or cases successfully closed. This can help identify the low-performing workflows that require improvement or deprecation.
Still, ROI is “eminently possible,” but it requires significant effort across complex workflows, the analysts noted; enterprises can’t simply rely on traditional systems and workflows. “Defaulting to generic autonomous intelligence will result in unbounded costs orders of magnitude higher than those of optimized product ecosystems,” they pointed out.
This article originally appeared on Computerworld.
