Why AI Inference Keeps Getting Cheaper — and What It Changes for Builders — Analysis

First, a definition that trips up a lot of coverage. "Training" is the one-time, compute-heavy process of building a model from data; "inference" is the repeated act of running that model to answer a prompt — every chatbot reply, every summary, every translated sentence. Builders pay inference cost on every call, so it is the line item that decides what is economically viable to ship. The story of the last two years is that this line item has quietly collapsed.

How steep the drop has been

The numbers are striking. Stanford's 2026 AI Index Report found that the cost of querying a model with GPT-3.5-level performance fell from roughly $20 per million tokens in late 2022 to about $0.07 per million tokens by late 2024 — a roughly 280-fold reduction in about 18 months. On the public market, the trajectory is just as clear: OpenAI's GPT-4 launched in March 2023 at $30 per million input tokens and $60 per million output tokens, while its GPT-4o mini, released in July 2024, was priced at $0.15 in / $0.60 out per million tokens — roughly a 99% drop versus the 2022-era baseline, according to OpenAI's own framing at launch. Independent trackers such as Artificial Analysis report that the cost of reaching a given capability level fell by well over 100× across 2025 alone. Put plainly: a customer-service bot that would have cost hundreds of dollars a month in compute two years ago now often runs for a few dollars.

Why it is falling — three forces at once

The decline isn't one cause; it is three compounding ones. The first is hardware. Each GPU generation delivers large efficiency gains — NVIDIA's Blackwell B200, presented at GTC 2024, was positioned as delivering up to roughly 25–30× the inference performance-per-watt of the prior H100 generation. More throughput per watt means more tokens served per dollar of data-center cost. The second is software: techniques like 4-bit quantization, speculative decoding, paged and continuous attention, and mixture-of-experts (MoE) routing squeeze far more useful output out of the same silicon. A model that only activates a fraction of its parameters per request simply costs less to run. The third is competition: open-weight models such as DeepSeek, Llama and Qwen have forced aggressive price discovery, pulling the entire market's per-token pricing downward as closed providers match or undercut to keep usage.

Training costs are falling too — but that's a different bill

It is easy to confuse the two. The Stanford AI Index also notes that the training cost of frontier-class models has compressed sharply. DeepSeek's V3 was reported to have been trained for about $5.6 million, a fraction of the $100M-plus attributed to GPT-4-class training runs — a figure widely cited in coverage of the 2025–2026 open-weight wave. But training is a sunk cost spread across every user of a model, whereas inference is paid per request. For the developer shipping a feature, the inference decline is the one that rewrites the business case, because it changes the marginal cost of serving each additional user to near-zero.

What it changes for builders

Cheap inference unlocks product shapes that were previously impossible. Agentic workflows — multi-step sequences where a model plans, calls tools, observes results and retries — were uneconomic when each step cost real money; at today's prices they are routine. A chatbot that triages support tickets, a document pipeline that summarizes thousands of contracts, a local assistant running on a single laptop: all become viable when the per-call cost is a fraction of a cent. The practical shift is from "can we afford to use AI here?" to "which model should we route this to?" — a question of model routing and token minimization rather than raw access. Smaller teams and indie developers, who were priced out of the 2023 era, are now first-class participants.

Where it is heading — and the floor

The downward trend is structurally supported: hardware roadmaps keep improving, software optimization is far from exhausted, and competition is intensifying. Stanford's AI Index and NVIDIA's Blackwell disclosures both point to continued efficiency gains through the late 2020s. But a true "race to zero" is unlikely. Analysts expect a tiered pricing landscape to emerge — much like cloud computing evolved from flat on-demand rates into reserved, spot and savings-plan tiers — where base capability becomes nearly free while premium latency, privacy guarantees, fine-tuning and reserved capacity carry a price. The end state is cheap-by-default inference with paid tiers on top, not zero-cost everywhere. For builders, the strategic implication is clear: design for a world where intelligence is abundant and nearly free, and compete on what you do with it rather than on whether you can afford to call a model at all.