Why Small Models Will Dominate, Just Not Yet
Small models are more capable and cheaper than ever, yet most companies still pay for hosted frontier APIs. The gap is not capability, it is awareness, operational complexity, evaluation, and trust.
When I speak with founders building AI products, one topic comes up almost every time: inference costs.
As products gain users, API bills grow with them. Everyone is looking for ways to reduce costs without sacrificing quality.
On paper, the solution seems obvious. Replace at least part of the workload with smaller, cheaper models running locally or on infrastructure you control.
Yet very few companies actually do.
Instead, they continue paying providers such as OpenAI, Anthropic or ElevenLabs, even though increasingly capable and significantly cheaper alternatives are available.
Why?
Because companies are not only paying for access to a model.
They are paying to avoid the complexity surrounding it.
Ironically, this is happening at a time when the ecosystem of small models has never been more exciting.
Google is expanding the Gemma family of lightweight models, Alibaba's Qwen family now spans a wide range of compact architectures and use cases, and Liquid AI is designing models specifically for efficient inference on phones, laptops and other edge devices.
Google explicitly positions several Gemma variants for mobile, browser and edge deployment, while Liquid targets CPUs, GPUs and NPUs across both local and cloud environments.
PrismML is pushing this even further with its Bonsai models, using extreme compression techniques such as 1-bit and ternary representations to dramatically reduce model size. Some variants are small enough to run on consumer devices such as smartphones.
The progress is remarkable. Models that would have looked limited only a few years ago can now summarise documents, extract structured data, call tools, process images, generate code and power specialised assistants, all while running on hardware that developers already own.
Yet this abundance creates a different problem.
Every new release comes with different benchmarks, parameter counts, context lengths, quantisations, licences, runtimes and hardware requirements.
Developers are no longer asking whether a small model can run locally.
They are asking which model, in which format, through which runtime and on which device will perform reliably enough for their particular application.
That question remains surprisingly difficult to answer.
The first barrier is awareness
Most people still do not know what small models can do.
Those of us working closely with AI are exposed to a continuous stream of model releases, benchmark comparisons and optimisation techniques. This creates a distorted impression of how widely that knowledge is distributed.
Outside that bubble, many companies are still learning how to use hosted products such as ChatGPT or Claude effectively. Expecting them to understand model architectures, quantisation formats or edge runtimes is unrealistic.
Smaller research labs often focus primarily on improving model quality and efficiency. They may release weights, technical reports and benchmark results without building the distribution, onboarding and developer experience required to reach mainstream users.
Mistral offers an interesting counterexample. Rather than focusing only on releasing new models, it has invested heavily in distribution through enterprise products, managed and self-hosted deployments, integrations and applied engineering support. Its strategy shows that model quality alone is not enough: adoption depends on making models easy to discover, integrate and operate in production. This emphasis on deployment and enterprise integration may also help explain why it has been slower to release new models compared to some of its competitors.
This matters because the best model does not automatically become the most widely adopted model. Distribution, documentation, integrations and trust frequently matter more than a marginal benchmark advantage.
The second barrier is operational complexity
Using a hosted AI service hides most of the system from the developer.
You send a request to an API and receive a response. Scaling, hardware allocation, runtime optimisation, model updates and availability are largely handled by the provider.
Running a model locally reverses that relationship. Much of the complexity moves back to the developer.
You must choose a model, select a quantisation, identify a compatible runtime, manage memory, evaluate output quality and determine whether the target device can sustain the required latency.
Even a setting as apparently simple as context length becomes an infrastructure decision.
When using a hosted model, context limits are usually presented as a product constraint in the documentation. When deploying locally, context length directly affects memory consumption through the KV cache. A configuration that looks harmless can make a model slow, unstable or impossible to run on the target device.
The developer therefore needs to understand not only whether a model fits in memory, but also how its memory consumption changes during inference.
This is one of the central obstacles facing local AI today. Models are becoming accessible faster than the expertise required to deploy them.
The third barrier is trust and predictability
Strong benchmark performance does not guarantee reliable behaviour in production.
A model may perform well on a public leaderboard yet struggle with structured outputs, tool calls, multilingual inputs, long contexts or unusual edge cases. Its behaviour may also change depending on the quantisation, runtime and target hardware.
This creates an evaluation problem.
Public benchmarks rarely reflect a company's actual workload, so teams need their own datasets and testing frameworks to determine whether a model is reliable enough for production. They must evaluate not only average quality, but also failure modes, regressions and performance on edge cases.
It also creates an observability problem.
Managed platforms often come with more mature tooling for tracing, usage measurement, error reporting and monitoring. With local inference, teams may need to assemble or build much of this infrastructure themselves.
They need to detect failures, measure output quality, compare model versions, understand regressions and identify why a model produced a particular result.
Without strong evaluation and observability, companies cannot confidently know when a local model is performing well, when its quality is degrading or when a request should be escalated to a larger model.
For many teams, paying for a hosted API remains easier than taking responsibility for a system they cannot yet measure or trust consistently.
Conclusion
My belief is that frontier models will increasingly focus on the hardest problems available to us.
Their potential market is effectively unlimited: scientific discovery, curing diseases, accelerating climate research, designing new materials and tackling moonshot projects that require levels of reasoning far beyond today's routine applications.
But most of our daily AI usage does not require that level of intelligence.
Summarising a document, extracting information, classifying an input, generating a reply, calling a tool or powering a specialised assistant can already be handled by many of today's small models.
The gap is no longer only about capability. It is about awareness, operational complexity, evaluation, observability and trust.
If we can overcome those barriers, small models can become the default layer for everyday inference, while frontier models remain focused on the problems where their additional intelligence creates the most value.
I don't think the future is one where frontier models disappear.
I think the future is one where they become increasingly specialised, solving humanity's hardest problems, while billions of everyday AI requests are handled by small models running closer to the user.
That is why I believe small models will dominate.
Just not yet.