· AI · 7 min read
What Share of AI Tasks Needs the Most Expensive Model
A developer team at Vercel now has an agent writing a quarter to a third of its pull requests. The exact share is not the interesting number. What is interesting is that someone measured it at all.
The share of AI tasks that a cheap open model handles just as well as the most expensive frontier model can be measured instead of guessed.
The measurement runs through a task inventory built from interviews and usage samples, clustered by task type and scored on two axes: difficulty and sensitivity. The result is a number, not a guess, and the same table also delivers the cost model and the data-routing profile as a byproduct.
On August 12, 2026, the developer team behind Vercel’s AI SDK published a blog post with an unusually concrete number. Four weeks earlier, they had put an agent process into production that writes pull requests for their own open source project. The result after four weeks: 25 to 35 percent of weekly merged pull requests come from the agent, over 75 percent of issues closed in July went to its account, open bugs dropped by roughly 25 percent. Nothing gets merged without human approval.
The exact number is not the point. The point is that Vercel knows it at all. Most companies could not say what share of their AI tasks would run just as well on a model that costs a fraction of the price. Not because the answer is secret, but because nobody looked.
The task mix matters, not the average
The previous post in this series quantified the quality gap between open and proprietary models: a few points at the benchmark level, unevenly distributed across task types. On routine tasks with a familiar pattern, the difference is barely measurable anymore, the so-called Center of Distribution. On novel problems with multi-step reasoning it stays real, the Edge.
The consequence was an open question: whether switching to a cheaper model pays off does not depend on the model, but on how large the center share of your own tasks actually is. That number is in no benchmark. It sits inside your own company, unmeasured.
How to actually measure the share
A reliable measurement needs two sources. Interviews reveal what employees do and want to do with AI, including tasks nobody has automated yet. Where accessible, pseudonymized usage samples from existing tools round out the picture, covered by the existing data processing agreement. Together, both show more than either alone: interviews surface tasks that are not in the logs yet, logs correct what people believe about their own work.
The second step clusters the collected tasks into types. Not “what Ms. Smith does,” but task types like email drafting, meeting summaries, contract review, or code snippets that recur across multiple people and departments.
Each type is then scored on two axes, not just one:
| Axis | Asks | Determines |
|---|---|---|
| Difficulty | Center of Distribution or Edge: familiar pattern, easily verifiable result, or novel problem with high error costs? | How much model quality the task actually needs |
| Sensitivity | Routine or confidential: data that can go anywhere, or data with a tighter circle? | Which data routes are viable for the task |
Two axes from one measurement is not a coincidence. The same table that yields the cost share also delivers the data-routing profile that answers which data is allowed to leave the building at all. One survey, two answers. The end result is a number: X percent of task volume is Center of Distribution and routine, and for exactly that share a cheap model carries the work without a noticeable quality loss.
Where the map does not sit still
A task that counts as Edge today can sit in the Center six weeks from now. The previous post showed how fast this shifts: when GLM-5.2 cracked the 50-point mark on the Intelligence Index in June as the first open model, the next open models trailed clearly behind. Six weeks later, Moonshot AI released the Kimi K3 weights; in the current index it sits just three points behind the frontier. A task that needed an expensive model in June might already run on a cheap one in August.
A one-time measurement is therefore a snapshot, not an end state. Anyone who surveys the task mix once and then leaves the table sitting is working with a guess again after six months, just with more effort behind it.
What the measurement does not replace
A measurement that broadly recommends switching to cheaper models would not be a measurement, it would be a sales pitch with an extra step. Interactive, demanding work with high error costs, such as software development or strategic analysis, sits mostly at the edge for most companies. There, a more expensive model often remains the right choice, even if other departments in the same company mostly handle routine tasks. The Vercel agent writes some of the pull requests, not all of them, and nothing gets merged without human review. That distinction is exactly why the measurement runs per task type, not per department or company as a whole.
Which part of this number can actually be read off the existing admin center, and which cannot, is the topic of the next post.
Frequently asked questions
How do you measure what share of your own AI tasks runs on a cheap model? Through a task inventory built from interviews and, where accessible, pseudonymized usage samples. Tasks are clustered into types and scored on two axes: difficulty (Center or Edge of Distribution) and sensitivity (routine or confidential). The distribution of types yields the share that a cheap model can carry.
Why is the admin center in Copilot or similar tools not enough for this? Admin centers show usage frequency and license consumption, but no task distribution, no cost comparison between models, and no sensitivity profile of the data. The raw numbers are a starting point, not an answer.
Does a high center share automatically mean you should switch to a cheap model? No. The measurement provides the basis for a decision, not an automatic recommendation. Teams with a lot of interactive, demanding work, such as development or strategy, are often still better off with a more expensive model, even if other departments mostly handle routine tasks.
Sources
- Vercel AI SDK Factory, four weeks in production: 25-35% of weekly merged pull requests, over 75% of issues closed in July 2026, roughly 25% fewer open bugs, no merge without human approval (published 2026-08-12): Vercel Blog, “Building a software factory for AI SDK”
- GLM-5.2 as the first open model above 50 points (June 2026); Kimi K3 weights released July 27, 2026, at 60 points in the current index against 61 to 63 at the frontier: Artificial Analysis, Tom’s Hardware on the weights release, Artificial Analysis, model leaderboard (retrieved 2026-08-26)
Related: The Quality Gap in Open AI Models, Honestly Quantified, Residency Is Not Jurisdiction, and Paid Per Seat, Used Per Task.
