· AI · 7 min read
What Share of AI Tasks Needs the Most Expensive Model
A developer team at Vercel now has an agent writing a quarter to a third of its pull requests. The exact share is not the interesting number. What is interesting is that someone measured it at all.
The share of AI tasks that a cheap open model handles just as well as the most expensive frontier model can be measured instead of guessed.
A task inventory shows which tasks occur and how often. Tests with specific models then establish which tasks run well enough on a cheaper model. Difficulty and data sensitivity help select candidates. Costs and permitted data routes are assessed separately.
On August 12, 2026, the developer team behind Vercel’s AI SDK published a blog post with an unusually concrete number. Four weeks earlier, they had put an agent process into production that writes pull requests for their own open source project. The result after four weeks: 25 to 35 percent of weekly merged pull requests come from the agent, over 75 percent of issues closed in July went to its account, open bugs dropped by roughly 25 percent. Nothing gets merged without human approval.
The exact number is not the point. The point is that Vercel knows it at all. Most companies could not say what share of their AI tasks would run just as well on a model that costs a fraction of the price. Not because the answer is secret, but because nobody looked.
The task mix matters, not the average
The previous post in this series quantified the quality gap between open and proprietary models: a few points at the benchmark level, unevenly distributed across task types. On routine tasks with a familiar pattern, the difference is barely measurable anymore, the so-called Center of Distribution. On novel problems with multi-step reasoning it stays real, the Edge.
Whether switching pays off depends on the task mix, the quality of the specific models, and their usage and operating costs. A high center share is a reason to test cheaper models on your own tasks. The distribution of those tasks needs to be measured within your own company.
How to actually measure the share
A reliable measurement needs two sources. Interviews reveal what employees do and want to do with AI, including tasks nobody has automated yet. Where permitted, pseudonymized usage samples from existing tools add to the picture. Their purpose, access rules, and handling of personal data must be settled first. Together, both show more than either alone: interviews surface tasks that are not in the logs yet, logs correct what people believe about their own work.
The second step clusters the collected tasks into types. Not “what Ms. Smith does,” but task types like email drafting, meeting summaries, contract review, or code snippets that recur across multiple people and departments.
Each type is then scored on two axes, not just one:
| Axis | Asks | Determines |
|---|---|---|
| Difficulty | Center of Distribution or Edge: familiar pattern, easily verifiable result, or novel problem with high error costs? | How much model quality the task actually needs |
| Sensitivity | Which data is processed, and who may access it? | Which data routes need to be assessed for the task |
This classification identifies candidates for a model comparison. Each model is tested on examples of the same task type. Criteria agreed beforehand determine which results are usable and how much rework they need. The share of usable results can then be weighted by how often those tasks occur.
Classifying a task as routine does not establish sufficient quality. Frequent tasks can also involve confidential data. Which data may be processed and where belongs in the assessment of residency and jurisdiction. A cost comparison also accounts for the specific model prices and operation.
Where the map does not sit still
A task that counts as Edge today can sit in the Center six weeks from now. The previous post showed how fast this shifts: when GLM-5.2 cracked the 50-point mark on the Intelligence Index in June as the first open model, the next open models trailed clearly behind. Six weeks later, Moonshot AI released the Kimi K3 weights; in the current index it sits just three points behind the frontier. A task that needed an expensive model in June might already run on a cheap one in August.
A one-time measurement is therefore a snapshot, not an end state. Anyone who surveys the task mix once and then leaves the table sitting is working with a guess again after six months, just with more effort behind it.
What the measurement does not replace
A measurement that broadly recommends switching to cheaper models would not be a measurement, it would be a sales pitch with an extra step. Interactive, demanding work with high error costs, such as software development or strategic analysis, sits mostly at the edge for most companies. There, a more expensive model often remains the right choice, even if other departments in the same company mostly handle routine tasks. The Vercel agent writes some of the pull requests, not all of them, and nothing gets merged without human review. That distinction is exactly why the measurement runs per task type, not per department or company as a whole.
The usage data already available from the admin center and Copilot Dashboard is covered in the next post.
Frequently asked questions
How do you measure what share of your own AI tasks runs on a cheap model? Interviews and permitted usage samples produce a task inventory. Tasks are grouped by difficulty and data sensitivity. Specific models are then compared using examples and agreed quality criteria. Only those tests establish which share runs well enough on a cheaper model.
Why is the admin center in Copilot or similar tools not enough for this? The Copilot usage report in the admin center shows prompts and activity. The Copilot Dashboard in Viva Insights adds broad prompt categories. These help with a task inventory. Choosing a model also requires tests on your own tasks, a cost comparison, and a review of the data involved.
Does a high center share automatically mean you should switch to a cheap model? No. The measurement provides the basis for a decision, not an automatic recommendation. Teams with a lot of interactive, demanding work, such as development or strategy, are often still better off with a more expensive model, even if other departments mostly handle routine tasks.
Sources
- Vercel AI SDK Factory, four weeks in production: 25-35% of weekly merged pull requests, over 75% of issues closed in July 2026, roughly 25% fewer open bugs, no merge without human approval (published 2026-08-12): Vercel Blog, “Building a software factory for AI SDK”
- GLM-5.2 as the first open model above 50 points (June 2026); Kimi K3 weights released July 27, 2026, at 60 points in the current index against 61 to 63 at the frontier: Artificial Analysis, Tom’s Hardware on the weights release, Artificial Analysis, model leaderboard (retrieved 2026-08-26)
Related: The Quality Gap in Open AI Models, Honestly Quantified, Residency Is Not Jurisdiction, and Paid Per Seat, Used Per Task.
